@c -*-texinfo-*-@c This is part of the GNU Emacs Lisp Reference Manual.@c Copyright (C) 1998, 1999 Free Software Foundation, Inc. @c See the file elisp.texi for copying conditions.@setfilename ../info/characters@node Non-ASCII Characters, Searching and Matching, Text, Top@chapter Non-@sc{ascii} Characters@cindex multibyte characters@cindex non-@sc{ascii} characters This chapter covers the special issues relating to non-@sc{ascii}characters and how they are stored in strings and buffers.@menu* Text Representations:: Unibyte and multibyte representations* Converting Representations:: Converting unibyte to multibyte and vice versa.* Selecting a Representation:: Treating a byte sequence as unibyte or multi.* Character Codes:: How unibyte and multibyte relate to codes of individual characters.* Character Sets:: The space of possible characters codes is divided into various character sets.* Chars and Bytes:: More information about multibyte encodings.* Splitting Characters:: Converting a character to its byte sequence.* Scanning Charsets:: Which character sets are used in a buffer?* Translation of Characters:: Translation tables are used for conversion.* Coding Systems:: Coding systems are conversions for saving files.* Input Methods:: Input methods allow users to enter various non-ASCII characters without special keyboards.* Locales:: Interacting with the POSIX locale.@end menu@node Text Representations@section Text Representations@cindex text representations Emacs has two @dfn{text representations}---two ways to represent textin a string or buffer. These are called @dfn{unibyte} and@dfn{multibyte}. Each string, and each buffer, uses one of these tworepresentations. For most purposes, you can ignore the issue ofrepresentations, because Emacs converts text between them asappropriate. Occasionally in Lisp programming you will need to payattention to the difference.@cindex unibyte text In unibyte representation, each character occupies one byte andtherefore the possible character codes range from 0 to 255. Codes 0through 127 are @sc{ascii} characters; the codes from 128 through 255are used for one non-@sc{ascii} character set (you can choose whichcharacter set by setting the variable @code{nonascii-insert-offset}).@cindex leading code@cindex multibyte text@cindex trailing codes In multibyte representation, a character may occupy more than onebyte, and as a result, the full range of Emacs character codes can bestored. The first byte of a multibyte character is always in the range128 through 159 (octal 0200 through 0237). These values are called@dfn{leading codes}. The second and subsequent bytes of a multibytecharacter are always in the range 160 through 255 (octal 0240 through0377); these values are @dfn{trailing codes}. Some sequences of bytes are not valid in multibyte text: for example,a single isolated byte in the range 128 through 159 is not allowed. Butcharacter codes 128 through 159 can appear in multibyte text,represented as two-byte sequences. All the character codes 128 through255 are possible (though slightly abnormal) in multibyte text; theyappear in multibyte buffers and strings when you do explicit encodingand decoding (@pxref{Explicit Encoding}). In a buffer, the buffer-local value of the variable@code{enable-multibyte-characters} specifies the representation used.The representation for a string is determined and recorded in the stringwhen the string is constructed.@defvar enable-multibyte-charactersThis variable specifies the current buffer's text representation.If it is non-@code{nil}, the buffer contains multibyte text; otherwise,it contains unibyte text.You cannot set this variable directly; instead, use the function@code{set-buffer-multibyte} to change a buffer's representation.@end defvar@defvar default-enable-multibyte-charactersThis variable's value is entirely equivalent to @code{(default-value'enable-multibyte-characters)}, and setting this variable changes thatdefault value. Setting the local binding of@code{enable-multibyte-characters} in a specific buffer is not allowed,but changing the default value is supported, and it is a reasonablething to do, because it has no effect on existing buffers.The @samp{--unibyte} command line option does its job by setting thedefault value to @code{nil} early in startup.@end defvar@defun position-bytes position@tindex position-bytesReturn the byte-position corresponding to buffer position @var{position}in the current buffer.@end defun@defun byte-to-position byte-position@tindex byte-to-positionReturn the buffer position corresponding to byte-position@var{byte-position} in the current buffer.@end defun@defun multibyte-string-p stringReturn @code{t} if @var{string} is a multibyte string.@end defun@node Converting Representations@section Converting Text Representations Emacs can convert unibyte text to multibyte; it can also convertmultibyte text to unibyte, though this conversion loses information. Ingeneral these conversions happen when inserting text into a buffer, orwhen putting text from several strings together in one string. You canalso explicitly convert a string's contents to either representation. Emacs chooses the representation for a string based on the text thatit is constructed from. The general rule is to convert unibyte text tomultibyte text when combining it with other multibyte text, because themultibyte representation is more general and can hold whatevercharacters the unibyte text has. When inserting text into a buffer, Emacs converts the text to thebuffer's representation, as specified by@code{enable-multibyte-characters} in that buffer. In particular, whenyou insert multibyte text into a unibyte buffer, Emacs converts the textto unibyte, even though this conversion cannot in general preserve allthe characters that might be in the multibyte text. The other naturalalternative, to convert the buffer contents to multibyte, is notacceptable because the buffer's representation is a choice made by theuser that cannot be overridden automatically. Converting unibyte text to multibyte text leaves @sc{ascii} charactersunchanged, and likewise character codes 128 through 159. It convertsthe non-@sc{ascii} codes 160 through 255 by adding the value@code{nonascii-insert-offset} to each character code. By setting thisvariable, you specify which character set the unibyte characterscorrespond to (@pxref{Character Sets}). For example, if@code{nonascii-insert-offset} is 2048, which is @code{(- (make-char'latin-iso8859-1) 128)}, then the unibyte non-@sc{ascii} characterscorrespond to Latin 1. If it is 2688, which is @code{(- (make-char'greek-iso8859-7) 128)}, then they correspond to Greek letters. Converting multibyte text to unibyte is simpler: it discards all butthe low 8 bits of each character code. If @code{nonascii-insert-offset}has a reasonable value, corresponding to the beginning of some characterset, this conversion is the inverse of the other: converting unibytetext to multibyte and back to unibyte reproduces the original unibytetext.@defvar nonascii-insert-offsetThis variable specifies the amount to add to a non-@sc{ascii} characterwhen converting unibyte text to multibyte. It also applies when@code{self-insert-command} inserts a character in the unibytenon-@sc{ascii} range, 128 through 255. However, the functions@code{insert} and @code{insert-char} do not perform this conversion.The right value to use to select character set @var{cs} is @code{(-(make-char @var{cs}) 128)}. If the value of@code{nonascii-insert-offset} is zero, then conversion actually uses thevalue for the Latin 1 character set, rather than zero.@end defvar@defvar nonascii-translation-tableThis variable provides a more general alternative to@code{nonascii-insert-offset}. You can use it to specify independentlyhow to translate each code in the range of 128 through 255 into amultibyte character. The value should be a char-table, or @code{nil}.If this is non-@code{nil}, it overrides @code{nonascii-insert-offset}.@end defvar@defun string-make-unibyte stringThis function converts the text of @var{string} to unibyterepresentation, if it isn't already, and returns the result. If@var{string} is a unibyte string, it is returned unchanged.Multibyte character codes are converted to unibyteby using just the low 8 bits.@end defun@defun string-make-multibyte stringThis function converts the text of @var{string} to multibyterepresentation, if it isn't already, and returns the result. If@var{string} is a multibyte string, it is returned unchanged.The function @code{unibyte-char-to-multibyte} is used to converteach unibyte character to a multibyte character.@end defun@node Selecting a Representation@section Selecting a Representation Sometimes it is useful to examine an existing buffer or string asmultibyte when it was unibyte, or vice versa.@defun set-buffer-multibyte multibyteSet the representation type of the current buffer. If @var{multibyte}is non-@code{nil}, the buffer becomes multibyte. If @var{multibyte}is @code{nil}, the buffer becomes unibyte.This function leaves the buffer contents unchanged when viewed as asequence of bytes. As a consequence, it can change the contents viewedas characters; a sequence of two bytes which is treated as one characterin multibyte representation will count as two characters in unibyterepresentation. Character codes 128 through 159 are an exception. Theyare represented by one byte in a unibyte buffer, but when the buffer isset to multibyte, they are converted to two-byte sequences, and viceversa.This function sets @code{enable-multibyte-characters} to record whichrepresentation is in use. It also adjusts various data in the buffer(including overlays, text properties and markers) so that they cover thesame text as they did before.You cannot use @code{set-buffer-multibyte} on an indirect buffer,because indirect buffers always inherit the representation of thebase buffer.@end defun@defun string-as-unibyte stringThis function returns a string with the same bytes as @var{string} buttreating each byte as a character. This means that the value may havemore characters than @var{string} has.If @var{string} is already a unibyte string, then the value is@var{string} itself. Otherwise it is a newly created string, with notext properties. If @var{string} is multibyte, any characters itcontains of charset @var{eight-bit-control} or @var{eight-bit-graphic}are converted to the corresponding single byte.@end defun@defun string-as-multibyte stringThis function returns a string with the same bytes as @var{string} buttreating each multibyte sequence as one character. This means that thevalue may have fewer characters than @var{string} has.If @var{string} is already a multibyte string, then the value is@var{string} itself. Otherwise it is a newly created string, with notext properties. If @var{string} is unibyte and contains any individual8-bit bytes (i.e.@: not part of a multibyte form), they are converted tothe corresponding multibyte character of charset @var{eight-bit-control}or @var{eight-bit-graphic}.@end defun@node Character Codes@section Character Codes@cindex character codes The unibyte and multibyte text representations use different charactercodes. The valid character codes for unibyte representation range from0 to 255---the values that can fit in one byte. The valid charactercodes for multibyte representation range from 0 to 524287, but not allvalues in that range are valid. The values 128 through 255 are notentirely proper in multibyte text, but they can occur if you do explicitencoding and decoding (@pxref{Explicit Encoding}). Some other charactercodes cannot occur at all in multibyte text. Only the @sc{ascii} codes0 through 127 are completely legitimate in both representations.@defun char-valid-p charcode &optional genericpThis returns @code{t} if @var{charcode} is valid for either one of the twotext representations.@example(char-valid-p 65) @result{} t(char-valid-p 256) @result{} nil(char-valid-p 2248) @result{} t@end exampleIf the optional argument @var{genericp} is non-nil, this functionreturns @code{t} if @var{charcode} is a generic character(@pxref{Splitting Characters}).@end defun@node Character Sets@section Character Sets@cindex character sets Emacs classifies characters into various @dfn{character sets}, each ofwhich has a name which is a symbol. Each character belongs to one andonly one character set. In general, there is one character set for each distinct script. Forexample, @code{latin-iso8859-1} is one character set,@code{greek-iso8859-7} is another, and @code{ascii} is another. AnEmacs character set can hold at most 9025 characters; therefore, in somecases, characters that would logically be grouped together are splitinto several character sets. For example, one set of Chinesecharacters, generally known as Big 5, is divided into two Emacscharacter sets, @code{chinese-big5-1} and @code{chinese-big5-2}. @sc{ascii} characters are in character set @code{ascii}. Thenon-@sc{ascii} characters 128 through 159 are in character set@code{eight-bit-control}, and codes 160 through 255 are in character set@code{eight-bit-graphic}.@defun charsetp objectReturns @code{t} if @var{object} is a symbol that names a character set,@code{nil} otherwise.@end defun@defun charset-listThis function returns a list of all defined character set names.@end defun@defun char-charset characterThis function returns the name of the character set that @var{character}belongs to.@end defun@defun charset-plist charset@tindex charset-plistThis function returns the charset property list of the character set@var{charset}. Although @var{charset} is a symbol, this is not the sameas the property list of that symbol. Charset properties are used forspecial purposes within Emacs; for example,@code{preferred-coding-system} helps determine which coding system touse to encode characters in a charset.@end defun@node Chars and Bytes@section Characters and Bytes@cindex bytes and characters@cindex introduction sequence@cindex dimension (of character set) In multibyte representation, each character occupies one or morebytes. Each character set has an @dfn{introduction sequence}, which isnormally one or two bytes long. (Exception: the @sc{ascii} characterset and the @sc{eight-bit-graphic} character set have a zero-lengthintroduction sequence.) The introduction sequence is the beginning ofthe byte sequence for any character in the character set. The rest ofthe character's bytes distinguish it from the other characters in thesame character set. Depending on the character set, there are eitherone or two distinguishing bytes; the number of such bytes is called the@dfn{dimension} of the character set.@defun charset-dimension charsetThis function returns the dimension of @var{charset}; at present, thedimension is always 1 or 2.@end defun@defun charset-bytes charset@tindex charset-bytesThis function returns the number of bytes used to represent a characterin character set @var{charset}.@end defun This is the simplest way to determine the byte length of a characterset's introduction sequence:@example(- (charset-bytes @var{charset}) (charset-dimension @var{charset}))@end example@node Splitting Characters@section Splitting Characters The functions in this section convert between characters and the bytevalues used to represent them. For most purposes, there is no need tobe concerned with the sequence of bytes used to represent a character,because Emacs translates automatically when necessary.@defun split-char characterReturn a list containing the name of the character set of@var{character}, followed by one or two byte values (integers) whichidentify @var{character} within that character set. The number of bytevalues is the character set's dimension.@example(split-char 2248) @result{} (latin-iso8859-1 72)(split-char 65) @result{} (ascii 65)(split-char 128) @result{} (eight-bit-control 128)@end example@end defun@defun make-char charset &optional code1 code2This function returns the character in character set @var{charset} whoseposition codes are @var{code1} and @var{code2}. This is roughly theinverse of @code{split-char}. Normally, you should specify either oneor both of @var{code1} and @var{code2} according to the dimension of@var{charset}. For example,@example(make-char 'latin-iso8859-1 72) @result{} 2248@end example@end defun@cindex generic characters If you call @code{make-char} with no @var{byte-values}, the result isa @dfn{generic character} which stands for @var{charset}. A genericcharacter is an integer, but it is @emph{not} valid for insertion in thebuffer as a character. It can be used in @code{char-table-range} torefer to the whole character set (@pxref{Char-Tables}).@code{char-valid-p} returns @code{nil} for generic characters.For example:@example(make-char 'latin-iso8859-1) @result{} 2176(char-valid-p 2176) @result{} nil(char-valid-p 2176 t) @result{} t(split-char 2176) @result{} (latin-iso8859-1 0)@end exampleThe character sets @sc{ascii}, @sc{eight-bit-control}, and@sc{eight-bit-graphic} don't have corresponding generic characters. If@var{charset} is one of them and you don't supply @var{code1},@code{make-char} returns the character code corresponding to thesmallest code in @var{charset}.@node Scanning Charsets@section Scanning for Character Sets Sometimes it is useful to find out which character sets appear in apart of a buffer or a string. One use for this is in determining whichcoding systems (@pxref{Coding Systems}) are capable of representing allof the text in question.@defun find-charset-region beg end &optional translationThis function returns a list of the character sets that appear in thecurrent buffer between positions @var{beg} and @var{end}.The optional argument @var{translation} specifies a translation table tobe used in scanning the text (@pxref{Translation of Characters}). If itis non-@code{nil}, then each character in the region is translatedthrough this table, and the value returned describes the translatedcharacters instead of the characters actually in the buffer.@end defun@defun find-charset-string string &optional translationThis function returns a list of the character sets that appear in thestring @var{string}. It is just like @code{find-charset-region}, exceptthat it applies to the contents of @var{string} instead of part of thecurrent buffer.@end defun@node Translation of Characters@section Translation of Characters@cindex character translation tables@cindex translation tables A @dfn{translation table} specifies a mapping of charactersinto characters. These tables are used in encoding and decoding, andfor other purposes. Some coding systems specify their own particulartranslation tables; there are also default translation tables whichapply to all other coding systems.@defun make-translation-table &rest translationsThis function returns a translation table based on the argument@var{translations}. Each element of @var{translations} should be alist of elements of the form @code{(@var{from} . @var{to})}; this saysto translate the character @var{from} into @var{to}.The arguments and the forms in each argument are processed in order,and if a previous form already translates @var{to} to some othercharacter, say @var{to-alt}, @var{from} is also translated to@var{to-alt}.You can also map one whole character set into another character set withthe same dimension. To do this, you specify a generic character (whichdesignates a character set) for @var{from} (@pxref{Splitting Characters}).In this case, @var{to} should also be a generic character, for anothercharacter set of the same dimension. Then the translation tabletranslates each character of @var{from}'s character set into thecorresponding character of @var{to}'s character set.@end defun In decoding, the translation table's translations are applied to thecharacters that result from ordinary decoding. If a coding system hasproperty @code{character-translation-table-for-decode}, that specifiesthe translation table to use. Otherwise, if@code{standard-translation-table-for-decode} is non-@code{nil}, decodinguses that table. In encoding, the translation table's translations are applied to thecharacters in the buffer, and the result of translation is actuallyencoded. If a coding system has property@code{character-translation-table-for-encode}, that specifies thetranslation table to use. Otherwise the variable@code{standard-translation-table-for-encode} specifies the translationtable.@defvar standard-translation-table-for-decodeThis is the default translation table for decoding, forcoding systems that don't specify any other translation table.@end defvar@defvar standard-translation-table-for-encodeThis is the default translation table for encoding, forcoding systems that don't specify any other translation table.@end defvar@node Coding Systems@section Coding Systems@cindex coding system When Emacs reads or writes a file, and when Emacs sends text to asubprocess or receives text from a subprocess, it normally performscharacter code conversion and end-of-line conversion as specifiedby a particular @dfn{coding system}. How to define a coding system is an arcane matter, and is notdocumented here.@menu* Coding System Basics:: Basic concepts.* Encoding and I/O:: How file I/O functions handle coding systems.* Lisp and Coding Systems:: Functions to operate on coding system names.* User-Chosen Coding Systems:: Asking the user to choose a coding system.* Default Coding Systems:: Controlling the default choices.* Specifying Coding Systems:: Requesting a particular coding system for a single file operation.* Explicit Encoding:: Encoding or decoding text without doing I/O.* Terminal I/O Encoding:: Use of encoding for terminal I/O.* MS-DOS File Types:: How DOS "text" and "binary" files relate to coding systems.@end menu@node Coding System Basics@subsection Basic Concepts of Coding Systems@cindex character code conversion @dfn{Character code conversion} involves conversion between the encodingused inside Emacs and some other encoding. Emacs supports manydifferent encodings, in that it can convert to and from them. Forexample, it can convert text to or from encodings such as Latin 1, Latin2, Latin 3, Latin 4, Latin 5, and several variants of ISO 2022. In somecases, Emacs supports several alternative encodings for the samecharacters; for example, there are three coding systems for the Cyrillic(Russian) alphabet: ISO, Alternativnyj, and KOI8. Most coding systems specify a particular character code forconversion, but some of them leave the choice unspecified---to be chosenheuristically for each file, based on the data.@cindex end of line conversion @dfn{End of line conversion} handles three different conventions usedon various systems for representing end of line in files. The Unixconvention is to use the linefeed character (also called newline). TheDOS convention is to use a carriage-return and a linefeed at the end ofa line. The Mac convention is to use just carriage-return.@cindex base coding system@cindex variant coding system @dfn{Base coding systems} such as @code{latin-1} leave the end-of-lineconversion unspecified, to be chosen based on the data. @dfn{Variantcoding systems} such as @code{latin-1-unix}, @code{latin-1-dos} and@code{latin-1-mac} specify the end-of-line conversion explicitly aswell. Most base coding systems have three corresponding variants whosenames are formed by adding @samp{-unix}, @samp{-dos} and @samp{-mac}. The coding system @code{raw-text} is special in that it preventscharacter code conversion, and causes the buffer visited with thatcoding system to be a unibyte buffer. It does not specify theend-of-line conversion, allowing that to be determined as usual by thedata, and has the usual three variants which specify the end-of-lineconversion. @code{no-conversion} is equivalent to @code{raw-text-unix}:it specifies no conversion of either character codes or end-of-line. The coding system @code{emacs-mule} specifies that the data isrepresented in the internal Emacs encoding. This is like@code{raw-text} in that no code conversion happens, but different inthat the result is multibyte data.@defun coding-system-get coding-system propertyThis function returns the specified property of the coding system@var{coding-system}. Most coding system properties exist for internalpurposes, but one that you might find useful is @code{mime-charset}.That property's value is the name used in MIME for the character codingwhich this coding system can read and write. Examples:@example(coding-system-get 'iso-latin-1 'mime-charset) @result{} iso-8859-1(coding-system-get 'iso-2022-cn 'mime-charset) @result{} iso-2022-cn(coding-system-get 'cyrillic-koi8 'mime-charset) @result{} koi8-r@end exampleThe value of the @code{mime-charset} property is also definedas an alias for the coding system.@end defun@node Encoding and I/O@subsection Encoding and I/O The principal purpose of coding systems is for use in reading andwriting files. The function @code{insert-file-contents} usesa coding system for decoding the file data, and @code{write-region}uses one to encode the buffer contents. You can specify the coding system to use either explicitly(@pxref{Specifying Coding Systems}), or implicitly using the defaultingmechanism (@pxref{Default Coding Systems}). But these methods may notcompletely specify what to do. For example, they may choose a codingsystem such as @code{undefined} which leaves the character codeconversion to be determined from the data. In these cases, the I/Ooperation finishes the job of choosing a coding system. Very oftenyou will want to find out afterwards which coding system was chosen.@defvar buffer-file-coding-systemThis variable records the coding system that was used for visiting thecurrent buffer. It is used for saving the buffer, and for writing partof the buffer with @code{write-region}. When those operations ask theuser to specify a different coding system,@code{buffer-file-coding-system} is updated to the coding systemspecified.However, @code{buffer-file-coding-system} does not affect sending textto a subprocess.@end defvar@defvar save-buffer-coding-systemThis variable specifies the coding system for saving the buffer (byoverriding @code{buffer-file-coding-system}). Note that it is not usedfor @code{write-region}.When a command to save the buffer starts out to use@code{buffer-file-coding-system} (or @code{save-buffer-coding-system}),and that coding system cannot handlethe actual text in the buffer, the command asks the user to chooseanother coding system. After that happens, the command also updates@code{buffer-file-coding-system} to represent the coding system that theuser specified.@end defvar@defvar last-coding-system-usedI/O operations for files and subprocesses set this variable to thecoding system name that was used. The explicit encoding and decodingfunctions (@pxref{Explicit Encoding}) set it too.@strong{Warning:} Since receiving subprocess output sets this variable,it can change whenever Emacs waits; therefore, you should copy thevalue shortly after the function call that stores the value you areinterested in.@end defvar The variable @code{selection-coding-system} specifies how to encodeselections for the window system. @xref{Window System Selections}.@node Lisp and Coding Systems@subsection Coding Systems in Lisp Here are the Lisp facilities for working with coding systems:@defun coding-system-list &optional base-onlyThis function returns a list of all coding system names (symbols). If@var{base-only} is non-@code{nil}, the value includes only thebase coding systems. Otherwise, it includes alias and variant codingsystems as well.@end defun@defun coding-system-p objectThis function returns @code{t} if @var{object} is a coding systemname.@end defun@defun check-coding-system coding-systemThis function checks the validity of @var{coding-system}.If that is valid, it returns @var{coding-system}.Otherwise it signals an error with condition @code{coding-system-error}.@end defun@defun coding-system-change-eol-conversion coding-system eol-typeThis function returns a coding system which is like @var{coding-system}except for its eol conversion, which is specified by @code{eol-type}.@var{eol-type} should be @code{unix}, @code{dos}, @code{mac}, or@code{nil}. If it is @code{nil}, the returned coding system determinesthe end-of-line conversion from the data.@end defun@defun coding-system-change-text-conversion eol-coding text-codingThis function returns a coding system which uses the end-of-lineconversion of @var{eol-coding}, and the text conversion of@var{text-coding}. If @var{text-coding} is @code{nil}, it returns@code{undecided}, or one of its variants according to @var{eol-coding}.@end defun@defun find-coding-systems-region from toThis function returns a list of coding systems that could be used toencode a text between @var{from} and @var{to}. All coding systems inthe list can safely encode any multibyte characters in that portion ofthe text.If the text contains no multibyte characters, the function returns thelist @code{(undecided)}.@end defun@defun find-coding-systems-string stringThis function returns a list of coding systems that could be used toencode the text of @var{string}. All coding systems in the list cansafely encode any multibyte characters in @var{string}. If the textcontains no multibyte characters, this returns the list@code{(undecided)}.@end defun@defun find-coding-systems-for-charsets charsetsThis function returns a list of coding systems that could be used toencode all the character sets in the list @var{charsets}.@end defun@defun detect-coding-region start end &optional highestThis function chooses a plausible coding system for decoding the textfrom @var{start} to @var{end}. This text should be a byte sequence(@pxref{Explicit Encoding}).Normally this function returns a list of coding systems that couldhandle decoding the text that was scanned. They are listed in order ofdecreasing priority. But if @var{highest} is non-@code{nil}, then thereturn value is just one coding system, the one that is highest inpriority.If the region contains only @sc{ascii} characters, the valueis @code{undecided} or @code{(undecided)}.@end defun@defun detect-coding-string string highestThis function is like @code{detect-coding-region} except that itoperates on the contents of @var{string} instead of bytes in the buffer.@end defun @xref{Process Information}, for how to examine or set the codingsystems used for I/O to a subprocess.@node User-Chosen Coding Systems@subsection User-Chosen Coding Systems@defun select-safe-coding-system from to &optional default-coding-system accept-default-pThis function selects a coding system for encoding specified text,asking the user to choose if necessary. Normally the specified textis the text in the current buffer between @var{from} and @var{to},defaulting to the whole buffer if they are @code{nil}. If @var{from}is a string, the string specifies the text to encode, and @var{to} isignored.If @var{default-coding-system} is non-@code{nil}, that is the firstcoding system to try; if that can handle the text,@code{select-safe-coding-system} returns that coding system. It canalso be a list of coding systems; then the function tries each of themone by one. After trying all of them, it next tries the user's mostpreferred coding system (@pxref{Recognize Coding,prefer-coding-system, the description of @code{prefer-coding-system},emacs, GNU Emacs Manual}), and after that the current buffer's valueof @code{buffer-file-coding-system} (if it is not @code{undecided}).If one of those coding systems can safely encode all the specifiedtext, @code{select-safe-coding-system} chooses it and returns it.Otherwise, it asks the user to choose from a list of coding systemswhich can encode all the text, and returns the user's choice.The optional argument @var{accept-default-p}, if non-@code{nil},should be a function to determine whether the coding system selectedwithout user interaction is acceptable. If this function returns@code{nil}, the silently selected coding system is rejected, and theuser is asked to select a coding system from a list of possiblecandidates.@vindex select-safe-coding-system-accept-default-pIf the variable @code{select-safe-coding-system-accept-default-p} isnon-@code{nil}, its value overrides the value of@var{accept-default-p}.@end defun Here are two functions you can use to let the user specify a codingsystem, with completion. @xref{Completion}.@defun read-coding-system prompt &optional defaultThis function reads a coding system using the minibuffer, prompting withstring @var{prompt}, and returns the coding system name as a symbol. Ifthe user enters null input, @var{default} specifies which coding systemto return. It should be a symbol or a string.@end defun@defun read-non-nil-coding-system promptThis function reads a coding system using the minibuffer, prompting withstring @var{prompt}, and returns the coding system name as a symbol. Ifthe user tries to enter null input, it asks the user to try again.@xref{Coding Systems}.@end defun@node Default Coding Systems@subsection Default Coding Systems This section describes variables that specify the default codingsystem for certain files or when running certain subprograms, and thefunction that I/O operations use to access them. The idea of these variables is that you set them once and for all to thedefaults you want, and then do not change them again. To specify aparticular coding system for a particular operation in a Lisp program,don't change these variables; instead, override them using@code{coding-system-for-read} and @code{coding-system-for-write}(@pxref{Specifying Coding Systems}).@defvar auto-coding-regexp-alistThis variable is an alist of text patterns and corresponding codingsystems. Each element has the form @code{(@var{regexp}. @var{coding-system})}; a file whose first few kilobytes match@var{regexp} is decoded with @var{coding-system} when its contents areread into a buffer. The settings in this alist take priority over@code{coding:} tags in the files and the contents of@code{file-coding-system-alist} (see below). The default value is setso that Emacs automatically recognizes mail files in Babyl format andreads them with no code conversions.@end defvar@defvar file-coding-system-alistThis variable is an alist that specifies the coding systems to use forreading and writing particular files. Each element has the form@code{(@var{pattern} . @var{coding})}, where @var{pattern} is a regularexpression that matches certain file names. The element applies to filenames that match @var{pattern}.The @sc{cdr} of the element, @var{coding}, should be either a codingsystem, a cons cell containing two coding systems, or a function name (asymbol with a function definition). If @var{coding} is a coding system,that coding system is used for both reading the file and writing it. If@var{coding} is a cons cell containing two coding systems, its @sc{car}specifies the coding system for decoding, and its @sc{cdr} specifies thecoding system for encoding.If @var{coding} is a function name, the function must return a codingsystem or a cons cell containing two coding systems. This value is usedas described above.@end defvar@defvar process-coding-system-alistThis variable is an alist specifying which coding systems to use for asubprocess, depending on which program is running in the subprocess. Itworks like @code{file-coding-system-alist}, except that @var{pattern} ismatched against the program name used to start the subprocess. The codingsystem or systems specified in this alist are used to initialize thecoding systems used for I/O to the subprocess, but you can specifyother coding systems later using @code{set-process-coding-system}.@end defvar @strong{Warning:} Coding systems such as @code{undecided}, whichdetermine the coding system from the data, do not work entirely reliablywith asynchronous subprocess output. This is because Emacs handlesasynchronous subprocess output in batches, as it arrives. If the codingsystem leaves the character code conversion unspecified, or leaves theend-of-line conversion unspecified, Emacs must try to detect the properconversion from one batch at a time, and this does not always work. Therefore, with an asynchronous subprocess, if at all possible, use acoding system which determines both the character code conversion andthe end of line conversion---that is, one like @code{latin-1-unix},rather than @code{undecided} or @code{latin-1}.@defvar network-coding-system-alistThis variable is an alist that specifies the coding system to use fornetwork streams. It works much like @code{file-coding-system-alist},with the difference that the @var{pattern} in an element may be either aport number or a regular expression. If it is a regular expression, itis matched against the network service name used to open the networkstream.@end defvar@defvar default-process-coding-systemThis variable specifies the coding systems to use for subprocess (andnetwork stream) input and output, when nothing else specifies what todo.The value should be a cons cell of the form @code{(@var{input-coding}. @var{output-coding})}. Here @var{input-coding} applies to input fromthe subprocess, and @var{output-coding} applies to output to it.@end defvar@defun find-operation-coding-system operation &rest argumentsThis function returns the coding system to use (by default) forperforming @var{operation} with @var{arguments}. The value has thisform:@example(@var{decoding-system} @var{encoding-system})@end exampleThe first element, @var{decoding-system}, is the coding system to usefor decoding (in case @var{operation} does decoding), and@var{encoding-system} is the coding system for encoding (in case@var{operation} does encoding).The argument @var{operation} should be a symbol, one of@code{insert-file-contents}, @code{write-region}, @code{call-process},@code{call-process-region}, @code{start-process}, or@code{open-network-stream}. These are the names of the Emacs I/O primitivesthat can do coding system conversion.The remaining arguments should be the same arguments that might be givento that I/O primitive. Depending on the primitive, one of thosearguments is selected as the @dfn{target}. For example, if@var{operation} does file I/O, whichever argument specifies the filename is the target. For subprocess primitives, the process name is thetarget. For @code{open-network-stream}, the target is the service nameor port number.This function looks up the target in @code{file-coding-system-alist},@code{process-coding-system-alist}, or@code{network-coding-system-alist}, depending on @var{operation}.@xref{Default Coding Systems}.@end defun@node Specifying Coding Systems@subsection Specifying a Coding System for One Operation You can specify the coding system for a specific operation by bindingthe variables @code{coding-system-for-read} and/or@code{coding-system-for-write}.@defvar coding-system-for-readIf this variable is non-@code{nil}, it specifies the coding system touse for reading a file, or for input from a synchronous subprocess.It also applies to any asynchronous subprocess or network stream, but ina different way: the value of @code{coding-system-for-read} when youstart the subprocess or open the network stream specifies the inputdecoding method for that subprocess or network stream. It remains inuse for that subprocess or network stream unless and until overridden.The right way to use this variable is to bind it with @code{let} for aspecific I/O operation. Its global value is normally @code{nil}, andyou should not globally set it to any other value. Here is an exampleof the right way to use the variable:@example;; @r{Read the file with no character code conversion.};; @r{Assume @sc{crlf} represents end-of-line.}(let ((coding-system-for-write 'emacs-mule-dos)) (insert-file-contents filename))@end exampleWhen its value is non-@code{nil}, @code{coding-system-for-read} takesprecedence over all other methods of specifying a coding system to use forinput, including @code{file-coding-system-alist},@code{process-coding-system-alist} and@code{network-coding-system-alist}.@end defvar@defvar coding-system-for-writeThis works much like @code{coding-system-for-read}, except that itapplies to output rather than input. It affects writing to files,as well as sending output to subprocesses and net connections.When a single operation does both input and output, as do@code{call-process-region} and @code{start-process}, both@code{coding-system-for-read} and @code{coding-system-for-write}affect it.@end defvar@defvar inhibit-eol-conversionWhen this variable is non-@code{nil}, no end-of-line conversion is done,no matter which coding system is specified. This applies to all theEmacs I/O and subprocess primitives, and to the explicit encoding anddecoding functions (@pxref{Explicit Encoding}).@end defvar@node Explicit Encoding@subsection Explicit Encoding and Decoding@cindex encoding text@cindex decoding text All the operations that transfer text in and out of Emacs have theability to use a coding system to encode or decode the text.You can also explicitly encode and decode text using the functionsin this section. The result of encoding, and the input to decoding, are not ordinarytext. They logically consist of a series of byte values; that is, aseries of characters whose codes are in the range 0 through 255. In amultibyte buffer or string, character codes 128 through 159 arerepresented by multibyte sequences, but this is invisible to Lispprograms. The usual way to read a file into a buffer as a sequence of bytes, soyou can decode the contents explicitly, is with@code{insert-file-contents-literally} (@pxref{Reading from Files});alternatively, specify a non-@code{nil} @var{rawfile} argument whenvisiting a file with @code{find-file-noselect}. These methods result ina unibyte buffer. The usual way to use the byte sequence that results from explicitlyencoding text is to copy it to a file or process---for example, to writeit with @code{write-region} (@pxref{Writing to Files}), and suppressencoding by binding @code{coding-system-for-write} to@code{no-conversion}. Here are the functions to perform explicit encoding or decoding. Thedecoding functions produce sequences of bytes; the encoding functionsare meant to operate on sequences of bytes. All of these functionsdiscard text properties.@defun encode-coding-region start end coding-systemThis function encodes the text from @var{start} to @var{end} accordingto coding system @var{coding-system}. The encoded text replaces theoriginal text in the buffer. The result of encoding is logically asequence of bytes, but the buffer remains multibyte if it was multibytebefore.@end defun@defun encode-coding-string string coding-systemThis function encodes the text in @var{string} according to codingsystem @var{coding-system}. It returns a new string containing theencoded text. The result of encoding is a unibyte string.@end defun@defun decode-coding-region start end coding-systemThis function decodes the text from @var{start} to @var{end} accordingto coding system @var{coding-system}. The decoded text replaces theoriginal text in the buffer. To make explicit decoding useful, the textbefore decoding ought to be a sequence of byte values, but bothmultibyte and unibyte buffers are acceptable.@end defun@defun decode-coding-string string coding-systemThis function decodes the text in @var{string} according to codingsystem @var{coding-system}. It returns a new string containing thedecoded text. To make explicit decoding useful, the contents of@var{string} ought to be a sequence of byte values, but a multibytestring is acceptable.@end defun@node Terminal I/O Encoding@subsection Terminal I/O Encoding Emacs can decode keyboard input using a coding system, and encodeterminal output. This is useful for terminals that transmit or displaytext using a particular encoding such as Latin-1. Emacs does not set@code{last-coding-system-used} for encoding or decoding for theterminal.@defun keyboard-coding-systemThis function returns the coding system that is in use for decodingkeyboard input---or @code{nil} if no coding system is to be used.@end defun@defun set-keyboard-coding-system coding-systemThis function specifies @var{coding-system} as the coding system touse for decoding keyboard input. If @var{coding-system} is @code{nil},that means do not decode keyboard input.@end defun@defun terminal-coding-systemThis function returns the coding system that is in use for encodingterminal output---or @code{nil} for no encoding.@end defun@defun set-terminal-coding-system coding-systemThis function specifies @var{coding-system} as the coding system to usefor encoding terminal output. If @var{coding-system} is @code{nil},that means do not encode terminal output.@end defun@node MS-DOS File Types@subsection MS-DOS File Types@cindex DOS file types@cindex MS-DOS file types@cindex Windows file types@cindex file types on MS-DOS and Windows@cindex text files and binary files@cindex binary files and text files On MS-DOS and Microsoft Windows, Emacs guesses the appropriateend-of-line conversion for a file by looking at the file's name. Thisfeature classifies files as @dfn{text files} and @dfn{binary files}. By``binary file'' we mean a file of literal byte values that are notnecessarily meant to be characters; Emacs does no end-of-line conversionand no character code conversion for them. On the other hand, the bytesin a text file are intended to represent characters; when you create anew file whose name implies that it is a text file, Emacs uses DOSend-of-line conversion.@defvar buffer-file-typeThis variable, automatically buffer-local in each buffer, records thefile type of the buffer's visited file. When a buffer does not specifya coding system with @code{buffer-file-coding-system}, this variable isused to determine which coding system to use when writing the contentsof the buffer. It should be @code{nil} for text, @code{t} for binary.If it is @code{t}, the coding system is @code{no-conversion}.Otherwise, @code{undecided-dos} is used.Normally this variable is set by visiting a file; it is set to@code{nil} if the file was visited without any actual conversion.@end defvar@defopt file-name-buffer-file-type-alistThis variable holds an alist for recognizing text and binary files.Each element has the form (@var{regexp} . @var{type}), where@var{regexp} is matched against the file name, and @var{type} may be@code{nil} for text, @code{t} for binary, or a function to call tocompute which. If it is a function, then it is called with a singleargument (the file name) and should return @code{t} or @code{nil}.When running on MS-DOS or MS-Windows, Emacs checks this alist to decidewhich coding system to use when reading a file. For a text file,@code{undecided-dos} is used. For a binary file, @code{no-conversion}is used.If no element in this alist matches a given file name, then@code{default-buffer-file-type} says how to treat the file.@end defopt@defopt default-buffer-file-typeThis variable says how to handle files for which@code{file-name-buffer-file-type-alist} says nothing about the type.If this variable is non-@code{nil}, then these files are treated asbinary: the coding system @code{no-conversion} is used. Otherwise,nothing special is done for them---the coding system is deduced solelyfrom the file contents, in the usual Emacs fashion.@end defopt@node Input Methods@section Input Methods@cindex input methods @dfn{Input methods} provide convenient ways of entering non-@sc{ascii}characters from the keyboard. Unlike coding systems, which translatenon-@sc{ascii} characters to and from encodings meant to be read byprograms, input methods provide human-friendly commands. (@xref{InputMethods,,, emacs, The GNU Emacs Manual}, for information on how usersuse input methods to enter text.) How to define input methods is notyet documented in this manual, but here we describe how to use them. Each input method has a name, which is currently a string;in the future, symbols may also be usable as input method names.@defvar current-input-methodThis variable holds the name of the input method now active in thecurrent buffer. (It automatically becomes local in each buffer when setin any fashion.) It is @code{nil} if no input method is active in thebuffer now.@end defvar@defvar default-input-methodThis variable holds the default input method for commands that choose aninput method. Unlike @code{current-input-method}, this variable isnormally global.@end defvar@defun set-input-method input-methodThis function activates input method @var{input-method} for the currentbuffer. It also sets @code{default-input-method} to @var{input-method}.If @var{input-method} is @code{nil}, this function deactivates any inputmethod for the current buffer.@end defun@defun read-input-method-name prompt &optional default inhibit-nullThis function reads an input method name with the minibuffer, promptingwith @var{prompt}. If @var{default} is non-@code{nil}, that is returnedby default, if the user enters empty input. However, if@var{inhibit-null} is non-@code{nil}, empty input signals an error.The returned value is a string.@end defun@defvar input-method-alistThis variable defines all the supported input methods.Each element defines one input method, and should have the form:@example(@var{input-method} @var{language-env} @var{activate-func} @var{title} @var{description} @var{args}...)@end exampleHere @var{input-method} is the input method name, a string;@var{language-env} is another string, the name of the languageenvironment this input method is recommended for. (That serves only fordocumentation purposes.)@var{activate-func} is a function to call to activate this method. The@var{args}, if any, are passed as arguments to @var{activate-func}. Alltold, the arguments to @var{activate-func} are @var{input-method} andthe @var{args}.@var{title} is a string to display in the mode line while this method isactive. @var{description} is a string describing this method and whatit is good for.@end defvar The fundamental interface to input methods is through thevariable @code{input-method-function}. @xref{Reading One Event}.@node Locales@section Locales@cindex locale POSIX defines a concept of ``locales'' which control which languageto use in language-related features. These Emacs variables controlhow Emacs interacts with these features.@defvar locale-coding-system@tindex locale-coding-systemThis variable specifies the coding system to use for decoding systemerror messages, for encoding the format argument to@code{format-time-string}, and for decoding the return value of@code{format-time-string}.@end defvar@defvar system-messages-locale@tindex system-messages-localeThis variable specifies the locale to use for generating system errormessages. Changing the locale can cause messages to come out in adifferent language or in a different orthography. If the variable is@code{nil}, the locale is specified by environment variables in theusual POSIX fashion.@end defvar@defvar system-time-locale@tindex system-time-localeThis variable specifies the locale to use for formatting time values.Changing the locale can cause messages to appear according to theconventions of a different language. If the variable is @code{nil}, thelocale is specified by environment variables in the usual POSIX fashion.@end defvar