annotate README.unicode @ 89331:1892a75ffcac

(CATEGORY_MASK_RAW_TEXT): New macro. (detect_coding_utf_8, detect_coding_utf_16) (detect_coding_emacs_mule, detect_coding_iso_2022) (detect_coding_sjis, detect_coding_big5) (detect_coding_ccl, detect_coding_charset): Change argument MASK to DETECT_INFO. Update DETECT_INFO and return 1 if the byte sequence is valid in this coding system. Callers changed. (MAX_ANNOTATION_LENGTH): New macro. (ADD_ANNOTATION_DATA): New macro. (ADD_COMPOSITION_DATA): Argument changed. Callers changed. Call ADD_ANNOTATION_DATA. The format of annotation data changed. (ADD_CHARSET_DATA): New macro. (emacs_mule_char): New argument ID. Callers changed. (decode_coding_emacs_mule, decode_coding_iso_2022) (decode_coding_sjis, decode_coding_big5, decode_coding_charset): Produce charset annotation data in coding->charbuf. (encode_coding_emacs_mule, encode_coding_iso_2022): Pay attention to charset annotation data in coding->charbuf. (setup_coding_system): Add CODING_ANNOTATE_CHARSET_MASK coding->common_flags if the coding system is iso-2022 based and uses designation. (produce_composition): Adjusted for the new annotation data format. (produce_charset): New function. (produce_annotation): Handle charset annotation. (handle_composition_annotation, handle_charset_annotation): New functions. (consume_chars): Handle charset annotation. Utilize the above two functions. (encode_coding_object): If SRC_OBJECT and DST_OBJECT are the same buffer, get the deleted text as a string and set coding->src_object to that string. (detect_coding, detect_coding_system): Use the new struct coding_detection_info.
author Kenichi Handa <handa@m17n.org>
date Mon, 06 Jan 2003 11:37:17 +0000
parents d2c6faa9211a
children 2ffdae6aec84
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
89231
a4f536fc73d9 Fix mode line.
Dave Love <fx@gnu.org>
parents: 89215
diff changeset
1 -*-mode: text; coding: latin-1;-*-
88740
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
2
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
3 Problems, fixmes and other issues in the emacs-unicode branch
88884
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
4 -------------------------------------------------------------
88740
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
5
88836
b53799fa4747 *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88763
diff changeset
6 Notes by fx to record various things of variable importance. handa
b53799fa4747 *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88763
diff changeset
7 needs to check them -- don't take too seriously, especially with
b53799fa4747 *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88763
diff changeset
8 regard to completeness.
88740
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
9
88884
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
10 _Do take seriously that you don't want this branch unless you're
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
11 actually working on it; you risk your data by actually using it._ If
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
12 you just want to edit Unicode and/or unify iso-8859 et al, see the
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
13 existing support and the extra stuff at
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
14 <URL:ftp://dlpx1.dl.ac.uk/fx/emacs/Mule>, mostly now in the CVS trunk.
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
15 (Editing support is mostly orthogonal to the internal representation.)
88740
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
16
88884
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
17 * SINGLE_BYTE_CHAR_P returns true for Latin-1 characters, which has
89190
9693e41cc2fd *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88979
diff changeset
18 undesirable effects. E.g.:
9693e41cc2fd *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88979
diff changeset
19 (multibyte-string-p (let ((s "x")) (aset s 0 ?£) s)) => nil
9693e41cc2fd *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88979
diff changeset
20 (multibyte-string-p (concat [?£])) => nil
9693e41cc2fd *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88979
diff changeset
21 (text-char-description ?£) => "M-#"
88740
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
22
89215
3b4e014cc159 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 89190
diff changeset
23 These examples are all fixed by the change of 2002-10-14, but
3b4e014cc159 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 89190
diff changeset
24 there still exist questionalble SINGLE_BYTE_CHAR_P in the
3b4e014cc159 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 89190
diff changeset
25 code.
3b4e014cc159 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 89190
diff changeset
26
88740
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
27 * Rationalize character syntax and its relationship to the Unicode
89190
9693e41cc2fd *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88979
diff changeset
28 database. (Applies mainly to symbol an punctuation syntax.)
88740
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
29
88884
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
30 * Fontset handling and customization needs work. We want to relate
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
31 fonts to scripts, probably based on the Unicode blocks. The
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
32 presence of small-repertoire 10646-encoded fonts in XFree 4 is a
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
33 pain, not currently worked round.
88740
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
34
88979
52c1682e6353 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88891
diff changeset
35 With the change on 2002-07-26, multiple fonts can be
52c1682e6353 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88891
diff changeset
36 specified in a fontset for a specific range of characters.
52c1682e6353 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88891
diff changeset
37 Each range can also be specified by script. Before using
52c1682e6353 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88891
diff changeset
38 ISO10646 fonts, Emacs checks their repertories to avoid such
52c1682e6353 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88891
diff changeset
39 fonts that don't have a glyph for a specific character.
88891
1c1bc8902236 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88884
diff changeset
40
88884
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
41 * Work is also needed on charset and coding system priorities.
88740
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
42
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
43 * The relevant bits of latin1-disp.el need porting (and probably
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
44 re-naming/updating). See also cyril-util.el.
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
45
88884
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
46 * Quail files need more work now the encoding is irrelevant.
88740
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
47
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
48 * What to do with the old coding categories stuff?
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
49
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
50 * The preferred-coding-system property of charsets should probably be
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
51 junked unless it can be made more useful now.
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
52
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
53 * find-multibyte-characters needs looking at.
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
54
89190
9693e41cc2fd *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88979
diff changeset
55 * Implement Korean cp949/UHC, BIG5-HKSCS and any other important missing
88740
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
56 charsets.
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
57
89190
9693e41cc2fd *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88979
diff changeset
58 * Check up on definition of alternativnj.
88740
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
59
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
60 * Lazy-load tables for unify-charset somehow?
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
61
88979
52c1682e6353 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88891
diff changeset
62 Actually, Emacs clear out all charset maps and unify-map just
52c1682e6353 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88891
diff changeset
63 before dumping, and their are loaded again on demand the
52c1682e6353 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88891
diff changeset
64 dumped emacs. But, those maps (char tables) generated while
52c1682e6353 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88891
diff changeset
65 temacs is running can't be get rid of from the dumped emacs.
52c1682e6353 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88891
diff changeset
66
88836
b53799fa4747 *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88763
diff changeset
67 * Translation tables for {en,de}code currently aren't supported.
88740
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
68
89215
3b4e014cc159 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 89190
diff changeset
69 This should be fixed by the changes of 2002-10-14.
3b4e014cc159 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 89190
diff changeset
70
88740
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
71 * Defining CCL coding systems currently doesn't work.
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
72
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
73 * iso-2022 charsets get unified on i/o.
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
74
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
75 * Revisit locale processing: look at treating the language and
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
76 charset parts separately. (Language should affect things like
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
77 speling and calendar, but that's not a Unicode issue.)
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
78
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
79 * Handle Unicode combining characters usefully, e.g. diacritics, and
88884
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
80 handle more scripts specifically (à la Devanagari). There are
88740
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
81 issues with canonicalization.
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
82
88836
b53799fa4747 *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88763
diff changeset
83 * Bidi is a separate issue with no support currently.
88740
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
84
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
85 * We need tabular input methods, e.g. for maths symbols. (Not
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
86 specific to Unicode.)
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
87
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
88 * Need multibyte text in menus, e.g. for the above. (Not specific to
62f922836c9b *** empty log message ***
Dave Love <fx@gnu.org>
parents:
diff changeset
89 Unicode.)
88836
b53799fa4747 *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88763
diff changeset
90
b53799fa4747 *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88763
diff changeset
91 * There's currently no support for Unicode normalization.
b53799fa4747 *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88763
diff changeset
92
88884
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
93 * Populate char-width-table correctly for Unicode chanaracters and
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
94 worry about what happens when double-width charsets covering
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
95 non-CJK characters are unified.
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
96
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
97 * Emacs 20/21 .elc files are currently not loadable. It may or may
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
98 not be possible to do this properly.
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
99
88891
1c1bc8902236 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88884
diff changeset
100 With the change on 2002-07-24, elc files generated by Emacs
1c1bc8902236 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88884
diff changeset
101 20.3 and later are correctly loaded (including those
1c1bc8902236 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88884
diff changeset
102 containing multibyte characters and compressed). But, elc
1c1bc8902236 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88884
diff changeset
103 files generated by 20.2 and the primer are still not loadable.
1c1bc8902236 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88884
diff changeset
104 Is it really worth working on it?
1c1bc8902236 *** empty log message ***
Kenichi Handa <handa@m17n.org>
parents: 88884
diff changeset
105
89248
d2c6faa9211a *** empty log message ***
Dave Love <fx@gnu.org>
parents: 89231
diff changeset
106 * Rmail won't work with non-ASCII text. Encoding issues for Babyl
d2c6faa9211a *** empty log message ***
Dave Love <fx@gnu.org>
parents: 89231
diff changeset
107 files need sorting out, but rms says Babyl will go before this is
d2c6faa9211a *** empty log message ***
Dave Love <fx@gnu.org>
parents: 89231
diff changeset
108 released.
88884
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
109
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
110 * Gnus still needs some attention, and we need to get changes
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
111 accepted by Gnus maintainers...
9d76d33fbcff *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88836
diff changeset
112
89248
d2c6faa9211a *** empty log message ***
Dave Love <fx@gnu.org>
parents: 89231
diff changeset
113 * There are type errors lurking, e.g. in
d2c6faa9211a *** empty log message ***
Dave Love <fx@gnu.org>
parents: 89231
diff changeset
114 Fcheck_coding_systems_region. Define ENABLE_CHECKING to find them.
d2c6faa9211a *** empty log message ***
Dave Love <fx@gnu.org>
parents: 89231
diff changeset
115
88836
b53799fa4747 *** empty log message ***
Dave Love <fx@gnu.org>
parents: 88763
diff changeset
116 * You can grep the code for lots of fixmes.