Extended Unix Code

Extended Unix Code (EUC) is a multibyte character encoding system used primarily for Japanese, Korean, and simplified Chinese.

The structure of EUC is based on the ISO-2022 standard, which specifies a way to represent character sets containing a maximum of 94 characters, or 8836 (942) characters, or 830584 (943) characters, as sequences of 7-bit codes. Only ISO-2022 compliant character sets can have EUC forms. Up to four coded character sets (referred to as G0, G1, G2, and G3 or as code sets 0, 1, 2, and 3) can be represented with the EUC scheme.

G0 is almost always an ISO-646 compliant coded character set such as US-ASCII, ISO 646:KR (KS X 1003) or ISO 646:JP (the lower half of JIS X 0201) that is invoked on GL (i.e. with the most significant bit cleared). An exception from US-ASCII is that 0x5C (backslash in US-ASCII) is often used to represent a Yen sign in EUC-JP (see below) and a Won sign in EUC-KR.

To get the EUC form of an ISO-2022 character, the most significant bit of each 7-bit byte of the original ISO 2022 codes is set (by adding 128 to each of these original 7-bit codes); this allows software to easily distinguish whether a particular byte in a character string belongs to the ISO-646 code or the ISO-2022 (EUC) code.

The most commonly used EUC codes are variable-width encodings with a character belonging to G0 (ISO-646 compliant coded character set) taking one byte and a character belonging to G1 (taken by a 94x94 coded character set) represented in two bytes. The EUC-CN form of GB2312 and EUC-KR are examples of such two-byte EUC codes. EUC-JP includes characters represented by up to three bytes whereas a single character in EUC-TW can take up to four bytes.

Modern applications are more likely to use UTF-8, which supports all of the glyphs of the EUC codes, and more, and is generally more portable with fewer vendor deviations and errors.

EUC-CN

EUC-CN
EUCCN encoding
MIME / IANAGB2312
Alias(es)csGB2312
Language(s)Simplified Chinese, English, Russian
StandardGB 2312 (1980)
ClassificationExtended ASCII, Variable-width encoding, CJK encoding, EUC
ExtendsUS-ASCII
Extensions748, GBK, GB18030, x-mac-chinesesimp
Transforms / EncodesGB 2312
Succeeded byGBK, GB18030

EUC-CN[1] is the usual way to use the GB2312 standard for simplified Chinese characters. Unlike the case of Japanese, the ISO-2022 form of GB2312 is not normally used, though a variant form called HZ was sometimes used on USENET. An ASCII character is represented in its usual encoding. A character from GB 2312 is represented by two bytes in the range 0xA1 – 0xFE.

Related encoding systems

748 code

An encoding related to EUC-CN is the "748" code used in the WITS typesetting system developed by Beijing's Founder Technology (now obsoleted by its newer FITS typesetting system). The 748 code contains all of GB2312, but is not ISO 2022–compliant and therefore not a true EUC code. (It uses an 8-bit lead byte but distinguishes between a second byte with its most significant bit set and one with its most significant bit cleared, and is therefore more similar in structure to Big5 and other non–ISO 2022–compliant DBCS encoding systems.) The non-GB2312 portion of the 748 code contains traditional and Hong Kong characters and other glyphs used in newspaper typesetting.

GBK and GB18030

GBK is an extension to GB2312. It defines an extended form of the EUC-CN encoding capable of representing a larger array of CJK characters sourced largely from Unicode 1.1, including traditional Chinese characters and characters used only in Japanese. It is not, however, a true EUC code, because ASCII bytes may appear as trail bytes (and C1 bytes, not limited to the single shifts, may appear as lead or trail bytes), due to a larger encoding space being required.

Variants of GBK are implemented by Windows code page 936 (the Microsoft Windows code page for simplified Chinese), and by IBM's code page 1386.

The Unicode-based GB18030 character encoding defines an extension of GBK capable of encoding the entirety of Unicode. However, Unicode encoded as GB18030 is a variable-width encoding which may use up to four bytes per character, due to an even larger encoding space being required. Being an extension of GBK, it is a superset of EUC-CN but is not itself a true EUC code. Being a Unicode encoding, its repertoire is identical to that of other Unicode transformation formats such as UTF-8.

Others

Other EUC-CN extensions include the Mac OS Chinese Simplified script[1] (known as Code page 10008 or x-mac-chinesesimp).[2]

EUC-JP

EUC-JP
EUC-JP
MIME / IANAEUC-JP
Alias(es)Unixized JIS (UJIS), csEUCPkdFmtJapanese
Language(s)Japanese, English, Russian
ClassificationExtended ISO 646, Variable-width encoding, CJK encoding, EUC
ExtendsUS-ASCII or ISO 646:JP
Transforms / EncodesJIS X 0208, JIS X 0212, JIS X 0201
Succeeded byEUC-JISx0213
EUC-JIS-2004
Alias(es)EUC-JISx0213
Language(s)Japanese, Ainu, English, Russian
StandardJIS X 0213
ClassificationExtended ASCII, Variable-width encoding, CJK encoding, EUC
ExtendsUS-ASCII
Transforms / EncodesJIS X 0213, JIS X 0201 (Kana)
Preceded byEUC-JP

EUC-JP is a variable-width encoding used to represent the elements of three Japanese character set standards, namely JIS X 0208, JIS X 0212, and JIS X 0201. 0.1% of all web pages use EUC-JP since August 2018.[3] Other names for this encoding include Unixized JIS (or UJIS) and AT&T JIS.[4] It is called Code page 954 by IBM. Microsoft has two code page numbers for this encoding (51932 and 20932).

This encoding scheme allows the easy mixing of 7-bit ASCII and 8-bit Japanese without the need for the escape characters employed by ISO-2022-JP, which is based on the same character set standards, and without ASCII bytes appearing as trail bytes (unlike Shift JIS).

A related and partially compatible encoding, called EUC-JISx0213 or EUC-JIS-2004, encodes JIS X 0201 and JIS X 0213[5] (similarly to Shift_JISx0213, its Shift_JIS-based counterpart).

Compared to EUC-CN or EUC-KR, EUC-JP did not become as widely adopted on PC and Macintosh systems in Japan, which used Shift JIS or its extensions (Windows code page 932 on Microsoft Windows, and MacJapanese on classic Mac OS), although it became heavily used by Unix or Unix-like operating systems (except for HP-UX). Therefore, whether Japanese web sites use EUC-JP or Shift_JIS often depends on what OS the author uses.

Vendor extensions to EUC-JP were usually allocated within the individual code sets,[6] as opposed to using invalid EUC sequences (as in popular extensions of EUC-CN and EUC-KR).

Characters are encoded as follows:

  • As an EUC/ISO 2022 compliant encoding, the C0 control characters, space and DEL are represented as in ASCII.
  • A graphical character from ASCII (code set 0) is represented as its usual one-byte representation, in the range 0x21 – 0x7E. While some variants of EUC-JP encode the lower half of JIS X 0201 here, most encode ASCII,[7] including the W3C/WHATWG Encoding standard used by HTML5,[8] and so does EUC-JIS-2004.[5] While this means that 0x5C is typically mapped to Unicode as U+005C REVERSE SOLIDUS (the ASCII backslash), U+005C may be displayed as a Yen sign by certain Japanese-locale fonts, e.g. on Microsoft Windows, for compatibility with the lower half of JIS X 0201.[9][10]
  • A character from JIS X 0208 (code set 1) is represented by two bytes, both in the range 0xA1 – 0xFE. This differs from the ISO-2022-JP representation by having the high bit set. This code set may also contain vendor extensions in some EUC-JP variants. In EUC-JIS-2004, the first plane of JIS X 0213 is encoded here, which is effectively a superset of standard JIS X 0208.[5]
  • A character from the upper half of JIS X 0201 (half-width kana, code set 2) is represented by two bytes, the first being 0x8E, the second being the usual JIS X 0201 representation in the range 0xA1 – 0xDF. This set may contain IBM vendor extensions in some variants.
  • A character from JIS X 0212 (code set 3) is represented in EUC-JP by three bytes, the first being 0x8F, the following two being in the range 0xA1–0xFE, i.e. with the high bit set. In addition to standard JIS X 0212, code set 3 may also contain IBM vendor extensions (in rows 83 and 84) in some EUC-JP variants.[6] In EUC-JIS-2004, the second plane of JIS X 0213 is encoded here,[5] which does not collide with the allocated rows in standard JIS X 0212.[11] Some implementations of EUC-JIS-2004, such as the one used by Python, allow both JIS X 0212 and JIS X 0213 plane 2 characters in this set.[11]

EUC-KR

EUC-KR
EUC-KR without extensions
EUC-KR code structure
MIME / IANAEUC-KR
Alias(es)Wansung, IBM-970
Language(s)Korean, English, Russian
StandardKS X 2901 (KS C 5861)
ClassificationExtended ISO 646, Variable-width encoding, CJK encoding, EUC
ExtendsUS-ASCII or ISO 646:KR
ExtensionsMac OS Korean, IBM-949, Unified Hangul Code (Windows-949)
Transforms / EncodesKS X 1001
Succeeded byUnified Hangul Code (web standards)

EUC-KR is a variable-width encoding to represent Korean text using two coded character sets, KS X 1001 (formerly KS C 5601)[12][13] and either ISO 646:KR (KS X 1003, formerly KS C 5636) or US-ASCII, depending on variant. KS X 2901 (formerly KS C 5861) stipulates the encoding and RFC 1557 dubbed it as EUC-KR.

A character drawn from KS X 1001 (G1, code set 1) is encoded as two bytes in GR (0xA1–0xFE) and a character from KS X 1003 or US-ASCII (G0, code set 0) takes one byte in GL (0x21–0x7E).

When used with ASCII, it is called Code page 970 by IBM.[14][15] It is known as Code page 51949 by Microsoft.[16] It is usually referred to as Wansung (Korean: 완성, romanizedWanseong, lit. 'precomposed[17]') in the Republic of Korea.

A common extension of EUC-KR is the Unified Hangul Code (통합형 한글 코드[18], Tonghabhyeong Hangeul Kodeu, or 통합 완성형, Tonghab Wansunghyung), which is the default Korean codepage on Microsoft Windows (code page 949, numbered 1363 by IBM). The W3C/WHATWG Encoding Standard used by HTML5 incorporates the Unified Hangul Code extensions into its definition of EUC-KR.[19] Other EUC-KR compatible extensions include the Mac OS Korean encoding, used by the classic Mac OS. IBM's code page 949 is yet another, unrelated, EUC-KR extension. Similarly to the EUC-CN extensions described above, these extensions do not conform to the EUC structure.

As of March 2019, 0.2% of all web pages use EUC-KR.[3] Including extensions, it is the most widely used legacy character encoding in Korea on all three major platforms (Unix-like OS, Windows and macOS), but its use has been very slowly decreasing as UTF-8 gains popularity, especially on Linux and macOS.

As with most other encodings, UTF-8 is now preferred for new use, solving problems with consistency between platforms and vendors.

EUC-TW

EUC-TW is a variable-width encoding that supports US-ASCII and 16 planes of CNS 11643, each of which is 94x94. It is a rarely used encoding for traditional Chinese characters as used in Taiwan. Big5 is much more common.

  • As an EUC/ISO 2022 encoding, the C0 control characters, ASCII space and DEL are encoded as in ASCII.
  • A graphical character from US-ASCII (G0, code set 0) is encoded in GL as its usual single byte representation (0x21–0x7E).
  • A character from CNS 11643 plane 1 (code set 1) is encoded as two bytes in GR (0xA1–0xFE).
  • A character in plane 1 through 16 of CNS 11643 (code set 2) is encoded as four bytes:
    • The first byte is always 0x8E (Single Shift 2).
    • The second byte (0xA1–0xB0) indicates the plane, the number of which is obtained by subtracting 0xA0 from that byte.
    • The third and fourth bytes are in GR (0xA1–0xFE).

Note that the plane 1 of CNS 11643 is encoded twice as code set 1 and a part of code set 2.

UTF-8 is becoming more common than EUC-TW, as with most code pages.

Packed versus fixed length form

The encodings described above (using bytes in 0x21–0x7E for code set 0, bytes in 0xA1–0xFE for code set 1, 0x8E followed by bytes in 0xA1–0xFE for code set 2 and 0x8F followed by bytes in 0xA1–0xFE for code set 3) are in a variable-width form referred to as the EUC packed format. This is the form usually labelled as EUC.[4]

Internal processing may make use of a fixed-length alternative form called the EUC complete two-byte format. This represents:[4]

  • Code set 0 as two bytes in the range 0x21–0x7E (except that the first may be 0x00).
  • Code set 1 as two bytes in the range 0xA0–0xFF (except that the first may be 0x80).
  • Code set 2 as a byte in the range 0x20–0x7E (or 0x00) followed by a byte in the range 0xA0–0xFF.
  • Code set 3 as a byte in the range 0xA0–0xFF (or 0x80) followed by a byte in the range 0x21–0x7E.

Initial bytes of 0x00 and 0x80 are used in cases where the code set uses only one byte. There is also a four-byte fixed length format.[4] These fixed length forms are suited to internal processing and are not usually encountered in interchange.

EUC-JP is registered with the IANA in both formats, the packed format as "EUC-JP" or "csEUCPkdFmtJapanese" and the fixed width format as "csEUCFixWidJapanese".[20] Only the packed format is included in the WHATWG Encoding Standard used by HTML5.[21]

See also

References

  1. ^ a b "Map (external version) from Mac OS Chinese Simplified encoding to Unicode 3.0 and later". Apple, Inc.
  2. ^ "Encoding.WindowsCodePage Property - .NET Framework (current version)". MSDN. Microsoft.
  3. ^ a b "Historical trends in the usage of character encodings for websites". W3Techs.
  4. ^ a b c d Lunde, Ken (2008). CJKV Information Processing: Chinese, Japanese, Korean, and Vietnamese Computing. O'Reilly. pp. 242–244. ISBN 9780596800925.
  5. ^ a b c d "JIS X 0213 Code Mapping Tables". x0213.org.
  6. ^ a b "4.2 Review Process of Rules for Code Set Conversion Between eucJP-open and UCS". Problems and Solutions for Unicode and User/Vendor Defined Characters. The Open Group Japan. Archived from the original on 1999-02-03.
  7. ^ "Ambiguities in conversion from Japanese EUC to Unicode (Non-Normative)". XML Japanese Profile. W3C.
  8. ^ "EUC-JP decoder". Encoding Standard. WHATWG. "If byte is an ASCII byte, return a code point whose value is byte."
  9. ^ "3.1.1 Details of Problems". Problems and Solutions for Unicode and User/Vendor Defined Characters. The Open Group Japan. Archived from the original on 1999-02-03.
  10. ^ Kaplan, Michael S. (2005-09-17). "When is a backslash not a backslash?".
  11. ^ a b Chang, Hyeshik. "Readme for CJKCodecs". cPython. Python Software Foundation.
  12. ^ "KS X 1001:1992" (PDF).
  13. ^ "KS C 5601:1987" (PDF). 1988-10-01.
  14. ^ "CCSID 970". IBM Globalization. IBM.
  15. ^ "ibm-970_P110_P110-2006_U2 (alias euc-kr)". Converter Explorer - ICU Demonstration. International Components for Unicode.
  16. ^ "Code Page Identifiers". Windows Dev Center. Microsoft.
  17. ^ Lunde, Ken (2009). "Chapter 3: Character Set Standards". CJKV Information Processing. p. 146. ISBN 0596514476.
  18. ^ "한글 코드에 대하여" (in Korean). W3C.
  19. ^ "5. Indexes (§ index EUC-KR)", Encoding Standard, WHATWG
  20. ^ "Character Sets". IANA.
  21. ^ "4.2. Names and labels". Encoding Standard. WHATWG.

External links

C0 and C1 control codes

The C0 and C1 control code or control character sets define control codes for use in text by computer systems that use the ISO/IEC 2022 system of specifying control and graphic characters. Most character encodings, in addition to representing printable characters, also have characters such as these that represent additional information about the text, such as the position of a cursor, an instruction to start a new line, or a message that the text has been received.

The C0 set defines codes in the range 00HEX–1FHEX and the C1 set defines codes in the range 80HEX–9FHEX. The default C0 set was originally defined in ISO 646 (ASCII), while the default C1 set was originally defined in ECMA-48 (harmonized later with ISO 6429). While other C0 and C1 sets are available for specialized applications, they are rarely used.

CNS 11643

The CNS 11643 character set (Chinese National Standard 11643), also officially known as the "Chinese Standard Interchange Code" (中文標準交換碼), is officially the standard character set of the Republic of China.

(In practice, variants of Big5 are de facto standard.)

CNS 11643 is a superset of ASCII designed to conform to ISO 2022.

It contains 16 planes, so the maximum possible number of encodable characters is 16×94×94 = 141376.

Planes 12 to 15 (35344 code points) are specifically designated for user-defined characters.

Unlike CCCII, the encoding of variant characters in CNS 11643 is not related.

EUC-TW is a representation of CNS 11643 in Extended Unix Code (EUC) form.

DBCS

A double-byte character set (DBCS) is a character encoding in which either all characters (including control characters) are encoded in two bytes, or merely every graphic character not representable by an accompanying single-byte character set (SBCS) is encoded in two bytes (Han characters would generally comprise most of these two-byte characters). A DBCS supports national languages that contain a large number of unique characters or symbols (the maximum number of characters that can be represented with one byte is 256 characters, while two bytes can represent up to 65,536 characters). Examples of such languages include Japanese and Chinese. Korean Hangul does not contain as many characters, but KS X 1001 supports both Hangul and Hanja, and uses two bytes per character.

Extended ASCII

Extended ASCII (EASCII or high ASCII) character encodings are eight-bit or larger encodings that include the standard seven-bit ASCII characters, plus additional characters. Using the term "extended ASCII" on its own is sometimes criticized, because it can be mistakenly interpreted to mean that the ASCII standard has been updated to include more than 128 characters or that the term unambiguously identifies a single encoding, neither of which is the case.

There are many extended ASCII encodings (more than 220 DOS and Windows codepages). EBCDIC ("the other" major 8-bit character code) likewise developed many extended variants (more than 186 EBCDIC codepages) over the decades.

Index of Japan-related articles (E)

This page lists Japan-related articles with romanized titles beginning with the letter E. For names of people, please list by surname (i.e., "Tarō Yamada" should be listed under "Y", not "T"). Please also ignore particles (e.g. "a", "an", "the") when listing articles (i.e., "A City with No People" should be listed under "City").

JIS X 0201

JIS X 0201, a Japanese Industrial Standard developed in 1969 (then called JIS C 6220 until the JIS category reform), was the first Japanese electronic character set to become widely used. It is either 7-bit encoding or 8-bit encoding, although 8-bit encoding is dominant for modern use. The full name of this standard is 7-bit and 8-bit coded character sets for information interchange (7ビット及び8ビットの情報交換用符号化文字集合).

The first 96 codes comprise an ISO 646 variant, mostly following ASCII with some differences, while the second 96 character codes represent the phonetic Japanese katakana signs. Since the encoding does not provide any way to express hiragana or kanji, it is only capable of expressing simplified written Japanese. Nevertheless, it is possible to express, at least phonetically, the full range of sounds in the language. In the 1980s, this was acceptable for media such as text mode computer terminals, telegrams, receipts or other electronically handled data.

JIS X 0201 was supplanted by subsequent encodings such as Shift JIS (which combines this standard and JIS X 0208) and later Unicode.

JIS X 0208

JIS X 0208 is a 2-byte character set specified as a Japanese Industrial Standard, containing 6879 graphic characters suitable for writing text, place names, personal names, and so forth in the Japanese language. The official title of the current standard is 7-bit and 8-bit double byte coded KANJI sets for information interchange (7ビット及び8ビットの2バイト情報交換用符号化漢字集合, Nana-Bitto Oyobi Hachi-Bitto no Ni-Baito Jōhō Kōkan'yō Fugōka Kanji Shūgō). It was originally established as JIS C 6226 in 1978, and has been revised in 1983, 1990, and 1997. It is also called Code page 952 by IBM. The 1978 version is also called Code page 955 by IBM.

Japanese language and computers

In relation to the Japanese language and computers many adaptation issues arise, some unique to Japanese and others common to languages which have a very large number of characters. The number of characters needed in order to write English is very small, and thus it is possible to use only one byte (28=256 possible values) to encode one English character. However, the number of characters in Japanese is much more than 256 and thus cannot be encoded using a single byte - Japanese is thus encoded using two or more bytes, in a so-called "double byte" or "multi-byte" encoding. Problems that arise relate to transliteration and romanization, character encoding, and input of Japanese text.

List of computing and IT abbreviations

This is a list of computing and IT acronyms and abbreviations.

Variable-width encoding

A variable-width encoding is a type of character encoding scheme in which codes of differing lengths are used to encode a character set (a repertoire of symbols) for representation in a computer. Most common variable-width encodings are multibyte encodings, which use varying numbers of bytes (octets) to encode different characters.

(Some authors, notably in Microsoft documentation, use the term multibyte character set, which is a misnomer, because representation size is an attribute of the encoding, not of the character set.)

Early variable width encodings using less than a byte per character were sometimes used to pack English text into fewer bytes in adventure games for early microcomputers. However disks (which unlike tapes allowed random access allowing text to be loaded on demand), increases in computer memory and general purpose compression algorithms have rendered such tricks largely obsolete.

Multibyte encodings are usually the result of a need to increase the number of characters which can be encoded without breaking backward compatibility with an existing constraint. For example, with one byte (8 bits) per character, one can encode 256 possible characters; in order to encode more than 256 characters, the obvious choice would be to use two or more bytes per encoding unit, two bytes (16 bits) would allow 65,536 possible characters, but such a change would break compatibility with existing systems and therefore might not be feasible at all.

Early telecommunications
ISO/IEC 8859
Bibliographic use
National standards
EUC
ISO/IEC 2022
MacOS code pages("scripts")
DOS code pages
IBM AIX code pages
IBM Apple MacIntoshemulations
IBM Adobe emulations
IBM DEC emulations
IBM HP emulations
Windows code pages
EBCDIC code pages
Platform specific
Unicode / ISO/IEC 10646
TeX typesetting system
Miscellaneous code pages
Related topics

This page is based on a Wikipedia article written by authors (here).
Text is available under the CC BY-SA 3.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.