Explain Unicode

[NeedHelp]

Can someone explain what UNICODE is and why I should optimize my code for it? I have read in several places that your C++ code should be optimized for it but I'm not ever sure what it is or how to optimize it.

[TkTech]

Unicode is a format for encoding text. Its mainly used for non-romanized languages such as Kanji or Mandarin.

[Sionofdarkness]

I really have no understanding of anything you just said. What are Kanji and Mandarin used for? I've never heard of those languages before, I'm guessing not many people know them.

[TkTech]

I'm guessing not many people know them.

Eh.....eh.........he.....
Only about 2 billion people, since Mandarin and Cantonese are China's offical languages, and Kanji, Kana, and hiranga are Japan's natianal languages. There written as Symbols, instead of the roman alphabet.

[WingedPanther]

ASCII was the code used to transmit information when C++ was first written. You could rely on 7 bits, which meant 128 characters. 8 bits gave you 256 characters. These are nice, but not reliable. Unicode uses (I think) 2 bytes, or 16 bits to store characters, so you suddenly have 65,536 characters available, which is enough to cover most languages.

[Lop]

What would make people in america want to use unicode then? Isn't ascii good enough if your target area is america?

[brackett]

Unfortunately, as in all things that transcend to the human world, text encoding is a royal PITA. Here's a basic primer:

ASCII: 7 bit encoding for standard 128 English characters.

Extended ASCII: 8 bit encoding, with the lower 128 being standard ASCII, and various promulgations of the higher 128 bit for encoding various internation characters, drawing symbols, etc.

The great thing about ASCII is that 1 byte = 1 character. Fairly simple. The bad thing about ASCII is it turns out that there's a lot more than 256 characters once you consider other languages. So, now we have:

Unicode: standard for encoding lots and lots of characters

Unfortunately, Unicode doesn't actually define an encoding standard....which leads us to:

UTF-7: Nobody uses it, AFAIK...

UTF-8: variable bit encoding for Unicode (I think it's currently from 1 byte to 4 bytes). The bottom 128 are the same as ASCII, so all of us English centric programmers can still assume 1 byte = 1 character. Upper bytes, for other languages, are variable length.

UTF-16: variable bit encoding for Unicode. The bottom 65,536 (the basic level) are 2 bytes, the rest are 4 bytes. UTF-16 should have a byte order mark to show whether it's big endian or little endian. UTF-16LE is little endian, and UTF-16BE is big endian.

UCS-2: A subset of UTF-16 that only supports the basic level.

UTF-32: 4 byte encoding for all of Unicode.

There's some other encoding standards I'm glossing over (like EBCDIC/UTF-EBCDIC, etc.) that aren't widely used.

IIRC, here's how the OS/Environment wars played out:

Windows NT: UCS-2
Windows 2000+: UTF-16
.NET Runtime: UTF-16
Java Runtime: UTF-16
*nix: UTF-8 or ASCII

[TkTech]

Good job, does that explain your question NeedHelp? The most used is UTF-8, which youll see most programs support.