SyntaxFix

[unicode] Unicode, UTF, ASCII, ANSI format differences

Going down your list:

  • "Unicode" isn't an encoding, although unfortunately, a lot of documentation imprecisely uses it to refer to whichever Unicode encoding that particular system uses by default. On Windows and Java, this often means UTF-16; in many other places, it means UTF-8. Properly, Unicode refers to the abstract character set itself, not to any particular encoding.
  • UTF-16: 2 bytes per "code unit". This is the native format of strings in .NET, and generally in Windows and Java. Values outside the Basic Multilingual Plane (BMP) are encoded as surrogate pairs. These used to be relatively rarely used, but now many consumer applications will need to be aware of non-BMP characters in order to support emojis.
  • UTF-8: Variable length encoding, 1-4 bytes per code point. ASCII values are encoded as ASCII using 1 byte.
  • UTF-7: Usually used for mail encoding. Chances are if you think you need it and you're not doing mail, you're wrong. (That's just my experience of people posting in newsgroups etc - outside mail, it's really not widely used at all.)
  • UTF-32: Fixed width encoding using 4 bytes per code point. This isn't very efficient, but makes life easier outside the BMP. I have a .NET Utf32String class as part of my MiscUtil library, should you ever want it. (It's not been very thoroughly tested, mind you.)
  • ASCII: Single byte encoding only using the bottom 7 bits. (Unicode code points 0-127.) No accents etc.
  • ANSI: There's no one fixed ANSI encoding - there are lots of them. Usually when people say "ANSI" they mean "the default locale/codepage for my system" which is obtained via Encoding.Default, and is often Windows-1252 but can be other locales.

There's more on my Unicode page and tips for debugging Unicode problems.

The other big resource of code is unicode.org which contains more information than you'll ever be able to work your way through - possibly the most useful bit is the code charts.

Examples related to unicode

• How to resolve TypeError: can only concatenate str (not "int") to str • (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape • UnicodeEncodeError: 'ascii' codec can't encode character at special name • Python NLTK: SyntaxError: Non-ASCII character '\xc3' in file (Sentiment Analysis -NLP) • HTML for the Pause symbol in audio and video control • Javascript: Unicode string to hex • Concrete Javascript Regex for Accented Characters (Diacritics) • Replace non-ASCII characters with a single space • UTF-8 in Windows 7 CMD • NameError: global name 'unicode' is not defined - in Python 3

Examples related to character-encoding

• Changing PowerShell's default output encoding to UTF-8 • JsonParseException : Illegal unquoted character ((CTRL-CHAR, code 10) • Change the encoding of a file in Visual Studio Code • What is the difference between utf8mb4 and utf8 charsets in MySQL? • How to open html file? • All inclusive Charset to avoid "java.nio.charset.MalformedInputException: Input length = 1"? • UTF-8 output from PowerShell • ERROR 1115 (42000): Unknown character set: 'utf8mb4' • "for line in..." results in UnicodeDecodeError: 'utf-8' codec can't decode byte • How to make php display \t \n as tab and new line instead of characters

Examples related to ascii

• Detect whether a Python string is a number or a letter • Is there any ASCII character for <br>? • UnicodeEncodeError: 'ascii' codec can't encode character at special name • Replace non-ASCII characters with a single space • Convert ascii value to char • What's the difference between ASCII and Unicode? • Invisible characters - ASCII • How To Convert A Number To an ASCII Character? • Convert ascii char[] to hexadecimal char[] in C • Convert character to ASCII numeric value in java

Examples related to ansi

• What is ANSI format? • Unicode, UTF, ASCII, ANSI format differences

Examples related to utf

• How to convert php array to utf8? • Which encoding opens CSV files correctly with Excel on both Mac and Windows? • Difference between UTF-8 and UTF-16? • Unicode, UTF, ASCII, ANSI format differences • UTF-8, UTF-16, and UTF-32