In our application, we receive text files (.txt, .csv, etc.) from diverse sources. When reading, these files sometimes contain garbage, because the files where created in a different/unknown codepage.
Is there a way to (automatically) detect the codepage of a text file?
The detectEncodingFromByteOrderMarks, on the StreamReader constructor, works for UTF8 and other unicode marked files, but I'm looking for a way to detect code pages, like ibm850, windows1252.
Thanks for your answers, this is what I've done.
The files we receive are from end-users, they do not have a clue about codepages. The receivers are also end-users, by now this is what they know about codepages: Codepages exist, and are annoying.
Solution:
This question is related to
c#
.net
text
encoding
globalization
If you're looking to detect non-UTF encodings (i.e. no BOM), you're basically down to heuristics and statistical analysis of the text. You might want to take a look at the Mozilla paper on universal charset detection (same link, with better formatting via Wayback Machine).
You can't detect the codepage
This is clearly false. Every web browser has some kind of universal charset detector to deal with pages which have no indication whatsoever of an encoding. Firefox has one. You can download the code and see how it does it. See some documentation here. Basically, it is a heuristic, but one that works really well.
Given a reasonable amount of text, it is even possible to detect the language.
Here's another one I just found using Google:
Since it basically comes down to heuristics, it may help to use the encoding of previously received files from the same source as a first hint.
Most people (or applications) do stuff in pretty much the same order every time, often on the same machine, so its quite likely that when Bob creates a .csv file and sends it to Mary it'll always be using Windows-1252 or whatever his machine defaults to.
Where possible a bit of customer training never hurts either :-)
If you can link to a C library, you can use libenca. See http://cihar.com/software/enca/. From the man page:
Enca reads given text files, or standard input when none are given, and uses knowledge about their language (must be supported by you) and a mixture of parsing, statistical analysis, guessing and black magic to determine their encodings.
It's GPL v2.
I use this code to detect Unicode and windows default ansi codepage when reading a file. For other codings a check of content is necessary, manually or by programming. This can de used to save the text with the same encoding as when it was opened. (I use VB.NET)
'Works for Default and unicode (auto detect)
Dim mystreamreader As New StreamReader(LocalFileName, Encoding.Default)
MyEditTextBox.Text = mystreamreader.ReadToEnd()
Debug.Print(mystreamreader.CurrentEncoding.CodePage) 'Autodetected encoding
mystreamreader.Close()
Got the same problem but didn't found a good solution yet for detecting it automatically . Now im using PsPad (www.pspad.com) for that ;) Works fine
Looking for different solution, I found that
https://code.google.com/p/ude/
this solution is kinda heavy.
I needed some basic encoding detection, based on 4 first bytes and probably xml charset detection - so I've took some sample source code from internet and added slightly modified version of
http://lists.w3.org/Archives/Public/www-validator/2002Aug/0084.html
written for Java.
public static Encoding DetectEncoding(byte[] fileContent)
{
if (fileContent == null)
throw new ArgumentNullException();
if (fileContent.Length < 2)
return Encoding.ASCII; // Default fallback
if (fileContent[0] == 0xff
&& fileContent[1] == 0xfe
&& (fileContent.Length < 4
|| fileContent[2] != 0
|| fileContent[3] != 0
)
)
return Encoding.Unicode;
if (fileContent[0] == 0xfe
&& fileContent[1] == 0xff
)
return Encoding.BigEndianUnicode;
if (fileContent.Length < 3)
return null;
if (fileContent[0] == 0xef && fileContent[1] == 0xbb && fileContent[2] == 0xbf)
return Encoding.UTF8;
if (fileContent[0] == 0x2b && fileContent[1] == 0x2f && fileContent[2] == 0x76)
return Encoding.UTF7;
if (fileContent.Length < 4)
return null;
if (fileContent[0] == 0xff && fileContent[1] == 0xfe && fileContent[2] == 0 && fileContent[3] == 0)
return Encoding.UTF32;
if (fileContent[0] == 0 && fileContent[1] == 0 && fileContent[2] == 0xfe && fileContent[3] == 0xff)
return Encoding.GetEncoding(12001);
String probe;
int len = fileContent.Length;
if( fileContent.Length >= 128 ) len = 128;
probe = Encoding.ASCII.GetString(fileContent, 0, len);
MatchCollection mc = Regex.Matches(probe, "^<\\?xml[^<>]*encoding[ \\t\\n\\r]?=[\\t\\n\\r]?['\"]([A-Za-z]([A-Za-z0-9._]|-)*)", RegexOptions.Singleline);
// Add '[0].Groups[1].Value' to the end to test regex
if( mc.Count == 1 && mc[0].Groups.Count >= 2 )
{
// Typically picks up 'UTF-8' string
Encoding enc = null;
try {
enc = Encoding.GetEncoding( mc[0].Groups[1].Value );
}catch (Exception ) { }
if( enc != null )
return enc;
}
return Encoding.ASCII; // Default fallback
}
It's enough to read probably first 1024 bytes from file, but I'm loading whole file.
I've done something similar in Python. Basically, you need lots of sample data from various encodings, which are broken down by a sliding two-byte window and stored in a dictionary (hash), keyed on byte-pairs providing values of lists of encodings.
Given that dictionary (hash), you take your input text and:
If you've also sampled UTF encoded texts that do not start with any BOM, the second step will cover those that slipped from the first step.
So far, it works for me (the sample data and subsequent input data are subtitles in various languages) with diminishing error rates.
I was actually looking for a generic, not programming way of detecting the file encoding, but I didn't find that yet. What I did find by testing with different encodings was that my text was UTF-7.
So where I first was doing: StreamReader file = File.OpenText(fullfilename);
I had to change it to: StreamReader file = new StreamReader(fullfilename, System.Text.Encoding.UTF7);
OpenText assumes it's UTF-8.
you can also create the StreamReader like this new StreamReader(fullfilename, true), the second parameter meaning that it should try and detect the encoding from the byteordermark of the file, but that didn't work in my case.
As addon to ITmeze post, I've used this function to convert the output of C# port for Mozilla Universal Charset Detector
private Encoding GetEncodingFromString(string codePageName)
{
try
{
return Encoding.GetEncoding(codePageName);
}
catch
{
return Encoding.ASCII;
}
}
You can't detect the codepage
This is clearly false. Every web browser has some kind of universal charset detector to deal with pages which have no indication whatsoever of an encoding. Firefox has one. You can download the code and see how it does it. See some documentation here. Basically, it is a heuristic, but one that works really well.
Given a reasonable amount of text, it is even possible to detect the language.
Here's another one I just found using Google:
Have you tried C# port for Mozilla Universal Charset Detector
Example from http://code.google.com/p/ude/
public static void Main(String[] args)
{
string filename = args[0];
using (FileStream fs = File.OpenRead(filename)) {
Ude.CharsetDetector cdet = new Ude.CharsetDetector();
cdet.Feed(fs);
cdet.DataEnd();
if (cdet.Charset != null) {
Console.WriteLine("Charset: {0}, confidence: {1}",
cdet.Charset, cdet.Confidence);
} else {
Console.WriteLine("Detection failed.");
}
}
}
The StreamReader class's constructor takes a 'detect encoding' parameter.
If you're looking to detect non-UTF encodings (i.e. no BOM), you're basically down to heuristics and statistical analysis of the text. You might want to take a look at the Mozilla paper on universal charset detection (same link, with better formatting via Wayback Machine).
Got the same problem but didn't found a good solution yet for detecting it automatically . Now im using PsPad (www.pspad.com) for that ;) Works fine
Since it basically comes down to heuristics, it may help to use the encoding of previously received files from the same source as a first hint.
Most people (or applications) do stuff in pretty much the same order every time, often on the same machine, so its quite likely that when Bob creates a .csv file and sends it to Mary it'll always be using Windows-1252 or whatever his machine defaults to.
Where possible a bit of customer training never hurts either :-)
The StreamReader class's constructor takes a 'detect encoding' parameter.
As addon to ITmeze post, I've used this function to convert the output of C# port for Mozilla Universal Charset Detector
private Encoding GetEncodingFromString(string codePageName)
{
try
{
return Encoding.GetEncoding(codePageName);
}
catch
{
return Encoding.ASCII;
}
}
The StreamReader class's constructor takes a 'detect encoding' parameter.
The tool "uchardet" does this well using character frequency distribution models for each charset. Larger files and more "typical" files have more confidence (obviously).
On ubuntu, you just apt-get install uchardet.
On other systems, get the source, usage & docs here: https://github.com/BYVoid/uchardet
I know it's very late for this question and this solution won't appeal to some (because of its english-centric bias and its lack of statistical/empirical testing), but it's worked very well for me, especially for processing uploaded CSV data:
http://www.architectshack.com/TextFileEncodingDetector.ashx
Advantages:
Note: I'm the one who wrote this class, so obviously take it with a grain of salt! :)
Thanks @Erik Aronesty for mentioning uchardet.
Meanwhile the (same?) tool exists for linux: chardet.
Or, on cygwin you may want to use: chardetect.
See: chardet man page: https://www.commandlinux.com/man-page/man1/chardetect.1.html
This will heuristically detect (guess) the character encoding for each given file and will report the name and confidence level for each file's detected character encoding.
I was actually looking for a generic, not programming way of detecting the file encoding, but I didn't find that yet. What I did find by testing with different encodings was that my text was UTF-7.
So where I first was doing: StreamReader file = File.OpenText(fullfilename);
I had to change it to: StreamReader file = new StreamReader(fullfilename, System.Text.Encoding.UTF7);
OpenText assumes it's UTF-8.
you can also create the StreamReader like this new StreamReader(fullfilename, true), the second parameter meaning that it should try and detect the encoding from the byteordermark of the file, but that didn't work in my case.
10Y (!) had passed since this was asked, and still I see no mention of MS's good, non-GPL'ed solution: IMultiLanguage2 API.
Most libraries already mentioned are based on Mozilla's UDE - and it seems reasonable that browsers have already tackled similar problems. I don't know what is chrome's solution, but since IE 5.0 MS have released theirs, and it is:
It is a native COM call, but here's some very nice work by Carsten Zeumer, that handles the interop mess for .net usage. There are some others around, but by and large this library doesn't get the attention it deserves.
I've done something similar in Python. Basically, you need lots of sample data from various encodings, which are broken down by a sliding two-byte window and stored in a dictionary (hash), keyed on byte-pairs providing values of lists of encodings.
Given that dictionary (hash), you take your input text and:
If you've also sampled UTF encoded texts that do not start with any BOM, the second step will cover those that slipped from the first step.
So far, it works for me (the sample data and subsequent input data are subtitles in various languages) with diminishing error rates.
If you can link to a C library, you can use libenca. See http://cihar.com/software/enca/. From the man page:
Enca reads given text files, or standard input when none are given, and uses knowledge about their language (must be supported by you) and a mixture of parsing, statistical analysis, guessing and black magic to determine their encodings.
It's GPL v2.
Looking for different solution, I found that
https://code.google.com/p/ude/
this solution is kinda heavy.
I needed some basic encoding detection, based on 4 first bytes and probably xml charset detection - so I've took some sample source code from internet and added slightly modified version of
http://lists.w3.org/Archives/Public/www-validator/2002Aug/0084.html
written for Java.
public static Encoding DetectEncoding(byte[] fileContent)
{
if (fileContent == null)
throw new ArgumentNullException();
if (fileContent.Length < 2)
return Encoding.ASCII; // Default fallback
if (fileContent[0] == 0xff
&& fileContent[1] == 0xfe
&& (fileContent.Length < 4
|| fileContent[2] != 0
|| fileContent[3] != 0
)
)
return Encoding.Unicode;
if (fileContent[0] == 0xfe
&& fileContent[1] == 0xff
)
return Encoding.BigEndianUnicode;
if (fileContent.Length < 3)
return null;
if (fileContent[0] == 0xef && fileContent[1] == 0xbb && fileContent[2] == 0xbf)
return Encoding.UTF8;
if (fileContent[0] == 0x2b && fileContent[1] == 0x2f && fileContent[2] == 0x76)
return Encoding.UTF7;
if (fileContent.Length < 4)
return null;
if (fileContent[0] == 0xff && fileContent[1] == 0xfe && fileContent[2] == 0 && fileContent[3] == 0)
return Encoding.UTF32;
if (fileContent[0] == 0 && fileContent[1] == 0 && fileContent[2] == 0xfe && fileContent[3] == 0xff)
return Encoding.GetEncoding(12001);
String probe;
int len = fileContent.Length;
if( fileContent.Length >= 128 ) len = 128;
probe = Encoding.ASCII.GetString(fileContent, 0, len);
MatchCollection mc = Regex.Matches(probe, "^<\\?xml[^<>]*encoding[ \\t\\n\\r]?=[\\t\\n\\r]?['\"]([A-Za-z]([A-Za-z0-9._]|-)*)", RegexOptions.Singleline);
// Add '[0].Groups[1].Value' to the end to test regex
if( mc.Count == 1 && mc[0].Groups.Count >= 2 )
{
// Typically picks up 'UTF-8' string
Encoding enc = null;
try {
enc = Encoding.GetEncoding( mc[0].Groups[1].Value );
}catch (Exception ) { }
if( enc != null )
return enc;
}
return Encoding.ASCII; // Default fallback
}
It's enough to read probably first 1024 bytes from file, but I'm loading whole file.
Got the same problem but didn't found a good solution yet for detecting it automatically . Now im using PsPad (www.pspad.com) for that ;) Works fine
Since it basically comes down to heuristics, it may help to use the encoding of previously received files from the same source as a first hint.
Most people (or applications) do stuff in pretty much the same order every time, often on the same machine, so its quite likely that when Bob creates a .csv file and sends it to Mary it'll always be using Windows-1252 or whatever his machine defaults to.
Where possible a bit of customer training never hurts either :-)
Notepad++ has this feature out-of-the-box. It also supports changing it.
I use this code to detect Unicode and windows default ansi codepage when reading a file. For other codings a check of content is necessary, manually or by programming. This can de used to save the text with the same encoding as when it was opened. (I use VB.NET)
'Works for Default and unicode (auto detect)
Dim mystreamreader As New StreamReader(LocalFileName, Encoding.Default)
MyEditTextBox.Text = mystreamreader.ReadToEnd()
Debug.Print(mystreamreader.CurrentEncoding.CodePage) 'Autodetected encoding
mystreamreader.Close()
Got the same problem but didn't found a good solution yet for detecting it automatically . Now im using PsPad (www.pspad.com) for that ;) Works fine
10Y (!) had passed since this was asked, and still I see no mention of MS's good, non-GPL'ed solution: IMultiLanguage2 API.
Most libraries already mentioned are based on Mozilla's UDE - and it seems reasonable that browsers have already tackled similar problems. I don't know what is chrome's solution, but since IE 5.0 MS have released theirs, and it is:
It is a native COM call, but here's some very nice work by Carsten Zeumer, that handles the interop mess for .net usage. There are some others around, but by and large this library doesn't get the attention it deserves.
I know it's very late for this question and this solution won't appeal to some (because of its english-centric bias and its lack of statistical/empirical testing), but it's worked very well for me, especially for processing uploaded CSV data:
http://www.architectshack.com/TextFileEncodingDetector.ashx
Advantages:
Note: I'm the one who wrote this class, so obviously take it with a grain of salt! :)
Have you tried C# port for Mozilla Universal Charset Detector
Example from http://code.google.com/p/ude/
public static void Main(String[] args)
{
string filename = args[0];
using (FileStream fs = File.OpenRead(filename)) {
Ude.CharsetDetector cdet = new Ude.CharsetDetector();
cdet.Feed(fs);
cdet.DataEnd();
if (cdet.Charset != null) {
Console.WriteLine("Charset: {0}, confidence: {1}",
cdet.Charset, cdet.Confidence);
} else {
Console.WriteLine("Detection failed.");
}
}
}
Thanks @Erik Aronesty for mentioning uchardet.
Meanwhile the (same?) tool exists for linux: chardet.
Or, on cygwin you may want to use: chardetect.
See: chardet man page: https://www.commandlinux.com/man-page/man1/chardetect.1.html
This will heuristically detect (guess) the character encoding for each given file and will report the name and confidence level for each file's detected character encoding.
Open file in AkelPad(or just copy/paste a garbled text), go to Edit -> Selection -> Recode... -> check "Autodetect".
If someone is looking for a 93.9% solution. This works for me:
public static class StreamExtension
{
/// <summary>
/// Convert the content to a string.
/// </summary>
/// <param name="stream">The stream.</param>
/// <returns></returns>
public static string ReadAsString(this Stream stream)
{
var startPosition = stream.Position;
try
{
// 1. Check for a BOM
// 2. or try with UTF-8. The most (86.3%) used encoding. Visit: http://w3techs.com/technologies/overview/character_encoding/all/
var streamReader = new StreamReader(stream, new UTF8Encoding(encoderShouldEmitUTF8Identifier: false, throwOnInvalidBytes: true), detectEncodingFromByteOrderMarks: true);
return streamReader.ReadToEnd();
}
catch (DecoderFallbackException ex)
{
stream.Position = startPosition;
// 3. The second most (6.7%) used encoding is ISO-8859-1. So use Windows-1252 (0.9%, also know as ANSI), which is a superset of ISO-8859-1.
var streamReader = new StreamReader(stream, Encoding.GetEncoding(1252));
return streamReader.ReadToEnd();
}
}
}
The tool "uchardet" does this well using character frequency distribution models for each charset. Larger files and more "typical" files have more confidence (obviously).
On ubuntu, you just apt-get install uchardet.
On other systems, get the source, usage & docs here: https://github.com/BYVoid/uchardet
Since it basically comes down to heuristics, it may help to use the encoding of previously received files from the same source as a first hint.
Most people (or applications) do stuff in pretty much the same order every time, often on the same machine, so its quite likely that when Bob creates a .csv file and sends it to Mary it'll always be using Windows-1252 or whatever his machine defaults to.
Where possible a bit of customer training never hurts either :-)
You can't detect the codepage
This is clearly false. Every web browser has some kind of universal charset detector to deal with pages which have no indication whatsoever of an encoding. Firefox has one. You can download the code and see how it does it. See some documentation here. Basically, it is a heuristic, but one that works really well.
Given a reasonable amount of text, it is even possible to detect the language.
Here's another one I just found using Google:
I've done something similar in Python. Basically, you need lots of sample data from various encodings, which are broken down by a sliding two-byte window and stored in a dictionary (hash), keyed on byte-pairs providing values of lists of encodings.
Given that dictionary (hash), you take your input text and:
If you've also sampled UTF encoded texts that do not start with any BOM, the second step will cover those that slipped from the first step.
So far, it works for me (the sample data and subsequent input data are subtitles in various languages) with diminishing error rates.
The StreamReader class's constructor takes a 'detect encoding' parameter.
If you're looking to detect non-UTF encodings (i.e. no BOM), you're basically down to heuristics and statistical analysis of the text. You might want to take a look at the Mozilla paper on universal charset detection (same link, with better formatting via Wayback Machine).
You can't detect the codepage
This is clearly false. Every web browser has some kind of universal charset detector to deal with pages which have no indication whatsoever of an encoding. Firefox has one. You can download the code and see how it does it. See some documentation here. Basically, it is a heuristic, but one that works really well.
Given a reasonable amount of text, it is even possible to detect the language.
Here's another one I just found using Google:
Notepad++ has this feature out-of-the-box. It also supports changing it.
If you're looking to detect non-UTF encodings (i.e. no BOM), you're basically down to heuristics and statistical analysis of the text. You might want to take a look at the Mozilla paper on universal charset detection (same link, with better formatting via Wayback Machine).
If someone is looking for a 93.9% solution. This works for me:
public static class StreamExtension
{
/// <summary>
/// Convert the content to a string.
/// </summary>
/// <param name="stream">The stream.</param>
/// <returns></returns>
public static string ReadAsString(this Stream stream)
{
var startPosition = stream.Position;
try
{
// 1. Check for a BOM
// 2. or try with UTF-8. The most (86.3%) used encoding. Visit: http://w3techs.com/technologies/overview/character_encoding/all/
var streamReader = new StreamReader(stream, new UTF8Encoding(encoderShouldEmitUTF8Identifier: false, throwOnInvalidBytes: true), detectEncodingFromByteOrderMarks: true);
return streamReader.ReadToEnd();
}
catch (DecoderFallbackException ex)
{
stream.Position = startPosition;
// 3. The second most (6.7%) used encoding is ISO-8859-1. So use Windows-1252 (0.9%, also know as ANSI), which is a superset of ISO-8859-1.
var streamReader = new StreamReader(stream, Encoding.GetEncoding(1252));
return streamReader.ReadToEnd();
}
}
}
Open file in AkelPad(or just copy/paste a garbled text), go to Edit -> Selection -> Recode... -> check "Autodetect".
Source: Stackoverflow.com