How to auto detect text file encoding?

Posted by ??? on Super User See other posts from Super User or by ???
Published on 2011-06-24T08:07:02Z Indexed on 2011/06/24 8:24 UTC
Read the original article Hit count: 153

Filed under:

linux

|

batch

|

encoding

|

detection

There are many plain text files which were encoded in variant charsets.

I want to convert them all to UTF-8, but before running iconv, I need to know its original encoding. Most browsers have an Auto Detect option in encodings, however, I can't check those text files one by one because there are too many.

Only having known the original encoding, I then can convert the texts by iconv -f DETECTED_CHARSET -t utf-8.

Is there any utility to detect the encoding of plain text files? It doesn't have to be a 100% perfect correct, but it should recognize most of them.

© Super User or respective owner

Related posts about linux

apt-get install and update fail

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I've got a problem with apt-get update and apt-get install ... commands . every time update or installing fails and errors are : Get:1 http://dl.google.com stable Release.gpg [198B] Ign http://dl.google.com/linux/chrome/deb/ stable/main Translation-en_US Get:2 http://dl… >>> More
kernel module compiling error

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
sh@ubuntu:/home/ccpp/helloworld$ make gcc-4.6 -O2 -DMODULE -D_KERNEL_ -W -Wall -Wstrict-prototypes -Wmissing-prototypes -isystem /lib/modules/`uname -r`/build/include -c -o hello-1.o hello-1.c hello-1.c:4:0: warning: "MODULE" redefined [enabled by default] <command-line>:0:0: note: this is… >>> More
Build-Essentials installation failing

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I am having trouble accessing the several critical header files that show to be a part of the build process. The "Ubuntu Software Center" shows "Build Essentials" as installed: Next I did the following two commands, which did not improve the problem: ~$ sudo apt-get install build-essential [sudo]… >>> More
Updating Debian kernel

as seen on Super User - Search for 'Super User'
I'm trying to update my Debian machine to 2.6.32-46 (which is the new stable). However, after doing apt-get update my apt-cache search linux-image shows me: linux-headers-2.6.32-5-486 - Header files for Linux 2.6.32-5-486 linux-headers-2.6.32-5-686-bigmem - Header files for Linux 2.6.32-5-686-bigmem linux-headers-2… >>> More
Serial connection over a single USB cable (Windows to linux, or linux to linux)

as seen on Server Fault - Search for 'Server Fault'
I'm helping out with a project for an embedded device that only has USB and no serial. This device is running Linux. These days, when we need to connect to a serial port on a device we typically use a USB to serial adapter (on something like a phone system or a load balancing device, etc). I would… >>> More

Related posts about batch

Running Java from a Windows batch file causes the batch file to stop

as seen on Super User - Search for 'Super User'
When I run Java from a Windows .cmd file (Vista 32-bit here), the Java command causes the batch file to stop executing additional commands. For example, this is a simple test.cmd file: java java This should cause Java to print its help message twice. However when I run it in cmd.exe, I get this: C:\>test C:\>java Usage:… >>> More
Batch break up after call other batch file

as seen on Stack Overflow - Search for 'Stack Overflow'
i have the problem, that my batch process breaking up each time after call a other batch file. The batch files are used to run a make process out from IBM Rhapsody. There convert the call from Rhapsody to the Visual Studio tools. So nmake will be called from the batch after make different settings… >>> More
Execute batch file after another batch file completes

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a batch file coordinating the activity of two other batch file. How can I make it execute the second batch file only after the first bat file finishes execution. I do not care if the first batch file executes successfully or not. >>> More
“Query cost (relative to the batch)” <> Query cost relative to batch

as seen on SQL Blogcasts - Search for 'SQL Blogcasts'
OK, so that is quite a contradictory title, but unfortunately it is true that a common misconception is that the query with the highest percentage relative to batch is the worst performing. Simply put, it is a lie, or more accurately we dont understand what these figures mean. Consider the… >>> More
Framework 4 Features: Support for Timed Jobs

as seen on Oracle Blogs - Search for 'Oracle Blogs'
One of the new features of the Oracle Utilities Application Framework V4 is the ability for the batch framework to support Timed Batch. Traditionally batch is associated with set processing in the background in a fixed time frame. For example, billing customers. Over the last few versions their has… >>> More