Unicode replacement characters for text matching

Posted by Christian Harms on Stack Overflow See other posts from Stack Overflow or by Christian Harms
Published on 2010-06-06T21:08:12Z Indexed on 2010/06/06 21:12 UTC
Read the original article Hit count: 385

Filed under:

unicode

|

special-characters

I have some fun with unicode text sources (all correct encodet) and I want to match names. The classic problem, one source comes correctly, an other has more flatten names:

"Elblag" vs. "Elblag" (see the character a)

How can I "flatten" a, á, â or à to a for better matching? Are there unicode to ascii- matching tables?

© Stack Overflow or respective owner

Related posts about unicode

Translating Between Unicode and Non-Unicode Character Sets in Java

as seen on Internet.com - Search for 'Internet.com'
You can use Java APIs not only to help translate characters, strings, and text streams to other languages, but also to convert Unicode character sets to non-Unicode and vice versa. >>> More
SQLite, python, unicode, and non-utf data

as seen on Stack Overflow - Search for 'Stack Overflow'
I started by trying to store strings in sqlite using python, and got the message: sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just… >>> More
SQLite, python, unicode, and non-utf data

as seen on Stack Overflow - Search for 'Stack Overflow'
I started by trying to store strings in sqlite using python, and got the message: sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just… >>> More
notepad sql Unicode and Non Unicode

as seen on Super User - Search for 'Super User'
Hi, I have a Microsoft Notepad flate file with data and Vertical Bar as column delimiter. I get following message: cannot convert between unicode and non-unicode string data types It seems it is my nvarchar(max) that creates my problem. I changed to varchar(max); but still the same problem. How… >>> More
On Windows 7, dir or tree can't show unicode characters, even starting cmd with cmd /U

as seen on Super User - Search for 'Super User'
On Windows 7, dir or tree can't show unicode characters, even starting cmd with cmd /U So I would press Window Key + R to run something, and type in cmd /U so that the content might handle Unicode. And then using dir or tree /F, the content in Unicode won't show as Unicode. (in Window Explorer… >>> More

Related posts about special-characters

Insert Special Characters & Coding in Online Forms in Firefox

as seen on How to geek - Search for 'How to geek'
If you are active in forums or comment areas on different websites then you most likely use some type of special characters, HTML, or other code throughout the day. Now you can easily insert commonly used “items” with the SKeys extension for Firefox. Your New Special Text Edit Bar After installing… >>> More
Insert Special Characters & Coding in Online Forms in Firefox

as seen on How to geek - Search for 'How to geek'
If you are active in forums or comment areas on different websites then you most likely use some type of special characters, HTML, or other code throughout the day. Now you can easily insert commonly used “items” with the SKeys extension for Firefox. Your New Special Text Edit Bar After installing… >>> More
removing special characters in asp

as seen on Stack Overflow - Search for 'Stack Overflow'
hi, I want to identify special characters and remove that special characters from my string or a word for example O'neil - i want to remove (') from this word. Muñoz, A. Patrick - i want to remove above character of n (ñ) similarly i want to remove all special characters from my strings. I… >>> More
Drupal node_save and special characters.

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, i'm trying to create nodes and taxonomy terms through a custom php script by using the node_save() function. I'm working on drupal 6. It's working well (thanks to previous questions on stackoverflow) except for accented letters. Indeed, when a title or a taxonomy term contain "é", "è" or… >>> More
Batch file script to remove special characters from filenames (Windows)

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a large set of files, some of which contain special characters in the filename (e.g. ä,ö,%, and others). I'd like a script file to iterate over these files and rename them removing the special characters. I don't really mind what it does, but it could replace them with underscores for example… >>> More