[SOLVED] Replacing a non-ascii chars in UTF8 string

[alpine]

Hi,

Is anyone figured out an easy way to replace non-ascii chars in a UTF8 string with a translation table?

The problem is when the user enters a license plate number into an edit box, it can look same as the real number but written in Cyrillic characters. In Cyrillic alphabet there is many letters that look as Latin ones. When such a wrong number enters into the database, it can't be found easily.

The obvious way should be to replace all similar looking chars with Latin ones before processing, but I can't find a simple way of doing that.

[winni]

Hi!

The UTF8 chars are grouped.

You should get the value of the UTF8char and look if it is member of the latin groups or the cyrilic groups.

The values:

Teh basic groups are

The Latin Groups U+0000 .. U+2CF7
The Cyrillic Groups U+0400 .. U+052F

There are some more.

Have a look at https://www.utf8-chartable.de/unicode-utf8-table.pl

Winni

[Thaddy]

It would be best if the database field would be defined as UTF8. Then it will always store correctly. This also how most database string fields are defined, nowadays.

[SymbolicFrank]

It would be best if the database field would be defined as UTF8. Then it will always store correctly. This also how most database string fields are defined, nowadays.

Well, yes, but only if you type the same char when searching for the value. The storage isn't the problem.

[alpine]

It would be best if the database field would be defined as UTF8. Then it will always store correctly. This also how most database string fields are defined, nowadays.

It is defined as such.

The trouble is at another place - when the users enters data, usually the keyboard is switched to Cyrillic and because of similarity of the letters it is not obvious what kind of letters are there. The next time the keyboard can be switched to Latin - and the same plate number can look exactly the same, but with Latin letters.

FYI: АВЕКМНОРСТУХ are shared between Cyrillic/Latin, but they of course have different codes 

UTF8 field can't resolve that issue, the chars must be replaced with a help of translation table:
 АВЕКМНОРСТУХ -> ABEKMHOPCTYX

[SymbolicFrank]

Letter E

Note the letters that look like 'H' and such.

I just checked and the Cyrillic "E" isn't even in the picture. There are far too many E's to fit the box.

[wp]

Using the string iterator of unit LazUnicode it is easy to run through the UTF8 codepoints of an input string and replace them as needed.

Code: Pascal  [Select][+][-]

1. uses
2.   LazUnicode;
3.  
4. function CheckAndReplace(const AText: String): String;
5. var
6.   cyrillicCh, latinCh: String;
7. begin
8.   Result := '';
9.   for cyrillicCh in AText do
10.   begin
11.     case cyrillicCh of
12.       'А': latinCh := 'A';
13.       'В': latinCh := 'B';
14.       'е': latinCh := 'e';
15.       'І': latinCh := 'I';
16.       'Ѕ': latinCh := 'S';
17.       'О': latinCh := 'O';
18.       'Р': latinCh := 'P';
19.       'Т': latinCh := 'T';
20.       'С': latinCh := 'C';
21.       'М': latinCh := 'M';
22.       'Н': latinCh := 'H';
23.       'а': latinCh := 'a';
24.       'о': latinCh := 'o';
25.       'р': latinCh := 'p';
26.       'с': latinCh := 'c';
27.       // add more...
28.       else latinCh := cyrillicCh;  
29.     end;
30.     Result := Result + LatinCh;
31.   end;
32. end;

[alpine]

Using the string iterator of unit LazUnicode it is easy to run through the UTF8 codepoints of an input string and replace them as needed.

That's exactly what I've meant. Thanks a lot!