Using utf8toansi in Linux

[fedkad]

Using the function s2:=utf8toansi(s1); in Linux seems to have no effect. The same (UTF-8) string is returned.

The page that describes this function (https://www.freepascal.org/docs-html/rtl/system/utf8toansi.html) says:

For this function to work, a widestring manager must be installed.

On page https://www.freepascal.org/docs-html/rtl/system/setwidestringmanager.html it is said that:

On Unix and Linux, an implementation based on the C library is available (in unit cwstring).

However, I have no idea on how to use SetWideStringManager.

A working example for Linux will be highly appreciated.

[Thaddy]

It is as easy as including unit cwstrings in the uses clause.

Well, that was easy to answer..... You probably did not fully understood the documentation.
And cwstrings is unix... for windows this is not necessary.
SetWideStringManager is automatically called when cwstrings is included.

[lucamar]

Note that using cwstring links your program to the C library. If you don't want that you can instead use fpwidestring, which is native Pascal. Although it's not as "automatic" as using cwstring (the default collation has to be set manually).

Because their simplicity they are both well documented. Not much to it, really

[fedkad]

Thaddy,

The unit name is called cwstring and I did include it on top of my uses section.

However, the code: length(utf8toansi('ç')) returns 2. Or, the code: length(utf8toansi('x')) returns 5.

Similarly, the code: utf8toansi('ç')='ç' returns True. Or, the code: utf8toansi('x')='x' returns True.

Note: In the last examples 'x' is a smiling face: Unicode hex code $1F929. The forum software had problems including this character.

[Zoran]

Fedkad, you didn't understand what "Ansi" in this function means.
Utf8ToAnsi is a function which converts UTF8 encoded string to "Ansi" encoding, which here actually means to "system code page".

In Windows, there is one ansi code page (really ansi - one byte) which system has as its default one-byte encoding.
In most Linux distors, the "system code page" is utf8.

So, normally, this function has no effect in Linux, it converts utf8 to utf8.

If you have a utf8 string that you should convert to some specific code page, you should use this:

Code: Pascal  [Select][+][-]

1. function ConvertToCP1250(const S: RawByteString): RawByteString;
2. begin
3.   Result := S;
4.   SetCodePage(Result, cp_utf8, False);
5.   SetCodePage(Result, 1250, True);
6. end;
7.  

Of course, replace 1250, with the encoding you actually want.

And you should know to which encoding you want to convert, even in Windows it is bad idea to rely on "system installed" code page.

[marcov]

Zoran says what I was thinking too.

Better explain what you are trying to do. If you have a database that delivers some windows ansi encoding, then the best way is to tweak the connection or database itself to convert it on the fly to utf8 on *nix.

Doing manual conversions should be a last resort.

[fedkad]

Thanks Zoran for the clarification and the example!

Marcov: I am actually trying to find a way to convert a UTF-8 string to its "closest" ASCII version. The code I was using in Windows was something like:

Code: Pascal  [Select][+][-]

1. procedure TForm1.Button1Click(Sender: TObject);
2. type
3.   USASCIIString = type AnsiString(20127);
4. begin
5.   memo2.text := String(USASCIIString(memo1.text));
6. end;

However, this does not work in Linux, because it returns question marks for every non-ASCII character. In a way, it acts like Zoran's example with code page set to 20127. In Windows it was replacing some non-ASCII Latin-based letters to their "closest" ASCII letters, which is something I want to do in Linux also.

This is discussed in topic: https://forum.lazarus.freepascal.org/index.php/topic,45802.0.html

[Zoran]

Try with CP_ASCII, so like this:

Code: Pascal  [Select][+][-]

1. function ConvertToASCII(const S: RawByteString): RawByteString;
2. begin
3.   Result := S;
4.   SetCodePage(Result, CP_UTF8, False);
5.   SetCodePage(Result, CP_ASCII, True);
6. end;
7.  

What do you get with this?

For latin letters, it should remove accents and get you the "closest" ascii letter.
For other characters (eg. cyrilic or greek letters), it just returns question marks, though.

EDIT:
Now I see that the constant CP_ASCII is actually declared as 20127, which you tried already... 

[fedkad]

Zoran: In Windows it works exactly as you described and just as my example I mentioned above.

However, in Linux it returns question marks for everything except ASCII characters, again giving similar output as my example.

[Zoran]

Zoran: In Windows it works exactly as you described and just as my example I mentioned above.

However, in Linux it returns question marks for everything except ASCII characters, again giving similar output as my example.

Yes, I am currently on Windows, sorry.
And I see now that constant CP_ASCII is declared as 20127, which you had already tried.