Umlaute as ?? under MacOS X

[uwe]

Hello everyone,

I'm having a problem with Lazarus on MacOS X for PowerPC. I'm using MacOS X 10.5.

I have written a program that connects to a MySQL Database in Windows on Lazarus 0.9.30.4 (under .2 I'm having the same problem). Now when I access the database and retrieve data I am converting it to an Ansi-String with Utf8-Decode and then use it in my program (for demonstration I just make a ShowMessage). Under Windows that works perfectly fine.

Now when I am doing the same thing on MacOS X all the Umlauts are displayed as "?". First I thought the error probably lies in the TMySQLConnection, so I tried the same thing with Zeos. But the result is the same.

Then instead of displaying the string to the screen I stored the information to a StringList and dumped it into a file, also with the same result. However if I don't use the DecodeUTF8-function but instead write the raw-information into a stringlist, dump it into a file and then reimport it into a stringlist under Windows and then use DecodeUTF8 from within the windows-program I get the result that I am expecting. If I interpret that correctly it is looking to me like if the access to the database works fine but DecodeUTF8 doesn't work correctly under MacOSX, is that right?

Can anybody explain that effect to me or does anyone have a workaround?

Thanks folks

Here's some sourcecode

var
  sl : TStringList;
  save : TSaveDialog;
  u : UTF8String;
  s : AnsiString;
begin
  if NOT DBConnection.Connected then
    exit;


  save := TSaveDialog.Create(self);
  //if not save.Execute then exit;
  //if save=NIL then exit;
  LB.Clear;
  sl := TStringList.Create;
  Query.Close;
  Query.Open;
  while NOT Query.EOF do
  begin
    sl.Add( Utf8ToSys(Query.FieldByName('NAME').AsString));
    s:= (Utf8Toansi(Utf8ToSys( Query.FieldByName('Name').AsString)));
    //s :=   'tätätü';
    ShowMessage (s);
    //ShowMessage(Query.FieldByName('Name').AsString);
    //LB.Items.Add(  ConvertEncoding(Query.FieldByName('NAME').AsString,GuessEncoding(Query.FieldByName('NAME').AsString),EncodingUTF8BOM));
    //ShowMessage ( UTF8Decode( Query.FieldByName('NAME').AsString) );
    Query.Next;
  end;
     // sl.SaveToFile(save.FileName); 

[jwdietrich]

What to you have in mind by writing Utf8Toansi(Utf8ToSys(...))?

You could try to replace Utf8ToSys() and Utf8Toansi(Utf8ToSys()) by UTF8Encode().

[uwe]

I didn't have anything particular in mind, I just tried all possible combinations of Utf8ToAnsi, Utf8ToSys, UTF8Encode, UTF8Decode and just just in case I missed something. That was more of a desparate attempt to fix things by trial-and-error because everything that should have worked didn't. So please don't consider the piece of sourcecode as a static thing.

And also don't forget that I compared everything with Windows where it worked perfectly well and just the way I intended it to, so I guess in case I have not really missed something important it must be a bug in the MacOS-X implementation.

[jwdietrich]

OK, try UTF8Decode for writing to the database and UTF8Encode for reading records back.

If this doesn't work then try to rebuild the records by using the same coding procedure on the Windows side.

[theo]

http://www.lazarusforum.de/

[uwe]

I just checked that. But it still doesn't work on MacOS X / PowerPC. I've also checked the same code on Windows, which works fine. It must be the UTF8Encode-function that for some reason doesn't work properly under MacOSX on PPC, I think I have ruled out everything else.

Here's the (now simplified) code:

var
  s : AnsiString;
begin
  if NOT DBConnection.Connected then
    exit;

  Query.Close;
  Query.Open;
  while NOT Query.EOF do
  begin
    s:= (UTF8Encode( Query.FieldByName('Name').AsString));
    ShowMessage (s);
    Query.Next;
  end;

[uwe]

Unfortunately I don't have an Intel-Mac, I'd be curious to know what happens there.

Just an idea: Could it have something to do with the Endianess of the PPC?

[jwdietrich]

Just an idea: Could it have something to do with the Endianess of the PPC?

I don't think so. In one of my applications I use this technique for XML files: UTF8Decode for writing to the XML file and UTF8Encode for reading it again. It works excellently on big endian Macs (with PPC processors), little endian Macs (with Intel processors) and with Windows machines in both directions.

Could it be that the database records in your project were written with another type of encoding by the Windows computer?

[uwe]

The database-records were written by the same application that originally was compiled with Lazarus 0.9.30.2 under Windows instead of .4.

However that cannot be the reason as Encoding works fine under Windows with the very same records.

[Bart]

On windows you use Utf8ToSys to display the string in a ShowMessage. So the strings are utf-8 encoded.
On Mac your locale probably uses utf-8 encoding by default, so converting it to Ansi makes no sense?

Have you tried just doing a ShowMessage with the string as you read it, that is without trying to decode it?

Bart

[ludob]

Can anybody explain that effect to me or does anyone have a workaround?

Let the MySQL client lib do the character conversion for you. Add

Code: [Select]

MYSQL_SET_CHARSET_NAME=UTF8to TMySQLxxConnection and forget all the decoding/encoding in lazarus.
Probably the default mysql charset on OSX is already utf8 while on windows it is the system charset. If you want to be cross platform compatible, set it to a known value with above parameter.

[uwe]

Have you tried just doing a ShowMessage with the string as you read it, that is without trying to decode it?

Yes I tried that, it looks same as in windows. Also instead of doing a showmessage I have written the raw-ansi into a file and reimportet it into a windows-project and did the conversation there, which was entirely correct. I have written the result of the conversation into a file again and reimportet it on a mac, and that also worked (it displayed correctly then). That's why I concluded it must really be the Converting function that doesn't work on mac.

Let the MySQL client lib do the character conversion for you

Very good idea, thanks for the workaround, that sounds good to me. I really hope that'll work. I will check this out this evening as soon as I get home. Question: To which property do I add in TMySQL50Connection

Code: [Select]

MYSQL_SET_CHARSET_NAME=UTF8
and does that also work with Zeos and if so how?

Thanks again.

[ludob]

Question: To which property do I add in TMySQL50Connection

I should have been clearer: TMySQL50Connection.Params.

and does that also work with Zeos and if so how?

Yes. Add to TZConnection.Properties

Code: [Select]

codepage=utf8