Issues with new strings of FPC trunk

[wp]

Since a test case fails when fspreadsheet is used with fpc trunk (in contrast to fpc 2.6.4) I am trying to understand the new strings introduced by fpc 3 and came across the following issue which is demonstrated in this program: it writes a string with a German umlaut to a console (on Windows, did not check Linux):

Code: [Select]

program project1;

{$mode objfpc}{$H+}

uses
  {$IFDEF UNIX}{$IFDEF UseCThreads}
  cthreads,
  {$ENDIF}{$ENDIF}
  Classes, SysUtils, LazUTF8
  { you can add units after this };
var
  s: String;
begin
  s := 'Ändern';
  WriteLn(s);
  WriteLn('UTF8ToConsole = ', UTF8ToConsole(s));
  WriteLn('UTF8ToAnsi    = ', UTF8ToAnsi(s));
  ReadLn;
end.

• Using fpc 2.6.4, the umlaut is not displayed correctly, as expected. If I apply UTF8ToConsole, it is correct, and if I apply UTF8ToAnsi it is wrong again (--> screenshot "fpc264.png")

• If I repeat the same with fpc 3.1.1 (no "EnableUTF8RTL") then the output of UTF8ToAnsi is correct, UTF8ToConsole is wrong and the non-modified string is wrong also (--> screenshot "fpc311-no-utf8rtl.png")

• If I enable the UTF8RTL for fpc 3.1.1 then the non-modified string is correct, and both converted strings are wrong (--> screenshot "fpc311-utf8rtl.png")

The behavior with fpc 2.6.4 and with 3.1.1/unmodified strings is reasonable. But the result of UTF8ToConsole and UTF8ToAnsi is very confusing. Is this a bug?

[EDIT] By incidence the screenshots were not selected correctly and were removed for this reason. Please find the correct images a few posts below.

[ChrisF]

Have you also used the -FcUTF8 option for your compilation with -dEnableUTF8RTL (it's not clear if you've enabled it "manually" or "automatically" using the "Set UTF-8 in RTL" button) ?

Anyhow, I can't see any difference between your 3 captures: 2.6.4, utft8rtl and no-utf8rtl. Is it normal? The umlaut is present and correctly displayed in all UTF8ToAnsi cases...


**edit** Duplicate post with taazz.

[taazz]

yes it is a reported misbehavior of the enableutf8rtl reported by ChrisF http://forum.lazarus.freepascal.org/index.php/topic,28891.0.html. By the way all three screenshots show the same results. Only UTF8ToAnsi show the correct results (I assume, I do not have any experience with umlauts).

[wp]

Sorry for the confusion - I was in a hurry and mixed up the too-similar images. And the description of the UTFRTL case is not correct either: both unmodified and UTF8ToAnsi strings are displayed correctly, UTF8ToAnsi is wrong.

Here are the correct screenshots and a summary of the observations ("+" meaning "correct", "-" meaning "wrong" display of the string "Ändern":

Compiler	unmodified string	UTF8ToConsole	UTF8ToAnsi
fpc264	-	+	-
fpc311 w/o UTF8RTL	-	-	+
fpc311 w/ UTF8RTL	+	-	+

@ChrisF: I used the button "Set UTF-8 in RTL" in the Project inspector.

[taazz]

Sorry for the confusion - I was in a hurry and mixed up the too-similar images. And the description of the UTFRTL case is not correct either: both unmodified and UTF8ToAnsi strings are displayed correctly, UTF8ToAnsi is wrong.

Here are the correct screenshots and a summary of the observations ("+" meaning "correct", "-" meaning "wrong" display of the string "Ändern":

Compiler	unmodified string	UTF8ToConsole	UTF8ToAnsi
fpc264	-	+	-
fpc311 w/o UTF8RTL	-	-	+
fpc311 w/ UTF8RTL	+	-	+

@ChrisF: I used the button "Set UTF-8 in RTL" in the Project inspector.

are you on windows? If yes try to add the windows unit in the uses clause and the following line at the top of your code

Code: [Select]

writeln(GetConsoleOutputCP); we should at least understand if the fail is on the console code page or on the conversion it self.

[wp]

Writeln(GetConsoleOutputCP) prints "850". This is also what "mode con: cp /status" reports in a cmd window.

The other results are unchanged in both cases (with and without EnableUTF8RTL)

[taazz]

well based on this page http://www.ascii-codes.com/cp850.html 850 is latin 1 and it supports umlauts correctly as far as I can see which makes this a conversion problem. I was expecting that with utf8rtl enabled it would print utf8  or something along those lines. Well now you should check the assembly if the writeln (no conversion on your part) does any automatic conversion it self (it should not). In any case I would report a bug in this case. Even if fpc changed the console to be able to use utf8, utf16 directly or what ever is the default rtl encoding a call to converting to utf8toconsole should make sure that the input is in utf8 and the output in the active console code page regardless if it has been changed. But that might be just me.

[wp]

What concerns me is that UTF8ToConsole is not working in fpc trunk. This breaks numerous existing code.

[taazz]

What concerns me is that UTF8ToConsole is not working in fpc trunk. This breaks numerous existing code.

Well two main points. 1. Do not use the utf8toconsole as a generic translation for anything else except a console window. I thing that is what utf8tosys is for 2. I agree that this is problematic and thats why I recommend reporting a bug and as soon as possible before fpc 3 is released.

[wp]

http://bugs.freepascal.org/view.php?id=28371

[ChrisF]

I can't say for the OEM stuff, as I've not looked at the concerned code, but concerning the "unmodified" and the "UTF8ToAnsi" cases, at the first glance, your results seem coherent with the current situation.

However, something that could eventually interest you concerning the OEM code page : the "automatic" code page conversion in Free Pascal.

Sample code (use it only with UTF8RTL):

Code: [Select]

type
  stroem = type ansistring(CP_OEMCP);

var
  s: String;
  soe: stroem;
begin
  s := 'Ändern';
  soe := s;
  WriteLn('oem           = ', soe);
  ReadLn;
end.


[taazz]

I can't say for the OEM stuff, as I've not looked at the concerned code, but concerning the "unmodified" and the "UTF8ToAnsi" cases, at the first glance, your results seem coherent with the current situation.

However, something that could eventually interest you concerning the OEM code page : the "automatic" code page conversion in Free Pascal.

Sample code (use it only with UTF8RTL):

Code: [Select]

type
  stroem = type ansistring(CP_OEMCP);

var
  s: String;
  soe: stroem;
begin
  s := 'Ändern';
  soe := s;
  WriteLn('oem           = ', soe);
  ReadLn;
end.


The none utf8rtl is problematic as well it should have the same results as the 2.6.4 compiler. The solution is even worst than the problem what happens if I try to do the same on cyrilic based countries? Should I create a OEMString type that is compile time decided or runtime?

[ChrisF]

The none utf8rtl is problematic as well it should have the same results as the 2.6.4 compiler. The solution is even worst than the problem what happens if I try to do the same on cyrilic based countries? Should I create a OEMString type that is compile time decided or runtime?

As you already know it, I'm afraid I don't have any answers: just a few more questions ...


Concerning the none utf8rtl case with my sample for instance, the "automatic" Free Pascal is also working indeed.

But the problem in this case, is that UTF8 data are put inside a string variable with an ANSI code page (a "real" ANSI code page, I mean: CP_ACP -> Windows.GetACP): so the "automatic" conversion can't work, as the expected data are not the real one.

The same source code with a fix for this problem, and my sample code is still working. For instance, add:

Code: [Select]

{$CODEPAGE UTF8}
at the beginning of the sample source code (which is encoded here in UTF8, by default in the Lazarus IDE).


IMHO, it could explain most of the wp's results concerning this case, except the UTF8ToConsole one.

Concerning UTF8ToConsole, apparently the UTF8ToConsole is a Lazarus function. So ,it may have the same problem as for UTF8toSys (but it's only a supposition of mine).

** Addition **

   The none utf8rtl is problematic as well it should have the same results as the 2.6.4 compiler.

I'm not sure if you mean in this test case (i.e. wp's sample code) or in general ?

Because in general, I'm almost sure it's not the case: not when the source code is UTF8 encoded (Lazarus IDE), and when "string" inside 2.7+ means by default string variable with ANSI data inside.

Of course, it's different if the source code is coming for another provenance, and ANSI encoded.

[taazz]

Addition:

The none utf8rtl is problematic as well it should have the same results as the 2.6.4 compiler.

I'm not sure if you mean in this test case (i.e. wp's sample code) or in general ?

In general. The console code page is 850 in WP's case that means that it should only print cp 850 characters correctly regardless of the input string. If the string is not in the correct CP then it should not print correctly.

Because in general, I'm almost sure it's not the case: not when the source code is UTF8 encoded (Lazarus IDE), and when "string" inside 2.7+ means by default string variable with ANSI data inside.

This is the same case as 2.6.4 all code in lazarus IDE is considered utf8. Why it works on 2.6.4 on windows and it does not work on 3.1.1 on the same windows installation?

Of course, it's different if the source code is coming for another provenance, and ANSI encoded.

No it is not.

[ChrisF]

If the string is not in the correct CP then it should not print correctly.

If I'm correct, it should; because Free Pascal 2.7.1+ is doing the conversion job internally.

For instance, the result for wp's code source with only {$CODEPAGE UTF8} added, and in the none utf8rtl case :

Code: [Select]

program project1;

{$mode objfpc}{$H+}
{$CODEPAGE UTF8}

uses
  {$IFDEF UNIX}{$IFDEF UseCThreads}
  cthreads,
  {$ENDIF}{$ENDIF}
  Classes, SysUtils, LazUTF8
  { you can add units after this };

type
  stroem = type ansistring(CP_OEMCP);

var
  s: string; //utf8String;
  soe: stroem;
begin
  s := 'Ändern';
  WriteLn(s);
  WriteLn('UTF8ToConsole = ', UTF8ToConsole(s));
  WriteLn('UTF8ToAnsi    = ', UTF8ToAnsi(s));
  WriteLn('UTF8ToSys     = ', UTF8ToSys(s));
  WriteLn('UTF8ToWinCP   = ', UTF8ToWinCP(s));
  soe := s;
  WriteLn('oem           = ', soe);
  ReadLn;
end.

I've just added a few more tests. And the results are:

Code: [Select]

Ändern
UTF8ToConsole = Zndern
UTF8ToAnsi    = Ändern
UTF8ToSys     = Ändern
UTF8ToWinCP   = Ändern
oem           = Ändern

As you can see, this time the 'unmodified' string (first line without any legend) is correctly displayed.