PChar and UTF-8. Weird stuff. (Fpc bug )

[beria]

Code: Pascal  [Select][+][-]

1. {$codepage utf8}
2. program project1;
3.  
4. var
5.   ar: PChar = 'Ёжик';
6. begin
7.  Writeln(strlen(ar));  //Error
8.   Writeln(Length(Ar)); //Error
9.   ar := 'Ёжик';
10.   Writeln(strlen(ar)); //Ok
11.   Writeln(Length(Ar)); //Ok
12.   Readln;
13. end.    

4
4
8
8

Code: Pascal  [Select][+][-]

1. program project1;
2.  
3. var
4.   ar: PChar = 'Ёжик';
5. begin
6.  Writeln(strlen(ar)); //Ok
7.   Writeln(Length(Ar)); //Ok
8.   ar := 'Ёжик';
9.   Writeln(strlen(ar)); //Ok
10.   Writeln(Length(Ar)); //Ok
11.   Readln;
12. end.

8
8
8
8

Why does the RECOMMENDED compile switch {$codepage utf8} in some cases fail to work with UTF-8 strings in the PChar container?

[MarkMLl]

Don't mix C-style strings (PChar etc.) and Pascal-style string handling functions. It's asking for trouble.

MarkMLl

[beria]

Don't mix C-style strings (PChar etc.) and Pascal-style string handling functions. It's asking for trouble.

MarkMLl

 Writeln(strlen(ar));  //Error 
strlen -  function specifically for the PChar type

[MarkMLl]

Writeln(strlen(ar));  //Error 
strlen -  function specifically for the PChar type

Yes, but ar is a pointer, and you're relying on compiler magic to assign a Pascal-style string containing UTF-8 encoded characters to it.

You're then demonstrating that it's not a happy combination :-)

Also see https://forum.lazarus.freepascal.org/index.php/topic,60324.msg450814.html#msg450814

MarkMLl

[JuhaManninen]

Why does the RECOMMENDED compile switch {$codepage utf8} in some cases fail to work with UTF-8 strings in the PChar container?

It is explained here:
 https://wiki.freepascal.org/Unicode_Support_in_Lazarus
Read especially the "String Literals" part. {$codepage utf8} is not recommended.
Yes, it is counter-intuitive. The fundamental reason is that the Lazarus UTF-8 solution is a hack in FPC's point of view. However the hack works amazingly well when you assign string literals always to a variable of type "String". After that you can iterate it using a PChar without problems.

[Martin_fr]

With fpc 3.3.1 it's 1,1,8,8 => IMHO a bug in fpc.

[tetrastes]

Why does the RECOMMENDED compile switch {$codepage utf8} in some cases fail to work with UTF-8 strings in the PChar container?

It is explained here:
 https://wiki.freepascal.org/Unicode_Support_in_Lazarus
Read especially the "String Literals" part. {$codepage utf8} is not recommended.

I would add "Read especially "What happens when I use $codepage utf8?" part". It explains why this switch is especially not recommended for OSes with UTF-8 system codepage.

By the way, at linux-x86_64 with fpc 3.2.2 I get (with $codepage utf8)
0
0
8
8

And if I add

Code: Pascal  [Select][+][-]

1. uses cwstring;

5
5
8
8

And of course if you add

Code: Pascal  [Select][+][-]

1. {$codepage utf8}
2. ...
3. var
4.   ar: PChar = 'Ёжик';
5. begin
6.   writeln(string(ar));
7. ...
8.  

you will not get Ёжик at output

[beria]

Read especially the "String Literals" part. {$codepage utf8} is not recommended.

With {$codepage utf8} or the compiler switch -FcUTF8, PChar does not work in principle. This I have already understood and accepted, although it complicates console output. But it goes on to say at "https://wiki.freepascal.org/Unicode_Support_in_Lazarus/ru" that "WideString/UnicodeString/UTF8String only work with {$codepage utf8} / -FcUTF8."
If anything, I use UTF8String to form a network query, and WideString is generally the only string type used to bind to Autocad and MSOffice (Win,Mac) scripting languages .....

[MarkMLl]

WideString is generally the only string type used to bind to Autocad and MSOffice (Win,Mac) scripting languages .....

...which is obviously a significant consideration. But my understanding is that the correct doctrine is that since the base language is Pascal, you're safer using Pascal-style strings and only converting to a PChar when calling an API or library that requires it.

I admit to being no great lover of this UTF-8 stuff, but there's really two issues here: Pascal vs C-type strings (AnsiString, String, WideString), and character representation (UTF-8 vs a fixed-width Unicode encoding).

MarkMLl

[PascalDragon]

Don't mix C-style strings (PChar etc.) and Pascal-style string handling functions. It's asking for trouble.

There are no Pascal-style string handling functions involved in this example (Length exists in a variant for both and a string constant is not a Pascal-style string).

Why does the RECOMMENDED compile switch {$codepage utf8} in some cases fail to work with UTF-8 strings in the PChar container?

It is explained here:
 https://wiki.freepascal.org/Unicode_Support_in_Lazarus
Read especially the "String Literals" part. {$codepage utf8} is not recommended.
Yes, it is counter-intuitive. The fundamental reason is that the Lazarus UTF-8 solution is a hack in FPC's point of view. However the hack works amazingly well when you assign string literals always to a variable of type "String". After that you can iterate it using a PChar without problems.

The example in question does not use any LCL units, thus what is suggested for Lazarus is not relevant here.

Why does the RECOMMENDED compile switch {$codepage utf8} in some cases fail to work with UTF-8 strings in the PChar container?

It's a bug in the handling of the constant string (the code that handles the constant value does not correctly handle the case of the string constant being a Unicode string constant unlike the code for expressions). Please report as a bug.

[beria]

Please report as a bug.

Ok!