using the CODEPAGE correctly?

[helio]

 using the CODEPAGE correctly?

Code: Pascal  [Select][+][-]

1. program Project1;
2. {$mode objfpc}{$H+}
3. {$Codepage UTF8}
4.  
5. Var
6.   S: string;
7.   I1,I2: ShortInt;
8. begin
9.   S:= 'CÇ';
10.   I1:= Length(S);
11.   I2:= Length('CÇ');
12.   WriteLn(I1);
13.   WriteLn(I2);
14. end.
15.  

       

[JuhaManninen]

Don't use {$Codepage UTF8}. It is explained (somehow) here:
  http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus#String_Literals

[helio]

@JuhaManninen.

 Could you give an example of use? I am very confused !!!

[JuhaManninen]

Could you give an example of use? I am very confused !!!

As I wrote, don't use {$Codepage UTF8}. Remove it. Then everything works.

[helio]

OK! without the directive the outputs are 3, the correct would be 2? 

[JuhaManninen]

OK! without the directive the outputs are 3, the correct would be 2? 

No, 3 is correct.
UTF8Length(S) would return 2.
Good luck learning Unicode.

[Bart]

IIRC:
S is a string, you assign 'CÇ' to it, the sourcecode is encode in UTF8, where 'CÇ' then is 3 bytes long and length returns 3 (3 byte size elements).
In Length('CÇ'), the CÇ is treated as a constant by the compiler and evaluated as UnicodeString, and then Length will be 2 (2 word size elements).

If you specify a Windows codepage and save the source with that encoding (in the Lazarus IDE you can do that via the context menu of the editor), then the Length(S) would be 2 as well, since the encoding of the string would be a 1-byte encoding.

Bart

[tetrastes]

In Length('CÇ'), the CÇ is treated as a constant by the compiler and evaluated as UnicodeString, and then Length will be 2 (2 word size elements).

But why without {$Codepage UTF8} Length('CÇ') is 3?

[rvk]

But why without {$Codepage UTF8} Length('CÇ') is 3?

Because 'CÇ' is then evaluated as string (which is a utf-8 string).
And because string consists of one byte elements (even a utf-8 string) you get 3 elements.

If it was unicode (widestring) as it is in Delphi, a string consists of widechars (which is 2 bytes per element).
So in Delphi it would result in 2 (because it uses 2 words/widechar elements).

You need to think if string is made up of WHAT kind of elements. Length will return that number of elements.

[JuhaManninen]

But why without {$Codepage UTF8} Length('CÇ') is 3?

Because then the data is interpreted correctly. Yes, it is counter-intuitive. Please read the wiki page I linked.
The fundamental reason is that our new UTF-8 mode switches the encoding at run-time, yet constants are evaluated at compile-time.
With {$Codepage UTF8} the compiler does a wrong conversion and indeed uses UnicodeString as a temporary medium.

If it was unicode (widestring) as it is in Delphi, ...

UTF-8 is also Unicode. Unfortunately Delphi named their string type as UnicodeString instead of a proper UTF16String. It causes lots of confusion.

Everybody please test my latest code attachment in "Encoding agnostic functions for codepoints + an iterator" thread.
It is ūber-cool and gives more insight of Unicode, codepoints and codeunits.

[Bart]

But why without {$Codepage UTF8} Length('CÇ') is 3?

Because the file encoding is still UTF8 (unless you changed it), but the compiler thinks the codepage is defaultsystemcodepage (probably 1252), it sees the bytes 43 C3 87 and sees them as a 3-byte string in a 1-byte per character (cp1252) encoding.

If you tell the compiler the codepage is UTF8 it will treat string constants as UTF16 (it converts them from UTF8 to UTF16).

I took your example and saved the file in cp1252 encoding (1 byte = 1 char) and removed the {$codepage UTF8}.
It will then correctly output 2 for both length's (the 'CÇ' is stored as 2 bytes 43 C7 in the file in this case).

Code: [Select]

43 C7, I1=2
43 C7, I2=2

Bart

[BeniBela]

If you tell the compiler the codepage is UTF8 it will treat string constants as UTF16

That sounds like a crazy compiler

[rvk]

If it was unicode (widestring) as it is in Delphi, ...

UTF-8 is also Unicode. Unfortunately Delphi named their string type as UnicodeString instead of a proper UTF16String. It causes lots of confusion.

Yeah... that was kind of what I meant. (That's why I put "(widestring)" after unicode but that was the incorrect term).

Unfortunately Delphi named their string type as UnicodeString instead of a proper UTF16String. It causes lots of confusion.

FPC also has a type unicodestring. So was it wrong for FPC to also define one ???

Unicode (unicodestring) can be a UTF-32, UTF-16 or UTF-8 string. I't up to the language (or setting) to determine which it really is. So there is no wrong or right here. (otherwise FPC would also be wrong to define a unicodestring)

[JuhaManninen]

If you tell the compiler the codepage is UTF8 it will treat string constants as UTF16

That sounds like a crazy compiler

Bart's sentence was a little inaccurate.
The compiler treats string constants as UTF-8 if you tell it so. However our UTF-8 "hack" sets the default encoding only later at run-time. At compile time the default string encoding is still the system's codepage, for example cp1252. Thus the compiler converts UTF-8 -> cp1252 when assigning a constant. Later our "hack" changes the default string encoding and the wrongly converted cp1252 data is treated as UTF-8 -> error!

Remember however that the potential problem is only with constants. Variables have dynamic encoding info and assignment goes right always.
Summa summarum:
 Do not define {$Codepage UTF8} and assign constants only to an AnsiString. Then everything works as magic.

Please also look at my LazUnicode unit and test program, attached in another thread.
You can see that I assign a constant to a temporary AnsiString also when {$ModeSwitch UnicodeStrings} is defined and String type maps to UnicodeString.

FPC also has a type unicodestring. So was it wrong for FPC to also define one ???

No, Delphi compatibility is important, they had to define it.
There is no problem if everybody knows the facts. However Unicode is confusing and complicated enough even without extra complications. Take the word "character", what does it mean in Unicode? At least the encodings should be called without ambiquity.

Unicode (unicodestring) can be a UTF-32, UTF-16 or UTF-8 string.

This kind of proves the confusion. Does "unicodestring" mean the Delphi's UTF-16 type or a string of Unicode in general?

[rvk]

Unicode (unicodestring) can be a UTF-32, UTF-16 or UTF-8 string.

This kind of proves the confusion. Does "unicodestring" mean the Delphi's UTF-16 type or a string of Unicode in general?

Yeah... maybe. I should have said (unicode string or unicode-string and not unicodestring ). Unicode (or a string with unicode) itself can be UTF-32, UTF-16 or UTF-8. Unicodestring is usually defined as UTF16 or UCS2. Is there actually a language which uses UTF-8 for type Unicodestring?