Backward character encoding

[hakelm]

I wanted to reverse the order of the characters in some strings but was in for an unsuspected surprise.
In the reversed strings some characters for instance 'ä'  (0xE4) where often but not always replaced by '?' (0x3F).
Find enclosed a little test program illustrating this.
Any tip on what I am doing wrong is appreciated.
H

[Blaazen]

Another encoding problem

I assume that your string is UTF-8. This encoding has not characters 0x80..0xFF. Such characters are encoded to 2 or more bytes.

[balazsszekely]

A quick workaround:

Code: [Select]

uses lazutf8;

procedure TForm1.Button1Click(Sender: TObject);
var
  n, i:integer;
  us1, us2, us3: unicodestring; //widestring if you prefer
begin
  memo2.Clear; memo3.Clear;
  for n:=0 to memo1.Lines.Count-1 do
  begin
    us1 := UTF8ToUTF16(memo1.Lines[n]);
    us2:=''; us3:='';
    for i:=1 to length(us1) do
    begin
      us2:=us1[i]+us2;
      us3:=us3+us1[i];
    end;
    memo2.Lines.Add(UTF16ToUTF8(us2));
   memo3.Lines.Add(UTF16ToUTF8(us3));
  end;
end;



[hakelm]

Thanks a lot, that did the trick.
I had already tried unicodestring and widestring but in my naivety I thought that the LCL would handle the conversions automatically so I didn't use the conversion functions.
In my mind a small question remains: why does the forward character insertion but not the backward work?
H

[Bart]

Mind you that this can also fail.
The assumption in this code is that each "character" (unicode codepoint) will fit into 1 widechar.
This is not the case (but for most western europese countries it won't be a problem).

It's not a Lazarus problem. Most code in Delphi (that uses widestrings now), also assumes that all codepoints fir into 1 widechar.

Unicode is lots of fun!

Bart

[hakelm]

Sorry, I am still a bit confused.
To my knowledge I am using codepage  ISO/IEC 8859-1 which I understand is not unicode.
The hex representation of my input data (memo1) is (as copied from the object inspector):
61 62 63 20 E5 E4 F6 20 F6 E4 E5 0D 0A E5 0D 0A E4 0D 0A F6 0D 0A                         
which doesn't look like unicode.
H

[Bart]

The hex representation of my input data (memo1) is (as copied from the object inspector):
61 62 63 20 E5 E4 F6 20 F6 E4 E5 0D 0A E5 0D 0A E4 0D 0A F6 0D 0A                         
which doesn't look like unicode.

Now you got me confused.
The hexadecimal representation of that string is indeed NOT unicode.
However, the Lazarus IDE is fully UTF8, so any content from a memo inside Lazarus (actually LCL) must be UTF8.
So how did you get the Hex representation?
Did you copy/paste it to some other editor (or even Lazarus IDE, but you set encoding of source to ISO/IEC 8859-1) and then started to "hexify" it, or did you use some other tool?

Just a test:
I assume this is the test string?

Code: [Select]

abc åäö öäå
å
ä
ö

If I put the above text in a TMemo and then "hexify" it I get this:

Code: [Select]

61 62 63 20 C3 A5 C3 A4 C3 B6 20 C3 B6 C3 A4 C3 A5 0D 0A C3 A5 0D 0A C3 A4 0D 0A C3 B6 0D 0A

Which is distictly different from your Hex representation (and it shows it is Utf8).

Bart

[hakelm]

You are using the right test string.
I did the following:
From the object inspector I opened up memo1.lines code editor, copied the text, and pasted it into Ultraeditor that directly shows me the hex representation. (1)
Now, later, I have saved memo1.lines into a textfile and on a Linux box done
hexdump -v -e '/1 "%02X "' abc.txt ;echo
Then I get exactly the same result as you. (2)
So somewhere somehow utf8 gets converted to single byte representation and vice versa.
Interestingly if I on the linux box use standard input for hexdump and paste the string I get still another representation (3) where the chars with higher order simply have been dumped.

(1): 61 62 63 20 E5 E4 F6 20 F6 E4 E5 0D 0A E5 0D 0A E4 0D 0A F6 0D 0A
(2): 61 62 63 20 C3 A5 C3 A4 C3 B6 20 C3 B6 C3 A4 C3 A5 0D 0A C3 A5 0D 0A C3 A4 0D 0A C3 B6 0D 0A
(3): 61 62 63 20 C3 A5 C3 A4 C3 B6 20 C3 B6 C3 A4 C3 A5 0A C3 A5 0A C3 A4 0A C3 B6 0A

H


So we have 3 representations of the same string.

[Bart]

From the object inspector I opened up memo1.lines code editor, copied the text, and pasted it into Ultraeditor that directly shows me the hex representation. (1)

Ultraeditor internally does not use UTF8 then I guess.

Interestingly if I on the linux box use standard input for hexdump and paste the string I get still another representation (3) where the chars with higher order simply have been dumped.

Looking at (3) I guess the "carriage returns" (0D) have been removed (Linux uses a single LineFeed (0A) as newline character.

Code: [Select]

(2): 61 62 63 20 C3 A5 C3 A4 C3 B6 20 C3 B6 C3 A4 C3 A5 0D 0A C3 A5 0D 0A C3 A4 0D 0A C3 B6 0D 0A
(3): 61 62 63 20 C3 A5 C3 A4 C3 B6 20 C3 B6 C3 A4 C3 A5 ** 0A C3 A5 ** 0A C3 A4 ** 0A C3 B6 ** 0A

Bart

[hakelm]

You are of course right.
Ultreditor has full support for UTF-8, it may be my clip board manager (Clip mate).
H