Text file: ANSI vs UTF-8

[asdf]

Why my UTF-8 text file has 3 hiden characters at the first 3 positions in the first line?
But not if converted to ANSI.

[asdf]

That's why I also can't use 'ansirightstr' in LZR/FPC.
http://www.lazarus.freepascal.org/index.php/topic,11054.0.html

Is LZR/FPC use only UTF-8?

[OpenLieroXor]

I suppose that's Byte Order Mark (http://en.wikipedia.org/wiki/Byte_order_mark).

Is LZR/FPC use only UTF-8?

AFAIK whole LCL uses UTF-8 encoding, but if you need ANSI representation of string, you can use Utf8ToAnsi and AnsiToUtf8 functions.

[asdf]

Thank you so much  .

[asdf]

In LZR, I have exported data into a text file ...

<<PRODUCT[00000001]PAGE[0109]PARAGRAPH[01]>>aaa
<<PRODUCT[00000001]PAGE[0109]PARAGRAPH[01]>>bbb
<<PRODUCT[00000001]PAGE[0109]PARAGRAPH[01]>>ccc
<<PRODUCT[00000001]PAGE[0109]PARAGRAPH[01]>>ddd
<<PRODUCT[00000001]PAGE[0109]PARAGRAPH[01]>>eee
.
.
.

In another procedure, I used the above text file ...
 
reset(tf1);
while not eof(tf1) do
begin
readln(tf1,ln);

showmessage(ansirightstr(ansileftstr(trimleft(utf8toansi(ln)),28),4)
           +' / ' +ansirightstr(ansileftstr(trimleft(utf8toansi(ln)),41),2));

Why ?
1. the result from first line was  [010 / [0
2. But the second line was 0109 / 01 as needed.

[asdf]

I used MSWord to view and then saw an hidden character.
How can I write the code without if it is the first line then + 1 character ?

[typo]

How did you write it to the file? Do the characters appear on NotePad?

[Zoran]

Add unit LConvEncoding to uses list. In this unit you will find many conversion functions between different encodings. Among them there is a function UTF8BOMToUTF8, which should solve your problem.

[asdf]

How did you write it to the file? Do the characters appear on NotePad?

assignfile(tfl,fl);
if fileexists(fl) then
Append(tfl)
else
rewrite(tfl);
writeln(tfl,ECheckListBox.Items);
end;
end;
closefile(tfl);
.
.
.
And there's nothing strange in Notepad.                      

[eny]

And there's nothing strange in Notepad.

IIRC Notepad suppresses the BOM and will not display it.

[asdf]

while not eof(tf1) do
begin
readln(tf1,ln);
ln:=utf8bomtoutf8(ln);
.
.
.
writeln(tf2,StuffString(ln, 1, 54, ''));

It worked well only in the first line,
but why did it replace 57 characters instead of 54 in every line starting in the second line until end ?

[marcov]

That's why I also can't use 'ansirightstr' in LZR/FPC.
http://www.lazarus.freepascal.org/index.php/topic,11054.0.html

Is LZR/FPC use only UTF-8?

FPC uses the encoding of the system. There is a small problem under Windows, because there there are three default encodings:

- OEM encoding for the textmode console
- ansi encoding for most 1-byte API functions (can be UTF-8, but typically isn't)
- 2-byte API functions (the so called "wide" or -W functions) API

In general FPC uses the ansi encoding on Windows, and inserts oem2ansi conversions where necessary. It also usually calls the 1-byte -A API functions.

Lazarus assumes all 1-byte strings (ansistring) contain UTF-8.  This means ansi input from the system needs to be converted. (and the same for OEM,  but textmode console is less important for Lazarus, moreover FPC units already abstract that)

Lazarus (probably) tries to use 2-byte -W functions as much as possible, but utf-8 can be converted into 2 byte UTF-16 without information loss.

Older Delphi's (till and including D2007) are like FPC. Assume default ansi encoding.

Newer Unicode Delphi's (Delphi 2009 and beyond) are nearly entirely 2-byte in everything.  Moreover, they allow to attach the encoding (UTF-8, default ansi or OEM) to ansistrings so that conversions are done automatically when properly typed.


A project to implement such functionality (as in D2009+) in FPC has started but is stalled.

[asdf]

while not eof(tf1) do
begin
readln(tf1,ln);
ln:=utf8tocp874(ln);

Using utf8tocp874, it worked so well  .
Could anybody teach me this function ?