to read an UTF8 text file

[jormik]

I can't read correctly characters from a text file encoded in UTF8. I have made many attemps with the functions of FCP (for example with LazUTF8) but without result.
Must I write a personal function to manage  the variable number of bytes of the UTF8 encoding?
Thanks.

[skalogryz]

I think you're reading the file just fine. The problem with outputting the results.

What are you trying to achieve?

[jormik]

I try to transform my old Firebird+Delphi6 project into a new UTF8 Firebird+Lazarus project. All is gone fine, but not the read of the textfiles.

I must scan textfiles (once ANSI, now UTF8), char by char, and then build appropriate strings to populate the db. The problem is in the variable number of bytes of UTF8, that implies a procedure. LazUTF8 makes this job, but not for files, only for obtain, for example, code points of Unicode (that I use in other parts of project).

[skalogryz]

hmm... could you please provide an example?

[jormik]

This is the skeleton of a procedure that inserts word into a table of the database.

============================================
procedure example;
var
  t: integer;
  doc: TextFile;
  character: char;
  word: string;
begin
  OpenDialog.Execute;
  AssignFile(doc, OpenDialog.FileName);
  reset(doc);
  word := '';
  for  t := 1 to 100 do
    begin
      read(doc, character);
      if character = ' ' then
        begin
          // write word in the database
          word := ''
        end
      else
         word := word + carattere;
      end;
  CloseFile(doc);
end;
============================================

It works fine with ANSI textfiles, where a character corresponds always to a byte. How can I obtain the same result with UTF8 textfiles, where NOT always a character corresponds to a byte?

[skalogryz]

is there a chance that the issue is with database collation?

no wait. I'm being stupid here.

actually, yes. it still should work. what's the collation?

[jormik]

I'm not sure to have understood, but I think that there is no problem with database collation.

I need a function that recognizes, in a UTF8 textfile, when it is to load, from the file, a bunch of one, two, three or four bytes to represent the opportune Unicode character.

[skalogryz]

Looking at the code of your function, I don't see a specific need for you to recognize a multi-byte characters. Just because you're looking for a space-character. Space-character is a single-byte character in UTF8 as well.

Now. Back to your problem. What I suspect is that the issue is either in db collation OR db component that you're using.

I don't know how you actually push the word to the database, but I'd suggest you to try the following:

Code: [Select]

  for  t := 1 to 100 do
    begin
      read(doc, character);
      if character = ' ' then
        begin
          word:=Utf8ToAnsi(word);
          // write word in the database
          word := ''
        end
      else
         word := word + carattere;
      end;


[jormik]

The db is OK. I turned it into UTF8 encoding, so now, when I put a field with not Ansi characters from a table of Firebird into a Lazarus textedit, and viceversa, all goes fine.

The problem comes with UTF8/Unicode files.

Please, take a look to the uploaded project (here summarized), compile it and read the two encoded textfiles (exUTF8.txt and exUnicode.txt). I followed what you have suggested (use Utf8ToAnsi function) but the result is wrong.

Code: [Select]

procedure TForm1.ButtonClick(Sender: TObject);
var
  doc: TextFile;
  character: Char;
begin
  Edit1.Text:='';
  Edit2.Text:='';
  memo.Text:='';
  OpenDialog.Execute;
  AssignFile(doc, OpenDialog.FileName);
  reset(doc);
  while not EOF(doc) do
    begin
      read(doc,character);
      Edit1.Text:=Edit1.Text+character;
      Edit2.Text:=Edit2.Text+Utf8ToAnsi(character);
      memo.Text:=memo.Text+IntToStr(ord(character))+chr(13);
    end;
  CloseFile(doc);
end;


[skalogryz]

Please, take a look to the uploaded project (here summarized), compile it and read the two encoded textfiles (exUTF8.txt and exUnicode.txt). I followed what you have suggested (use Utf8ToAnsi function) but the result is wrong.
...

yes, they're wrong, because you've changed the code completely!

Originally, you've been building a word of different characters and then pushing the complete word (that keeps the whole UTF8 encoding).
In this project, you're forcefully breaking the utf8 encoded word into separate characters.

I've updated the project (pretty much returned to the original code you posted).
As you can see, UTF8 now loads fine. (I've also added BOM skipping to the code).

Unicode loading will require a different approach.
You will either need to read "WideChars" and build them into "WideString" (which is straight forward).
Or reading "char" by "char", build a buffer that turns into "WideString".

[jormik]

Great!
Thank you.

[nk]

Code: Pascal  [Select][+][-]

1.  
2.  
3. program project1;
4.  
5. uses
6.   Classes,
7.   SysUtils;
8.  
9. var
10.   F: Text;
11.   SL: TStringList;
12.   s: string;
13.   i: Int32;
14. begin
15.  
16.   // WRITE FILE,  option 1
17.   try
18.     assignfile(f, 'text1.txt', CP_UTF8);
19.     Rewrite(f);
20.     writeln(f, 'Привет мир');
21.     writeln(f, 'Möhren');
22.   finally
23.     Close(f);
24.   end;
25.  
26.   // WRITE FILE, option 2
27.   SL := TStringList.Create;
28.   try
29.     SL.Add('∫cos(x)dx = sin(x) + C');
30.     SL.Add('¬(a ∧ b) ⇔ ¬a ∨ ¬b');
31.     SL.Add('Привет мир');
32.     SL.Add('Möhren');
33.     SL.SaveToFile('text2.txt', TEncoding.UTF8);
34.   finally
35.     SL.Free;
36.   end;
37.  
38.   // READ FILE, option 1
39.   try
40.     assignfile(f, 'text1.txt', CP_UTF8);
41.     Reset(f);
42.     repeat
43.       Readln(f, S);
44.       Writeln(S);
45.     until S = '';
46.   finally
47.     Close(f);
48.   end;
49.  
50.   // READ FILE, option 2
51.   SL := TStringList.Create;
52.   try
53.     sl.LoadFromFile('text2.txt', TEncoding.UTF8);
54.     for i:=0 to sl.Count-1 do begin
55.       writeln(SL[i]);
56.     end;
57.   finally
58.     SL.Free;
59.   end;
60.  
61.   ReadLn();
62. end.  
63.  

[jamie]

why not simply first determine the whole file size, set the length of a string to that size and then use a blockread or just read to load the complete contents in a single line?

Read(FileHanle, Pchar(THeString)^, SizeOfString);

of course you need to first set the string length to that size before loading it.

after that, it will all be there.

In fact, you can use a TStringList and it should not have any problems loading it.

jamie

[cdbc]

Hi
I use these:

Code: Pascal  [Select][+][-]

1. procedure bcLoadFromFileUTF8(aStrings: TStrings; const aFilename: string);
2. var
3.   Fs: TFileStream;
4.   Buf: string;
5.   Cp: word;
6. begin
7.   Fs:= TFileStream.Create(aFilename,fmOpenRead or fmShareDenyNone);
8.   try
9.     SetLength({%H-}Buf,3);
10.     Fs.Read(Buf[1],3);
11.     if Buf = bcUTF8bom then begin
12.       aStrings.LoadFromStream(Fs);
13.     end else begin
14.       Fs.Seek(0,soFromBeginning);
15.       aStrings.LoadFromStream(Fs);
16.       Cp:= bcGuessEncoding(aStrings.Text);
17.       aStrings.Text:= bcEncodeUTF8(aStrings.Text,Cp);
18.     end;
19.   finally Fs.Free; end;
20. end; { bcLoadFromFileUTF8 }
21.  
22. procedure bcSaveToFileUTF8(aStrings: TStrings; const aFilename: string);
23. var
24.   Fs: TFileStream;
25. begin
26.   Fs:= TFileStream.Create(aFilename,fmCreate or fmOpenWrite);
27.   try
28.     Fs.Write(bcUTF8bom[1],length(bcUTF8bom));
29.     aStrings.SaveToStream(Fs);
30.   finally Fs.Free; end;
31. end; { bcSaveToFileUTF8 }

In the hope it can inspire someone...
Regards Benny