Lazarus

Free Pascal => Beginners => Topic started by: jormik on July 14, 2014, 06:00:36 pm

Title: to read an UTF8 text file
Post by: jormik on July 14, 2014, 06:00:36 pm
I can't read correctly characters from a text file encoded in UTF8. I have made many attemps with the functions of FCP (for example with LazUTF8) but without result.
Must I write a personal function to manage  the variable number of bytes of the UTF8 encoding?
Thanks.
Title: Re: to read an UTF8 text file
Post by: skalogryz on July 14, 2014, 06:06:00 pm
I think you're reading the file just fine. The problem with outputting the results.

What are you trying to achieve?
Title: Re: to read an UTF8 text file
Post by: jormik on July 14, 2014, 06:53:15 pm
I try to transform my old Firebird+Delphi6 project into a new UTF8 Firebird+Lazarus project. All is gone fine, but not the read of the textfiles.

I must scan textfiles (once ANSI, now UTF8), char by char, and then build appropriate strings to populate the db. The problem is in the variable number of bytes of UTF8, that implies a procedure. LazUTF8 makes this job, but not for files, only for obtain, for example, code points of Unicode (that I use in other parts of project).
Title: Re: to read an UTF8 text file
Post by: skalogryz on July 14, 2014, 07:05:46 pm
hmm... could you please provide an example?
Title: Re: to read an UTF8 text file
Post by: jormik on July 14, 2014, 09:41:07 pm
This is the skeleton of a procedure that inserts word into a table of the database.

============================================
procedure example;
var
  t: integer;
  doc: TextFile;
  character: char;
  word: string;
begin
  OpenDialog.Execute;
  AssignFile(doc, OpenDialog.FileName);
  reset(doc);
  word := '';
  for  t := 1 to 100 do
    begin
      read(doc, character);
      if character = ' ' then
        begin
          // write word in the database
          word := ''
        end
      else
         word := word + carattere;
      end;
  CloseFile(doc);
end;
============================================

It works fine with ANSI textfiles, where a character corresponds always to a byte. How can I obtain the same result with UTF8 textfiles, where NOT always a character corresponds to a byte?
Title: Re: to read an UTF8 text file
Post by: skalogryz on July 14, 2014, 10:01:14 pm
is there a chance that the issue is with database collation?

no wait. I'm being stupid here.

actually, yes. it still should work. what's the collation?
Title: Re: to read an UTF8 text file
Post by: jormik on July 14, 2014, 11:27:09 pm
I'm not sure to have understood, but I think that there is no problem with database collation.

I need a function that recognizes, in a UTF8 textfile, when it is to load, from the file, a bunch of one, two, three or four bytes to represent the opportune Unicode character.
Title: Re: to read an UTF8 text file
Post by: skalogryz on July 15, 2014, 05:06:17 am
Looking at the code of your function, I don't see a specific need for you to recognize a multi-byte characters. Just because you're looking for a space-character. Space-character is a single-byte character in UTF8 as well.

Now. Back to your problem. What I suspect is that the issue is either in db collation OR db component that you're using.

I don't know how you actually push the word to the database, but I'd suggest you to try the following:
Code: [Select]
  for  t := 1 to 100 do
    begin
      read(doc, character);
      if character = ' ' then
        begin
          word:=Utf8ToAnsi(word);
          // write word in the database
          word := ''
        end
      else
         word := word + carattere;
      end;
Title: Re: to read an UTF8 text file
Post by: jormik on July 15, 2014, 10:27:03 am
The db is OK. I turned it into UTF8 encoding, so now, when I put a field with not Ansi characters from a table of Firebird into a Lazarus textedit, and viceversa, all goes fine.

The problem comes with UTF8/Unicode files.

Please, take a look to the uploaded project (here summarized), compile it and read the two encoded textfiles (exUTF8.txt and exUnicode.txt). I followed what you have suggested (use Utf8ToAnsi function) but the result is wrong.

Code: [Select]
procedure TForm1.ButtonClick(Sender: TObject);
var
  doc: TextFile;
  character: Char;
begin
  Edit1.Text:='';
  Edit2.Text:='';
  memo.Text:='';
  OpenDialog.Execute;
  AssignFile(doc, OpenDialog.FileName);
  reset(doc);
  while not EOF(doc) do
    begin
      read(doc,character);
      Edit1.Text:=Edit1.Text+character;
      Edit2.Text:=Edit2.Text+Utf8ToAnsi(character);
      memo.Text:=memo.Text+IntToStr(ord(character))+chr(13);
    end;
  CloseFile(doc);
end;
Title: Re: to read an UTF8 text file
Post by: skalogryz on July 15, 2014, 02:20:45 pm
Quote from: jormik on July 15, 2014, 10:27:03 am
Please, take a look to the uploaded project (here summarized), compile it and read the two encoded textfiles (exUTF8.txt and exUnicode.txt). I followed what you have suggested (use Utf8ToAnsi function) but the result is wrong.
...
yes, they're wrong, because you've changed the code completely!

Originally, you've been building a word of different characters and then pushing the complete word (that keeps the whole UTF8 encoding).
In this project, you're forcefully breaking the utf8 encoded word into separate characters.

I've updated the project (pretty much returned to the original code you posted).
As you can see, UTF8 now loads fine. (I've also added BOM skipping to the code).

Unicode loading will require a different approach.
You will either need to read "WideChars" and build them into "WideString" (which is straight forward).
Or reading "char" by "char", build a buffer that turns into "WideString".
Title: Re: to read an UTF8 text file
Post by: jormik on July 15, 2014, 06:07:04 pm
Great!
Thank you.
Title: Re: to read an UTF8 text file
Post by: nk on March 14, 2025, 10:24:30 pm
Code: Pascal  [Select][+][-]
  1.  
  2.  
  3. program project1;
  4.  
  5. uses
  6.   Classes,
  7.   SysUtils;
  8.  
  9. var
  10.   F: Text;
  11.   SL: TStringList;
  12.   s: string;
  13.   i: Int32;
  14. begin
  15.  
  16.   // WRITE FILE,  option 1
  17.   try
  18.     assignfile(f, 'text1.txt', CP_UTF8);
  19.     Rewrite(f);
  20.     writeln(f, 'Привет мир');
  21.     writeln(f, 'Möhren');
  22.   finally
  23.     Close(f);
  24.   end;
  25.  
  26.   // WRITE FILE, option 2
  27.   SL := TStringList.Create;
  28.   try
  29.     SL.Add('∫cos(x)dx = sin(x) + C');
  30.     SL.Add('¬(a ∧ b) ⇔ ¬a ∨ ¬b');
  31.     SL.Add('Привет мир');
  32.     SL.Add('Möhren');
  33.     SL.SaveToFile('text2.txt', TEncoding.UTF8);
  34.   finally
  35.     SL.Free;
  36.   end;
  37.  
  38.   // READ FILE, option 1
  39.   try
  40.     assignfile(f, 'text1.txt', CP_UTF8);
  41.     Reset(f);
  42.     repeat
  43.       Readln(f, S);
  44.       Writeln(S);
  45.     until S = '';
  46.   finally
  47.     Close(f);
  48.   end;
  49.  
  50.   // READ FILE, option 2
  51.   SL := TStringList.Create;
  52.   try
  53.     sl.LoadFromFile('text2.txt', TEncoding.UTF8);
  54.     for i:=0 to sl.Count-1 do begin
  55.       writeln(SL[i]);
  56.     end;
  57.   finally
  58.     SL.Free;
  59.   end;
  60.  
  61.   ReadLn();
  62. end.  
  63.  
Title: Re: to read an UTF8 text file
Post by: jamie on March 16, 2025, 02:02:19 am
why not simply first determine the whole file size, set the length of a string to that size and then use a blockread or just read to load the complete contents in a single line?

Read(FileHanle, Pchar(THeString)^, SizeOfString);

of course you need to first set the string length to that size before loading it.

after that, it will all be there.

In fact, you can use a TStringList and it should not have any problems loading it.

jamie
Title: Re: to read an UTF8 text file
Post by: cdbc on March 16, 2025, 07:41:32 am
Hi
I use these:
Code: Pascal  [Select][+][-]
  1. procedure bcLoadFromFileUTF8(aStrings: TStrings; const aFilename: string);
  2. var
  3.   Fs: TFileStream;
  4.   Buf: string;
  5.   Cp: word;
  6. begin
  7.   Fs:= TFileStream.Create(aFilename,fmOpenRead or fmShareDenyNone);
  8.   try
  9.     SetLength({%H-}Buf,3);
  10.     Fs.Read(Buf[1],3);
  11.     if Buf = bcUTF8bom then begin
  12.       aStrings.LoadFromStream(Fs);
  13.     end else begin
  14.       Fs.Seek(0,soFromBeginning);
  15.       aStrings.LoadFromStream(Fs);
  16.       Cp:= bcGuessEncoding(aStrings.Text);
  17.       aStrings.Text:= bcEncodeUTF8(aStrings.Text,Cp);
  18.     end;
  19.   finally Fs.Free; end;
  20. end; { bcLoadFromFileUTF8 }
  21.  
  22. procedure bcSaveToFileUTF8(aStrings: TStrings; const aFilename: string);
  23. var
  24.   Fs: TFileStream;
  25. begin
  26.   Fs:= TFileStream.Create(aFilename,fmCreate or fmOpenWrite);
  27.   try
  28.     Fs.Write(bcUTF8bom[1],length(bcUTF8bom));
  29.     aStrings.SaveToStream(Fs);
  30.   finally Fs.Free; end;
  31. end; { bcSaveToFileUTF8 }
In the hope it can inspire someone...
Regards Benny
TinyPortal © 2005-2018