Read file as UTF-8

[GreatCorn]

I've been trying to make this work for too long and I'm out of ideas. All I need is to read a UTF-8 text file (for example, the contents are "█©"). Of course, reading a file of char results in gibberish ("тЦИ┬й"), reading as widechar results in question marks.
My code is:

Code: Pascal  [Select][+][-]

1. var
2.   openFile: file of widechar;
3.   fileChar: widechar;
4.   command: widestring;
5.  
6. begin
7.   Assign(openFile, 'UTF8.txt');
8.   Reset(openFile);
9.   command := '';
10.   Reset(openFile);
11.   while not EOF(openFile) do
12.   begin
13.     Read(openFile, fileChar);
14.     command := command+fileChar;
15.   end;
16.   Close(openFile);
17.   WriteLn(command);
18.   ReadLn(command);
19. end.

Output: ??
Are there any ways to properly read and display a UTF-8 file? I'm using FreePascal.

[howardpc]

You could try this approach:

Code: Pascal  [Select][+][-]

1. program project1;
2.  
3. {$mode objfpc}{$H+}
4.  
5. uses
6.   Classes;
7.  
8. var
9.   sl: TStringList;
10.   s: String;
11.  
12. begin
13.   sl := TStringList.Create;
14.   try
15.     sl.LoadFromFile('utf8.txt');
16.     for s in sl do
17.       WriteLn(s);
18.     ReadLn;
19.   finally
20.     sl.Free;
21.   end;
22. end.

[GreatCorn]

Thanks, but I'm trying not to use objfpc and Classes library due to compatibility reasons (also Classes weighting 200+ KB)

[marcov]

Thanks, but I'm trying not to use objfpc and Classes library due to compatibility reasons (also Classes weighting 200+ KB)

You can do without classes, maybe, using

Code: Pascal  [Select][+][-]

1. assignfile(f,name,65001);

But that still requires mode objfpc or delphi.

[GreatCorn]

Code: Pascal  [Select][+][-]

1. assignfile(f,name,65001);

What is the third parameter though? Couldn't find any reference and even Lazarus doesn't recognize it.

[Awkward]

objpas.pp (OBJPAS mode)

Procedure AssignFile(out t:Text;p:pchar; aCodePage : TSystemCodePage);

[GreatCorn]

objpas.pp (OBJPAS mode)

Procedure AssignFile(out t:Text;p:pchar; aCodePage : TSystemCodePage);

Strangely my objpas doesn't contain an overload with aCodePage.. Is this OS or version specific? I've got Win7, Lazarus 2.0.8.

[flowCRANE]

Code: Pascal  [Select][+][-]

1. program Project1;
2. var
3.   Input: TextFile;
4.   Line: String;
5. begin
6.   Assign(Input, 'foo.txt');
7.   Reset(Input);
8.  
9.   while not EoF(Input) do
10.   begin
11.     ReadLn(Input, Line); // just read the line
12.     {..}
13.   end;
14.  
15.   Close(Input);
16. end.

Looks good for me — the Line always contain proper data in the form of UTF-8 codepoints.

(also Classes weighting 200+ KB)

So what?

[lucamar]

Try like this (note: untested):

Code: Pascal  [Select][+][-]

1. var
2.   openFile: File of Char;
3.   fileChar: Char;
4.   tempStr: RawByteString;
5.   command: UTF8String;
6.  
7. begin
8.   Assign(openFile, 'UTF8.txt');
9.   Reset(openFile);
10.   tempStr := '';
11.   while not EOF(openFile) do
12.   begin
13.     Read(openFile, fileChar);
14.     tempStr := tempStr + fileChar;
15.   end;
16.   Close(openFile);
17.   comand := tempStr;
18.   WriteLn(command);
19.   ReadLn(command);
20. end.

A little convoluted but it should ensure that no conversion is applied either when assigning chars to tempStr or when assigning this last to command.

But frankly, for such a task and unless you're very much constrained by memory/disk considerations I would simply use ReadFileToString() (from unit FileUtil of LazUtils). I'm as lazy as that

[GreatCorn]

Using

Code: Pascal  [Select][+][-]

1. Input: TextFile;
2. {...}
3. ReadLn(Input, Line);

didn't work but gave a different result: п>їв-?Вс;
using

Code: Pascal  [Select][+][-]

1. tempStr: RawByteString;
2. command: UTF8String;
3. {...}
4. comand := tempStr;

also didn't work but gave yet another result: я╗┐тЦИ┬й, which is closer to the first one I've got 

[marcov]

But is that problem in the reading, or the displaying? the windows console is terribly well at utf8.

[lucamar]

using

Code: Pascal  [Select][+][-]

1. tempStr: RawByteString;
2. command: UTF8String;
3. {...}
4. comand := tempStr;

also didn't work but gave yet another result: я╗┐тЦИ┬й, which is closer to the first one I've got 

Are you completely sure the file is coded as UTF8? Because I just tested something like that here with a guaranteed UTF8 text file I use for this kind of tests and it worked as it should...

But is that problem in the reading, or the displaying? the windows console is terribly well at utf8.

I didn't though of that; it might well be it.

[Awkward]

objpas.pp (OBJPAS mode)

Procedure AssignFile(out t:Text;p:pchar; aCodePage : TSystemCodePage);

Strangely my objpas doesn't contain an overload with aCodePage.. Is this OS or version specific? I've got Win7, Lazarus 2.0.8.

idk, i copied it from FPC trunk sources

[GreatCorn]

Are you completely sure the file is coded as UTF8?

File was saved with Notepad as UTF-8 (Notepad++ said it's with BOM, so I set it to plain UTF-8 and resaved it, which gave a yet another result (%u0432-?%u0412%u0441), but not the one I'm trying to achieve.
However, I tried just importing the Clipbrd (or even LCLClasses, testing purposes only) unit (LCLBase) and it actually worked. Displayed everything perfectly. Importing the Crt unit changed the result too (but to an another mess of gibberish). Weird.
As for

But is that problem in the reading, or the displaying? the windows console is terribly well at utf8.

, I made the program write to an output file whatever it read by char. The saved file had everything properly encoded and was pretty much identical to the file it read. But my main concern is displaying the text in the console.

[lucamar]

I made the program write to an output file whatever it read by char. The saved file had everything properly encoded and was pretty much identical to the file it read. But my main concern is displaying the text in the console.

So it is a console problem, then. I'm not much versed anymore in Windows console idiosyncrazies, sorry, but this theme arises with surprising regularity in the forum; search for "Windows console UTF-8" and you'll probably find some tips to solve or mitigate the problem.

Or wait for the answer of someone more knowledgeable than I