Recent

Author Topic: TMemoryStreamUTF8 cannot LoadFromFile  (Read 3883 times)

martinrame

  • Full Member
  • ***
  • Posts: 119
Re: TMemoryStreamUTF8 cannot LoadFromFile
« Reply #15 on: June 16, 2022, 03:04:33 am »
A similar program in Python is this:

Code: Pascal  [Select][+][-]
  1. import os
  2.  
  3. # folder path
  4. dir_path = r'/path/to/file/'
  5.  
  6. # Iterate directory
  7. for path in os.listdir(dir_path):
  8.     # check if current path is a file
  9.     if os.path.isfile(os.path.join(dir_path, path)):
  10.         if path.endswith("19900.pdf"):
  11.             f = open(dir_path + path, "rb")
  12.             content = f.read()
  13.             print(content)

And it can open the file without issues.

PascalDragon

  • Hero Member
  • *****
  • Posts: 5444
  • Compiler Developer
Re: TMemoryStreamUTF8 cannot LoadFromFile
« Reply #16 on: June 16, 2022, 11:15:52 am »
Code: Pascal  [Select][+][-]
  1.             with TStringList.Create do  
  2.             begin                                                                                
  3.                LoadFromFile(Dir + SR.Name); // <-- Unable to open file....
  4.             end;
  5.  

Sidenote: this will lead to a memory leak.

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: TMemoryStreamUTF8 cannot LoadFromFile
« Reply #17 on: June 17, 2022, 09:10:16 am »
I did an: ls /directory/file|hexdump -C to see the actual ascii codes of the filename and found the Ñ is replaced to the hex a5, then I did this:

Code: Pascal  [Select][+][-]
  1. lFile := AnsiReplaceStr(lFile, 'Ñ', chr($A5));                                        
  2. lPdfStream.LoadFromFile(lFile);

But still I can't open the file with LoadFromFile.

The ASCII code of 'Ñ' in ISO-8850-1 is D1, not A5. So, if your settings are as you said, they translate the wrong code page.

Find out what code page is used and translate it yourself. There's almost certainly a Free Pascal function somewhere to do that for you.

Josh

  • Hero Member
  • *****
  • Posts: 1270
Re: TMemoryStreamUTF8 cannot LoadFromFile
« Reply #18 on: June 17, 2022, 10:02:58 am »
does SysToUTF8 function in lazutf8 unit solve the problem?

Code: Pascal  [Select][+][-]
  1. LoadFromFile(SysToUTF8(Dir + SR.Name));
The best way to get accurate information on the forum is to post something wrong and wait for corrections.

tetrastes

  • Sr. Member
  • ****
  • Posts: 469
Re: TMemoryStreamUTF8 cannot LoadFromFile
« Reply #19 on: June 17, 2022, 10:14:29 am »
The ASCII code of 'Ñ' in ISO-8850-1 is D1, not A5. So, if your settings are as you said, they translate the wrong code page.
Code for Ñ is $A5 in CP437 and CP850, which are used for Windows console in USA or Western Europe. As I understand, Windows uses console CP for network mounts.

Jorg3000

  • Jr. Member
  • **
  • Posts: 64
Re: TMemoryStreamUTF8 cannot LoadFromFile
« Reply #20 on: June 17, 2022, 11:46:31 am »
Code: Pascal  [Select][+][-]
  1. var lFile: RawByteString;
  2. ...
  3. lFile:= ...
  4. ...
  5. SetCodepage(lFile,Windows.GetConsoleCP,false);
  6. lFile:=Utf8Encode(lFile);
  7.  

PascalDragon

  • Hero Member
  • *****
  • Posts: 5444
  • Compiler Developer
Re: TMemoryStreamUTF8 cannot LoadFromFile
« Reply #21 on: June 17, 2022, 01:27:10 pm »
Code: Pascal  [Select][+][-]
  1. var lFile: RawByteString;
  2. ...
  3. lFile:= ...
  4. ...
  5. SetCodepage(lFile,Windows.GetConsoleCP,false);
  6. lFile:=Utf8Encode(lFile);
  7.  

martinrame is on FreeBSD, so using the Windows unit is not an option.

Jorg3000

  • Jr. Member
  • **
  • Posts: 64
Re: TMemoryStreamUTF8 cannot LoadFromFile
« Reply #22 on: June 17, 2022, 02:44:10 pm »
Quote
martinrame is on FreeBSD, so using the Windows unit is not an option.

Oops, I confused that with another post that was explicitly about Windows. Sorry.

PS: If the codepage is known, e.g. 437 or 850, my code snippet could be used for conversion.

Quote
Code for Ñ is $A5 in CP437 and CP850
« Last Edit: June 17, 2022, 02:56:27 pm by Jorg3000 »

martinrame

  • Full Member
  • ***
  • Posts: 119
Re: TMemoryStreamUTF8 cannot LoadFromFile
« Reply #23 on: June 23, 2022, 09:16:22 pm »
Hi!, thanks for taking some time to study this, and sorry for not showing up (I got the flu and wasn't able to work).

I created a small program that basically does the same as the CGI (read the file name from the database in UTF8, and open that file from the mounted filesystem)...and it worked!.

So, now I'm starting to think the problem is related to the CGI and/or the Apache24 web server.

martinrame

  • Full Member
  • ***
  • Posts: 119
Re: TMemoryStreamUTF8 cannot LoadFromFile
« Reply #24 on: June 23, 2022, 10:27:56 pm »
This is the console program that works:

Code: Pascal  [Select][+][-]
  1. program test;
  2.  
  3. {$mode objfpc}{$H+}
  4. uses
  5.   Classes, SysUtils, pqconnection, sqldb;
  6.  
  7. var
  8.   lStream: TFileStream;
  9.   PQConnection2: TPQConnection;
  10.   SQLTransaction2: TSQLTransaction;
  11.   lQuery: TSqlQuery;
  12. begin
  13.   PQConnection2 := TPQConnection.Create(nil);
  14.   SQLTransaction2 := TSqlTransaction.Create(nil);
  15.   lQuery := TSqlQuery.Create(nil);
  16.   try
  17.     PQConnection2.HostName:= '192.168.0.130';
  18.     PQConnection2.DatabaseName:= 'app2';
  19.     PQConnection2.UserName:= '******;
  20.    PQConnection2.Password:= '******';
  21.    PQConnection2.Transaction:= SQLTransaction2;
  22.    PQConnection2.Connected:= True;
  23.    lQuery.Database := PQConnection2;
  24.    lQuery.SQL.Text := 'select ' +
  25.      'replace(' +
  26.      '   replace(' +
  27.      '      replace(' +
  28.      '         archivo,''\\'', ''\''),' +
  29.      '      ''SERVIDOR\Sistema\Proyecto'',''mnt/assist''),' +
  30.      '   ''\'',''/'') as archivo ' +
  31.      'from vista_pacs where idsld_pedidoestudio_det=:accession';
  32.    lQuery.ParamByName('accession').AsInteger := StrToInt(argv[1]);
  33.    lQuery.Open;
  34.    lStream := TFileStream.Create(lQuery.FieldByName('archivo').AsString, fmOpenRead);
  35.    Writeln(lStream.Size);
  36.    lStream.Free;
  37.  finally
  38.    lQuery.Free;
  39.    SQLTransaction2.Free;
  40.    PQConnection2.Free;
  41.  end;
  42. end.

And this is the unit, part of the CGI (using Brook Framework), that doesn't work:

Code: Pascal  [Select][+][-]
  1. unit acttest;  
  2.                                                                                            
  3. {$mode objfpc}{$H+}                
  4.                                              
  5. interface                    
  6.                                              
  7. uses                                                                                        
  8.   BrookAction,                                                                              
  9.   Classes,                  
  10.   SysUtils,
  11.   sqldb,        
  12.   dm;
  13.                                              
  14. type
  15.                                              
  16.   TActPDFOnTheFlyTEST = class(TBrookAction)                                                
  17.   private
  18.   public
  19.     procedure Get; override;                                                                
  20.   end;
  21.  
  22. implementation
  23.  
  24. procedure TActPDFOnTheFlyTEST.Get;
  25. var
  26.   lPdfStream: TFileStream;
  27.   lAccession: string;
  28.   lQuery: TSqlQuery;
  29.   lFile: string;
  30. begin
  31.   lQuery := TSQLQuery.Create(nil);
  32.   try
  33.     lAccession := Variable['accession'];
  34.     lQuery.DataBase := DataModule1.PQConnection2;
  35.     lQuery.SQL.Text := 'select ' +
  36.       'replace(' +
  37.       '   replace(' +
  38.       '      replace(' +
  39.       '         archivo,''\\'', ''\''),' +
  40.       '      ''SERVIDOR\Sistema\Proyecto'',''mnt/assist''),' +
  41.       '   ''\'',''/'') as archivo ' +
  42.       'from vista_pacs where idsld_pedidoestudio_det=:accession';
  43.     lQuery.ParamByName('accession').AsInteger := StrToInt(lAccession);
  44.     lQuery.Open;
  45.     lPdfStream := TFileStream.Create(lQuery.FieldByName('archivo').AsString, fmOpenRead);
  46.     raise exception.create('aaaa'); // this is never reached, because the previous line returns Unable to open file ... No such file or directory on files with Ñ.
  47.     //lPdfStream.LoadFromFile(lFile);
  48.     lPdfStream.Position := 0;
  49.     HttpResponse.ContentStream := lPdfStream;
  50.     HttpResponse.ContentType:='application/pdf';
  51.     HttpResponse.ContentLength:= lPdfStream.Size;
  52.     HttpResponse.SendContent;
  53.   finally
  54.     lQuery.Free;
  55.   end;
  56. end;
  57.  
  58. initialization
  59.   TActPDFOnTheFlyTEST.Register('/pdfontheflytest/:accession');
  60.  
  61. end.
  62.  

The partial conclusion is the CGI has some limitation to opening this the file, but it can open files without Ñ wihout issues.
« Last Edit: June 23, 2022, 10:32:04 pm by martinrame »

martinrame

  • Full Member
  • ***
  • Posts: 119
Re: TMemoryStreamUTF8 cannot LoadFromFile
« Reply #25 on: June 24, 2022, 12:20:42 am »
I'm re-trying with cwstring, as suggested by PascalDragon many replies ago. And it seems to try opening the file, but the program hangs (100% cpu usage...) on this line:

Code: Pascal  [Select][+][-]
  1. lPdfStream := TFileStream.Create(lQuery.FieldByName('archivo').AsString, fmOpenRead);

martinrame

  • Full Member
  • ***
  • Posts: 119
Re: TMemoryStreamUTF8 cannot LoadFromFile
« Reply #26 on: June 24, 2022, 12:43:45 am »
As the original file is in a SMB filesystem I copied it to my home directory and adapted the program to read from there, but now I get:

Code: Pascal  [Select][+][-]
  1. 500 - Internal server error
  2.  
  3. @error

What's that @error?

martinrame

  • Full Member
  • ***
  • Posts: 119
Re: TMemoryStreamUTF8 cannot LoadFromFile
« Reply #27 on: June 24, 2022, 12:45:34 am »
Is the default template for the Brook Framework's 500 error.

Thaddy

  • Hero Member
  • *****
  • Posts: 14169
  • Probably until I exterminate Putin.
Re: TMemoryStreamUTF8 cannot LoadFromFile
« Reply #28 on: June 24, 2022, 10:45:09 am »
TmemorystreamUTF8 is a nonsense. A stream is a stream and always byte sized.
I would have fired the author. Simply because of proven lack of knowledge.
Specialize a type, not a var.

bytebites

  • Hero Member
  • *****
  • Posts: 625
Re: TMemoryStreamUTF8 cannot LoadFromFile
« Reply #29 on: June 24, 2022, 10:56:21 am »
TmemorystreamUTF8...
Quote
Implements a memory stream which supports file names with UTF-8 encoding

 

TinyPortal © 2005-2018