Recent

Author Topic: TFileStream file size limit  (Read 14365 times)

lagprogramming

  • Sr. Member
  • ****
  • Posts: 404
Re: TFileStream file size limit
« Reply #30 on: January 22, 2022, 05:27:30 pm »
I think this is a question of a separate topic. In Windows, a zero file of 3 GB is written.

I've double checked. It's not a linux file system limitation. Development fpc sources(trunc) have been used, but it's an old problem.
If SaveToFile calls something like "WriteBuffer(Memory^,Size);" where Size is int64 on x86_64 targets then the rtl devs might have hit the same problem as I did, meaning that they might have used a routine limited to longints. In this topic I've suggested adding some overloaded functions for 64bit integers. Anyway, the forum moderators and the rtl developers should know about this situation. Maybe this situation is encountered only when targeting linux-x86_64, I don't know.

AlexTP

  • Hero Member
  • *****
  • Posts: 2365
    • UVviewsoft
Re: TFileStream file size limit
« Reply #31 on: January 22, 2022, 05:29:52 pm »

ASerge

  • Hero Member
  • *****
  • Posts: 2212
Re: TFileStream file size limit
« Reply #32 on: January 22, 2022, 08:59:58 pm »
If SaveToFile calls something like "WriteBuffer(Memory^,Size);" where Size is int64 on x86_64 targets then the rtl devs might have hit the same problem as I did
The WriteBuffer function has a limitation, but the SaveToFile function does not.
Code: Pascal  [Select][+][-]
  1. {$MODE OBJFPC}
  2.  
  3. uses Classes;
  4.  
  5. var
  6.   M: TMemoryStream;
  7. begin
  8.   M := TMemoryStream.Create;
  9.   try
  10.     M.Size := LongWord(3 * 1024 * 1024 * 1024); // Even as LongWord
  11.     M.SaveToFile('3Gb.bin');
  12.   finally
  13.     M.Free;
  14.   end;
  15. end.
File 3Gb.bin has a size of 3 GB. FPC 3.2.2 x64, Windows.

AlexTP

  • Hero Member
  • *****
  • Posts: 2365
    • UVviewsoft
Re: TFileStream file size limit
« Reply #33 on: January 22, 2022, 09:42:01 pm »
@ASerge
It's strange, because I have the same test code, and I got 0-sized file.
Code: Pascal  [Select][+][-]
  1. uses SysUtils, Classes;
  2.  
  3. procedure WriteGarbage(const fn: string; const aSize: Int64);
  4. var
  5.   s:TMemoryStream;
  6. begin
  7.   s:=TMemoryStream.Create;
  8.   s.SetSize(aSize);
  9.   s.SaveToFile(fn);
  10.   s.Free;
  11. end;
  12.  
  13. begin
  14.   WriteGarbage('3g.bin', Int64(3)*1024*1024*1024);
  15. end.
  16.  

lagprogramming

  • Sr. Member
  • ****
  • Posts: 404
Re: TFileStream file size limit
« Reply #34 on: January 24, 2022, 07:25:45 pm »
Can you take a look at the "Small safety precaution" (042eb7e8) commit.
https://gitlab.com/freepascal.org/fpc/source/-/commit/042eb7e8c196b6a0c8f6016af9985b8b3062703b
The first silent error that remains undetected if when more bytes than intended are read or written. This one is very nasty because if you write data backwards it may write over the data that has been previously written. The second silent error is when the Read/Write functions return errors(The r<=0 and w<=0 conditions).

Code: Pascal  [Select][+][-]
  1. procedure TStream.ReadBuffer(var Buffer; Count: NativeInt);
  2. var
  3.   r,t: NativeInt;
  4. begin
  5.   t:=0;
  6.   repeat
  7.     r:=Count-t;
  8.     if r>High(Longint) then r:=High(Longint);
  9.     r:=Read(PByte(@Buffer)[t],r);
  10.     inc(t,r);
  11.   until (t>=Count) or (r<=0);
  12.   if (t<Count) then
  13.     raise EReadError.Create(SReadError);
  14. end;
  15.  
  16. procedure TStream.WriteBuffer(const Buffer; Count: NativeInt);
  17. var
  18.   w,t: NativeInt;
  19. begin
  20.   t:=0;
  21.   repeat
  22.     w:=Count-t;
  23.     if w>High(Longint) then w:=High(Longint);
  24.     w:=Write(PByte(@Buffer)[t],w);
  25.     inc(t,w);
  26.   until (t>=count) or (w<=0);
  27.   if (t<Count) then
  28.     raise EWriteError.Create(SWriteError);
  29. end;
In the rtl documentation I have, regarding the Write and Read functions of TStreams and TStreams descendants, I can't find anything written about the situation when the TStream descendant reads/writes more than the Count parameter value. When looking at TStream.ReadBuffer and TStream.WriteBuffer I've found the following texts: "ReadBuffer reads Count bytes of the stream into Buffer. If the stream does not contain Count bytes, then an exception is raised." and "WriteBuffer writes Count bytes to the stream from Buffer. If the stream does not allow Count bytes to be written, then an exception is raised.". Does the quoted texts mean that ReadBuffer and WriteBuffer should raise exceptions if we try to read/write a less or equal to zero amount of bytes!?
« Last Edit: January 25, 2022, 06:34:26 pm by lagprogramming »

ASerge

  • Hero Member
  • *****
  • Posts: 2212
Re: TFileStream file size limit
« Reply #35 on: January 24, 2022, 07:29:06 pm »
It's strange, because I have the same test code, and I got 0-sized file.
Your code also successfully creates a 3 GB file. Maybe there is no disk space, or you did not wait for the end of the program?

AlexTP

  • Hero Member
  • *****
  • Posts: 2365
    • UVviewsoft
Re: TFileStream file size limit
« Reply #36 on: January 24, 2022, 07:45:21 pm »
@ASerge, with the last FPC modification, it works good (I tested before this, with the Old Write() which failed on 3Gb size).

lagprogramming

  • Sr. Member
  • ****
  • Posts: 404
Re: TFileStream file size limit
« Reply #37 on: January 25, 2022, 06:30:50 pm »
@Alextp & @ASerge
I've updated the FPC development sources and now stream data can be saved in files larger than high(longint), at least on linux-x86_64. So, to me it appears like the problem has been solved.
Good job!

Thaddy

  • Hero Member
  • *****
  • Posts: 14162
  • Probably until I exterminate Putin.
Re: TFileStream file size limit
« Reply #38 on: January 25, 2022, 06:57:25 pm »
Hmm....
A stream is always looking forward (any look-backs are Aberration by theory). A windowed stream on top of a file has unlimited size (upto the ordinal for which it is declared). That means that overlapped files applied to a windowed stream are not limited by any size and can be as large as the disk size allows. Strange nobody mentioned that. It is just basic computer science, very basic. A windowed stream can be implemented as a descendant of TStream or TFileStream.

Just my 2 cents. And OS independent. I am sure this is already implemented somewhere by someone in Pascal.
« Last Edit: January 25, 2022, 07:01:50 pm by Thaddy »
Specialize a type, not a var.

 

TinyPortal © 2005-2018