Recent

Author Topic: TFileStream file size limit  (Read 14400 times)

lagprogramming

  • Sr. Member
  • ****
  • Posts: 405
TFileStream file size limit
« on: October 04, 2021, 10:45:30 am »
   Hi!
   When trying to save a file in a linux-x86_64 environment, an application returns an error. I've noticed that the error appears when the file exceeds the maximum value of a longint. Looking at the assembler window of Lazarus I've assumed that the error was related to the procedure TStream.WriteBuffer(const Buffer; Count: Longint); that is found in rtl/objpas/classes/streams.inc.
Because the situation appears to be simmilar I've modified both TStream.WriteBuffer(const Buffer; Count: Longint) and TStream.ReadBuffer(var Buffer; Count: Longint);
from
Code: Pascal  [Select][+][-]
  1. procedure TStream.ReadBuffer(var Buffer; Count: Longint);
  2.  
  3. Var
  4.   r,t : longint;
  5.  
  6. begin
  7.   t:=0;
  8.   repeat
  9.     r:=Read(PByte(@Buffer)[t],Count-t);
  10.     inc(t,r);
  11.   until (t=Count) or (r<=0);
  12.   if (t<Count) then
  13.     Raise EReadError.Create(SReadError);
  14. end;
  15.  
  16. procedure TStream.WriteBuffer(const Buffer; Count: Longint);
  17.  
  18. var
  19.   r,t : Longint;
  20.  
  21.   begin
  22.     T:=0;
  23.     Repeat
  24.        r:=Write(PByte(@Buffer)[t],Count-t);
  25.        inc(t,r);
  26.     Until (t=count) or (r<=0);
  27.     if (t<Count) then
  28.        Raise EWriteError.Create(SWriteError);
  29.   end;
to
Code: Pascal  [Select][+][-]
  1. procedure TStream.ReadBuffer(var Buffer; Count: NativeInt);
  2.  
  3. Var
  4.   r,t : NativeInt;
  5.  
  6. begin
  7.   t:=0;
  8.   repeat
  9.     r:=Count-t;
  10.     if r>maxlongint then r:=maxlongint;
  11.     r:=Read(PByte(@Buffer)[t],r);
  12.     inc(t,r);
  13.   until (t=Count) or (r<=0);
  14.   if (t<Count) then
  15.     Raise EReadError.Create(SReadError);
  16. end;
  17.  
  18. procedure TStream.WriteBuffer(const Buffer; Count: NativeInt);
  19.  
  20. var
  21.   r,t : NativeInt;
  22.  
  23.   begin
  24.     T:=0;
  25.     Repeat
  26.        r:=Count-t;
  27.        if r>maxlongint then r:=maxlongint;
  28.        r:=Write(PByte(@Buffer)[t],r);
  29.        inc(t,r);
  30.     Until (t=count) or (r<=0);
  31.     if (t<Count) then
  32.        Raise EWriteError.Create(SWriteError);
  33.   end;
   I've built fpc clean and after that I've rebuilt Lazarus clean but the application returned the same error when trying to write files with a size greater than the maximum value of a longint. I've looked at the assembler window of Lazarus again and to my surprise I've seen a "CLASSES$_$TSTREAM_$__$$_WRITEBUFFER$formal$LONGINT" line followed by the same assembly code. It's like I haven't changed anything in the streams.inc file.

   What am I doing wrong, how comes that the assembly code remained the same? Remember that it's a x86_64 target. I'm using relatively new fpc and Lazarus sources and the error in the application appears at a TFilestream.WriteBuffer routine. Could it be that an optimized assembly procedure is used instead of a generic pascal-written procedure!? By the way, I think there is or was a bug report related to this situation, the new interface of the bug tracking system is...new to me :-[.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: TFileStream file size limit
« Reply #1 on: October 04, 2021, 11:05:59 am »
There are two issues here:

  • Using files with a 64-bit size
  • reading/writing blocks that are 64-bit (>2 or 4GB) in one go

The first should be ok, the second is not yet supported. Afaik the variable declaration was considered not worth it, and many 64-bit targets also don't support it. So since the safest blocksize is probably 2GB-1, easiest is to write in one GB increments.

p.s. It might you hit some overload issue somewhere with your recompile.
« Last Edit: October 04, 2021, 11:09:43 am by marcov »

lagprogramming

  • Sr. Member
  • ****
  • Posts: 405
Re: TFileStream file size limit
« Reply #2 on: October 04, 2021, 02:55:54 pm »
   The following code fails to write to file
Code: Pascal  [Select][+][-]
  1. procedure writefail;
  2. var
  3.   x:TFilestream;
  4.   p:pointer;
  5. begin
  6.   p:=getmem(maxlongint+1);
  7.   x:=TFilestream.Create('/home/user/bin.txt',fmcreate);
  8.   x.WriteBuffer(p^,maxlongint+1);//Fails to write garbage in linux-x86_64
  9.   x.Free;
  10. end;

   If procedure TStream.ReadBuffer(var Buffer; Count: Longint) and procedure TStream.WriteBuffer(const Buffer; Count: Longint) should be left intact then wouldn't be a good idea to modify the following two procedures?
   
Code: Pascal  [Select][+][-]
  1. procedure TStream.ReadBuffer(var Buffer: TBytes; Offset, Count: NativeInt);
  2. begin
  3.   ReadBuffer(Buffer[OffSet],Count);
  4. end;
  5.  
  6. procedure TStream.WriteBuffer(const Buffer: TBytes; Offset, Count: NativeInt);
  7. begin
  8.   WriteBuffer(Buffer[Offset],Count);
  9. end;

   If the above code would be modified adding a repeat/while loop, fpc might have two routines that work as expected. In my point of view it should be better than writing new procedures within local units of developing projects, procedures that would try to do what a developer expects from the above procedure declarations.

korba812

  • Sr. Member
  • ****
  • Posts: 392
Re: TFileStream file size limit
« Reply #3 on: October 04, 2021, 03:28:12 pm »
   The following code fails to write to file
Code: Pascal  [Select][+][-]
  1. procedure writefail;
  2. var
  3.   x:TFilestream;
  4.   p:pointer;
  5. begin
  6.   p:=getmem(maxlongint+1);
  7.   x:=TFilestream.Create('/home/user/bin.txt',fmcreate);
  8.   x.WriteBuffer(p^,maxlongint+1);//Fails to write garbage in linux-x86_64
  9.   x.Free;
  10. end;
The result of "MaxLongint + 1" is Int64. Try "MaxLongint - 1".

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: TFileStream file size limit
« Reply #4 on: October 04, 2021, 04:00:40 pm »
If the function has a type that is limited to 2GB-1, why he would try 2GB or more is a mystery to me.   

Note that the upper limit is not 64-bit, but how large a block an application can allocate , which depends on a lot of things.

Anyway, afaik there are reports for them, search them and comment on them.

Keep in mind that  adding a loop to those functions adds code for people using this for short writes, just to benefit the few people creating a disk/ssd exerciser/benchmark or so doing gigantic writes.

Thaddy

  • Hero Member
  • *****
  • Posts: 14205
  • Probably until I exterminate Putin.
Re: TFileStream file size limit
« Reply #5 on: October 04, 2021, 04:31:14 pm »
AFAIK Address space is limited to high(qword), at lot.., but single read/writes are limited to high(nativeint) which is indeed 2G
Note that in practise it limits to available memrory otherwise a EOutOfMemory is thrown.
Note that for 32 bit WIN there is a PE flag - since windows 7 - that extends available memory to 4G but still with the limitation ffor single read/writes
« Last Edit: October 04, 2021, 04:37:29 pm by Thaddy »
Specialize a type, not a var.

Thaddy

  • Hero Member
  • *****
  • Posts: 14205
  • Probably until I exterminate Putin.
Re: TFileStream file size limit
« Reply #6 on: October 04, 2021, 04:39:33 pm »
The result of "MaxLongint + 1" is Int64. Try "MaxLongint - 1".
In Freepascal simply use High(<type>);
Specialize a type, not a var.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: TFileStream file size limit
« Reply #7 on: October 04, 2021, 04:53:00 pm »
AFAIK Address space is limited to high(qword), at lot.., but single read/writes are limited to high(nativeint) which is indeed 2G

size_t does not need to cover the whole address space. If the kernel doesn't hand out such large consecutive blocks to applications, it is only a waste of bits.


Thaddy

  • Hero Member
  • *****
  • Posts: 14205
  • Probably until I exterminate Putin.
Re: TFileStream file size limit
« Reply #8 on: October 04, 2021, 05:06:44 pm »
true. btw I should have written High(NativeInt). And that on 32 bit the flag for windows is IMAGE_FILE_LARGE_ADDRESS_AWARE. I don't know if you defined those consts in windows.pas.
See MSDN or https://docwiki.embarcadero.com/RADStudio/Alexandria/en/PE_(portable_executable)_header_flags_(Delphi)

The value of the flag is $20
« Last Edit: October 04, 2021, 05:14:22 pm by Thaddy »
Specialize a type, not a var.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: TFileStream file size limit
« Reply #9 on: October 04, 2021, 05:08:44 pm »
Image header structs and constants have been reworked one or two years ago.

Doesn't mean it will be perfect, but most should be there, including fairly recent stuff.

Thaddy

  • Hero Member
  • *****
  • Posts: 14205
  • Probably until I exterminate Putin.
Re: TFileStream file size limit
« Reply #10 on: October 04, 2021, 05:15:11 pm »
Yes, you are doing a good job there. Tnx
Specialize a type, not a var.

lagprogramming

  • Sr. Member
  • ****
  • Posts: 405
Re: TFileStream file size limit
« Reply #11 on: October 04, 2021, 05:51:10 pm »
   Targeting linux-x86_64 we have:
sizeof(nativeint)=sizeof(sizeuint)=sizeof(int64)=8
high(nativeint)=high(int64)
sizeof(longint)=sizeof(longword)=4

   In streams.inc we have:
procedure TStream.ReadBuffer(var Buffer: TBytes; Offset, Count: NativeInt);
procedure TStream.WriteBuffer(const Buffer: TBytes; Offset, Count: NativeInt);

   The purpose of having these two procedures declared in rtl knowing that it's guaranteed to have a failure if you pass a Count value greater than maxlongint remains a mistery to me.
   Thank you for your patience.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: TFileStream file size limit
« Reply #12 on: October 04, 2021, 05:57:22 pm »
   In streams.inc we have:
procedure TStream.ReadBuffer(var Buffer: TBytes; Offset, Count: NativeInt);
procedure TStream.WriteBuffer(const Buffer: TBytes; Offset, Count: NativeInt);

I missed that, it is still longint in 3.2.x.  I don't know who changed that (michael?) Please file a bug for the trunk issue.

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: TFileStream file size limit
« Reply #13 on: October 05, 2021, 09:15:18 am »
AFAIK Address space is limited to high(qword), at lot.., but single read/writes are limited to high(nativeint) which is indeed 2G

High(NativeInt) on a 64-bit system is equal to High(Int64).

Thaddy

  • Hero Member
  • *****
  • Posts: 14205
  • Probably until I exterminate Putin.
Re: TFileStream file size limit
« Reply #14 on: October 05, 2021, 09:47:15 am »
In de context of 32 bit, Sarah. That's why I suggested it to express address space.
Specialize a type, not a var.

 

TinyPortal © 2005-2018