Forum > FPC development

TFileStream file size limit

(1/8) > >>

lagprogramming:
   Hi!
   When trying to save a file in a linux-x86_64 environment, an application returns an error. I've noticed that the error appears when the file exceeds the maximum value of a longint. Looking at the assembler window of Lazarus I've assumed that the error was related to the procedure TStream.WriteBuffer(const Buffer; Count: Longint); that is found in rtl/objpas/classes/streams.inc.
Because the situation appears to be simmilar I've modified both TStream.WriteBuffer(const Buffer; Count: Longint) and TStream.ReadBuffer(var Buffer; Count: Longint);
from

--- Code: Pascal  [+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i < x.length; i++) { x[i].style.maxHeight='none'; x[i].style.height = Math.min(x[i].clientHeight+15,306)+'px'; x[i].style.resize = "vertical";}};} ---procedure TStream.ReadBuffer(var Buffer; Count: Longint); Var  r,t : longint; begin  t:=0;  repeat    r:=Read(PByte(@Buffer)[t],Count-t);    inc(t,r);  until (t=Count) or (r<=0);  if (t<Count) then    Raise EReadError.Create(SReadError);end; procedure TStream.WriteBuffer(const Buffer; Count: Longint); var  r,t : Longint;   begin    T:=0;    Repeat       r:=Write(PByte(@Buffer)[t],Count-t);       inc(t,r);    Until (t=count) or (r<=0);    if (t<Count) then       Raise EWriteError.Create(SWriteError);  end;to

--- Code: Pascal  [+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i < x.length; i++) { x[i].style.maxHeight='none'; x[i].style.height = Math.min(x[i].clientHeight+15,306)+'px'; x[i].style.resize = "vertical";}};} ---procedure TStream.ReadBuffer(var Buffer; Count: NativeInt); Var  r,t : NativeInt; begin  t:=0;  repeat    r:=Count-t;    if r>maxlongint then r:=maxlongint;    r:=Read(PByte(@Buffer)[t],r);    inc(t,r);  until (t=Count) or (r<=0);  if (t<Count) then    Raise EReadError.Create(SReadError);end; procedure TStream.WriteBuffer(const Buffer; Count: NativeInt); var  r,t : NativeInt;   begin    T:=0;    Repeat       r:=Count-t;       if r>maxlongint then r:=maxlongint;       r:=Write(PByte(@Buffer)[t],r);       inc(t,r);    Until (t=count) or (r<=0);    if (t<Count) then       Raise EWriteError.Create(SWriteError);  end;   I've built fpc clean and after that I've rebuilt Lazarus clean but the application returned the same error when trying to write files with a size greater than the maximum value of a longint. I've looked at the assembler window of Lazarus again and to my surprise I've seen a "CLASSES$_$TSTREAM_$__$$_WRITEBUFFER$formal$LONGINT" line followed by the same assembly code. It's like I haven't changed anything in the streams.inc file.

   What am I doing wrong, how comes that the assembly code remained the same? Remember that it's a x86_64 target. I'm using relatively new fpc and Lazarus sources and the error in the application appears at a TFilestream.WriteBuffer routine. Could it be that an optimized assembly procedure is used instead of a generic pascal-written procedure!? By the way, I think there is or was a bug report related to this situation, the new interface of the bug tracking system is...new to me :-[.

marcov:
There are two issues here:


* Using files with a 64-bit size
* reading/writing blocks that are 64-bit (>2 or 4GB) in one go
The first should be ok, the second is not yet supported. Afaik the variable declaration was considered not worth it, and many 64-bit targets also don't support it. So since the safest blocksize is probably 2GB-1, easiest is to write in one GB increments.

p.s. It might you hit some overload issue somewhere with your recompile.

lagprogramming:
   The following code fails to write to file

--- Code: Pascal  [+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i < x.length; i++) { x[i].style.maxHeight='none'; x[i].style.height = Math.min(x[i].clientHeight+15,306)+'px'; x[i].style.resize = "vertical";}};} ---procedure writefail;var  x:TFilestream;  p:pointer;begin  p:=getmem(maxlongint+1);  x:=TFilestream.Create('/home/user/bin.txt',fmcreate);  x.WriteBuffer(p^,maxlongint+1);//Fails to write garbage in linux-x86_64  x.Free;end;
   If procedure TStream.ReadBuffer(var Buffer; Count: Longint) and procedure TStream.WriteBuffer(const Buffer; Count: Longint) should be left intact then wouldn't be a good idea to modify the following two procedures?
   

--- Code: Pascal  [+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i < x.length; i++) { x[i].style.maxHeight='none'; x[i].style.height = Math.min(x[i].clientHeight+15,306)+'px'; x[i].style.resize = "vertical";}};} ---procedure TStream.ReadBuffer(var Buffer: TBytes; Offset, Count: NativeInt);begin  ReadBuffer(Buffer[OffSet],Count);end; procedure TStream.WriteBuffer(const Buffer: TBytes; Offset, Count: NativeInt);begin  WriteBuffer(Buffer[Offset],Count);end;
   If the above code would be modified adding a repeat/while loop, fpc might have two routines that work as expected. In my point of view it should be better than writing new procedures within local units of developing projects, procedures that would try to do what a developer expects from the above procedure declarations.

korba812:

--- Quote from: lagprogramming on October 04, 2021, 02:55:54 pm ---   The following code fails to write to file

--- Code: Pascal  [+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i < x.length; i++) { x[i].style.maxHeight='none'; x[i].style.height = Math.min(x[i].clientHeight+15,306)+'px'; x[i].style.resize = "vertical";}};} ---procedure writefail;var  x:TFilestream;  p:pointer;begin  p:=getmem(maxlongint+1);  x:=TFilestream.Create('/home/user/bin.txt',fmcreate);  x.WriteBuffer(p^,maxlongint+1);//Fails to write garbage in linux-x86_64  x.Free;end;
--- End quote ---
The result of "MaxLongint + 1" is Int64. Try "MaxLongint - 1".

marcov:
If the function has a type that is limited to 2GB-1, why he would try 2GB or more is a mystery to me.   

Note that the upper limit is not 64-bit, but how large a block an application can allocate , which depends on a lot of things.

Anyway, afaik there are reports for them, search them and comment on them.

Keep in mind that  adding a loop to those functions adds code for people using this for short writes, just to benefit the few people creating a disk/ssd exerciser/benchmark or so doing gigantic writes.

Navigation

[0] Message Index

[#] Next page

Go to full version