Recent

Author Topic: TFileStream, read files larger than 2gig  (Read 3505 times)

Thaddy

  • Hero Member
  • *****
  • Posts: 14373
  • Sensorship about opinions does not belong here.
Re: TFileStream, read files larger than 2gig
« Reply #15 on: June 15, 2019, 10:47:28 pm »
Indeed.
Object Pascal programmers should get rid of their "component fetish" especially with the non-visuals.

lucamar

  • Hero Member
  • *****
  • Posts: 4219
Re: TFileStream, read files larger than 2gig
« Reply #16 on: June 16, 2019, 03:13:24 am »
I did an actual test in a 32 bits machine (Pentium 4) with Ubuntu 12 - i686 and Lazarus 2.0.2 (Gtk2).

First wrote a big (2.1 GiB) file by old-style means and then read it with a TFileStream. Worked alright. Slooow, but read it OK from beginning to end.

This is (part of) the code I used:

Code: Pascal  [Select][+][-]
  1. type
  2.   TTestRecord = packed record
  3.     AnInt: Integer;
  4.     AString: ShortString;
  5.   end;
  6.  
  7. const
  8.   Limit: Int64 = MaxLongint;
  9.   Filename = 'testfile.dat';
  10.  
  11. procedure TMainForm.DoTheTest(Sender: TObject);
  12. var
  13.   AFile: File of TTestRecord;
  14.   ARecord: TTestRecord;
  15.   Current, Count, Total: LongInt;
  16. begin
  17.   {Make it a one-shot!}
  18.   if Sender.InheritsFrom(TButton) then
  19.     (Sender as TButton).Enabled := False;
  20.  
  21.   {Some initializations}
  22.   Memo.Clear;
  23.   ARecord.AString := StringOfChar(' ', SizeOf(ShortString));
  24.  
  25.   { If there is not already one or it's too small,
  26.     create a 2GiB+ file the old-style way }
  27.   if not FileExists(Filename) or (FileSize(Filename) < Limit) then begin
  28.     AssignFile(AFile, Filename);
  29.     Rewrite(AFile);
  30.     try
  31.       Current := 0;
  32.       while FileSize(Filename) < Limit do begin
  33.         Inc(Current);
  34.         ARecord.AnInt := Current;
  35.         ARecord.AString := Current.ToString;
  36.         Write(AFile, ARecord);
  37.         if Current mod 1000 = 0 then begin { Don't waste much time in UI }
  38.           StatusBar.SimpleText := 'Written record #'+Current.ToString;
  39.           Application.ProcessMessages;
  40.           if Application.Terminated then Exit;
  41.         end;
  42.       end;
  43.     finally
  44.       CloseFile(AFile);
  45.     end;
  46.   end else begin
  47.     Total := FileSize(Filename) div SizeOf(ARecord);
  48.     StatusBar.SimpleText := Format('Seems to be %d records', [Total]);
  49.   end;
  50.  
  51.   Application.ProcessMessages; {Tidy-up UI}
  52.  
  53.   {Now test reading it with a stream}
  54.   Current := 0;
  55.   FStream := TFileStream.Create(Filename, fmOpenRead);
  56.   try
  57.     repeat
  58.       Inc(Current);
  59.       Count := FStream.Read(ARecord, SizeOf(TTestRecord));
  60.       if Count > 0 then begin
  61.         if Current mod 2000 = 0 then
  62.           Memo.Clear;  {Let's not waste too much memory ...}
  63.         Memo.Lines.Add('Read # %d (%d bytes): %d - %s',
  64.                        [Current, Count, ARecord.AnInt, ARecord.AString]);
  65.         if Current mod 1000 = 0 then begin  { Don't waste much updating UI }
  66.           Memo.CaretPos := Point(0, Memo.Lines.Count-1);
  67.           Application.ProcessMessages;
  68.           if Application.Terminated then Exit;
  69.         end;
  70.       end; {if Count > 0 }
  71.     until (FStream.Position >= FStream.Size) or (Count <= 0);
  72.   finally
  73.     FStream.Free;
  74.   end;
  75.   Memo.Lines.Add('----- DONE!!!');
  76.   Memo.SelStart := Length(Memo.Lines.Text);
  77. end;

Attached images of the results.

Note: I did reuse another test project, thence the (not relevant) check group; just ignore it
« Last Edit: June 16, 2019, 03:22:47 am by lucamar »
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus/FPC 2.0.8/3.0.4 & 2.0.12/3.2.0 - 32/64 bits on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: TFileStream, read files larger than 2gig
« Reply #17 on: June 16, 2019, 03:32:35 am »
Not sure, but does not seem to be over 2GB by more than part of a record.

Limit is maxLongint  = $7fffffff

Number of records is 8259553 with 260 bytes each leads to a file of 8259553*260=$80000084 compare it to the size of one record $104



lucamar

  • Hero Member
  • *****
  • Posts: 4219
Re: TFileStream, read files larger than 2gig
« Reply #18 on: June 16, 2019, 05:05:58 am »
Not sure, but does not seem to be over 2GB by more than part of a record.

It can't be much, given the record size. The question is that it's bigger than 2 GiB, which was the limit the OP posted about; whether it's by little or by much shouldn't matter: it should fail at lest in the last record, but it doesn't.

I could try increasing the limit to 2,25 GiB, if you want, but that will have to wait until late tomorrow.

ETA: OK, I did a small quick test passing well over the 2 GiB barrier. It did work (see attached image).
« Last Edit: June 16, 2019, 05:48:25 am by lucamar »
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus/FPC 2.0.8/3.0.4 & 2.0.12/3.2.0 - 32/64 bits on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

 

TinyPortal © 2005-2018