Recent

Author Topic: fpsStreams unit problems  (Read 1055 times)

AlexTP

  • Hero Member
  • *****
  • Posts: 2287
    • UVviewsoft
fpsStreams unit problems
« on: October 03, 2023, 08:36:45 am »
I got this unit
https://sourceforge.net/p/lazarus-ccr/svn/HEAD/tree/components/fpspreadsheet/source/common/fpsstreams.pas
and replaced TFileStream with it in ATSynEdit.
You can get ATSynEdit and do it too.
Change is in atsynedit/atstrings_save.inc,
procedure TATStrings.SaveToFile.

Code: Pascal  [Select][+][-]
  1.  
  2. procedure TATStrings.SaveToFile(const AFilename: string);
  3. var
  4.   fs: TBufStream;
  5.   NMode: word;
  6. begin
  7.   NMode:= fmOpenReadWrite or fmShareDenyWrite;
  8.   //don't set fmCreate for existing file, to keep NTFS file streams
  9.   if not FileExists(AFilename) then
  10.     NMode:= NMode or fmCreate;
  11.  
  12.   fs:= TBufStream.Create(AFilename, NMode, 64*1024);
  13.   try
  14.     fs.Size:= 0; //important to avoid tail from old contents
  15.     fs.Seek(0, soFromBeginning);
  16.     SaveToStream(fs, FEncoding, IsSavingWithSignature);
  17.   finally
  18.     FreeAndNil(fs);
  19.   end;
  20.  
  21.   DoFinalizeSaving;
  22. end;  
  23.  

Problems when ATSynEdit saves 30M big log file

- before, saving took 10-20 seconds with TFileStream, or 1-2 seconds with TMemoryStream+TFileStream. Now it is forever. ATSynedit saving progressbar stops. Disk file grows to 20G and app freezes.

- you always add fmCreate to FileMode so NTFS file streams are not preserved
« Last Edit: October 03, 2023, 09:08:06 am by AlexTP »

wp

  • Hero Member
  • *****
  • Posts: 11445
Re: fpsStreams unit problems
« Reply #1 on: October 03, 2023, 12:45:10 pm »
I cannot reproduce the issue with the following minimal project:
Code: Pascal  [Select][+][-]
  1. program project1;
  2. uses
  3.   SysUtils, Classes, fpsStreams;
  4. const
  5.   N = 8*1024*1024;  // there are N*4 Bytes in the file --> 32 MB
  6. var
  7.   stream: TStream;
  8.   data: array[0..N-1] of DWord;
  9.   i: Integer;
  10.   t: TDateTime;
  11.   fn: String;
  12. begin
  13.   for i := 0 to N-1 do
  14.     data[i] := i;
  15.   fn := ExtractFilepath(ParamStr(0)) + 'test.dat';
  16.  
  17.   t := Now();
  18.   stream := TBufStream.Create(fn, fmCreate, 64*1024);  // --> 0.09 seconds
  19.   //stream := TFileStream.Create(fn, fmCreate);        // --> 19 seconds
  20.   //stream := TMemoryStream.Create;                      // --> 0.22 seconds
  21.   try
  22.     for i := 0 to N-1 do
  23.       stream.Write(data[i], SizeOf(DWord));
  24.     if stream is TMemoryStream then TMemoryStream(stream).SaveToFile(fn);
  25.   finally
  26.     stream.Free;
  27.   end;
  28.   t := Now() - t;
  29.  
  30.   WriteLn('Done ', FormatDateTime('s.zzz', t), ' seconds.');
  31.   ReadLn;
  32. end.

AlexTP

  • Hero Member
  • *****
  • Posts: 2287
    • UVviewsoft
Re: fpsStreams unit problems
« Reply #2 on: October 03, 2023, 04:00:08 pm »
Maybe you can repro on the maximal project, ie do as I wrote in the 1st post and save the 30M file using ATSynEdit demo.

wp

  • Hero Member
  • *****
  • Posts: 11445
Re: fpsStreams unit problems
« Reply #3 on: October 03, 2023, 04:15:13 pm »
Sorry, I promised myself that I will not try to fix bugs occuring only in third-party software.

AlexTP

  • Hero Member
  • *****
  • Posts: 2287
    • UVviewsoft
Re: fpsStreams unit problems
« Reply #4 on: October 03, 2023, 06:54:41 pm »
If i run this example, i get the file test.log with content
Code: [Select]
line one;
line two;
line end;
which is not expected.
see the code, test.log must have
Code: [Select]
first.
Code
Code: Pascal  [Select][+][-]
  1. {$mode objfpc}
  2. {$h+}
  3.  
  4. uses sysutils, classes, fpsStreams;
  5.  
  6. procedure Save(const filename: string; L: TStringList);
  7. var
  8.   fs: TBufStream;
  9.   NMode: word;
  10.   buf: string;
  11.   i: integer;
  12. begin
  13.   NMode:= fmOpenReadWrite or fmShareDenyWrite;
  14.   //don't set fmCreate for existing file, to keep NTFS file streams
  15.   if not FileExists(filename) then
  16.     NMode:= NMode or fmCreate;
  17.  
  18.   fs:= TBufStream.Create(filename, NMode);
  19.   try
  20.     fs.Size:= 0;
  21.     for i:= 0 to L.Count-1 do
  22.     begin
  23.       buf:= L[i]+#10;
  24.       fs.WriteBuffer(buf[1], length(buf));
  25.     end;
  26.   finally
  27.     FreeAndNil(fs);
  28.   end;
  29. end;
  30.  
  31. var
  32.   L: TStringList;
  33. begin
  34.   L:= TStringList.Create;
  35.   L.Add('line one;');
  36.   L.Add('line two;');
  37.   L.Add('line end;');
  38.   Save('test.log', L);
  39.  
  40.   L.Clear;
  41.   L.Add('first.');
  42.   Save('test.log', L);
  43.  
  44.   L.Free;
  45.  
  46. end.
  47.  
  48.  

AlexTP

  • Hero Member
  • *****
  • Posts: 2287
    • UVviewsoft
Re: fpsStreams unit problems
« Reply #5 on: October 03, 2023, 06:57:49 pm »
(Linux x64, FPC 3.2.3)

wp

  • Hero Member
  • *****
  • Posts: 11445
Re: fpsStreams unit problems
« Reply #6 on: October 03, 2023, 10:44:02 pm »
OK, now I understand. Never had thought of such a requirement to set the Size of the TBufStream...

Please test the new version at ccr.

However, having to consider both buffer state and existing file size, I think it will be very complicated to set the Size to an arbitrary value. Therefore, I decided to allow only a new size of zero; any other value raises an exception. For simplification, there is also a TBufStream.Clear method. (Why does the abstract class TStream declare Size to be public?)

 

TinyPortal © 2005-2018