fpsStreams unit problems

[AlexTP]

I got this unit
https://sourceforge.net/p/lazarus-ccr/svn/HEAD/tree/components/fpspreadsheet/source/common/fpsstreams.pas
and replaced TFileStream with it in ATSynEdit.
You can get ATSynEdit and do it too.
Change is in atsynedit/atstrings_save.inc,
procedure TATStrings.SaveToFile.

Code: Pascal  [Select][+][-]

1.  
2. procedure TATStrings.SaveToFile(const AFilename: string);
3. var
4.   fs: TBufStream;
5.   NMode: word;
6. begin
7.   NMode:= fmOpenReadWrite or fmShareDenyWrite;
8.   //don't set fmCreate for existing file, to keep NTFS file streams
9.   if not FileExists(AFilename) then
10.     NMode:= NMode or fmCreate;
11.  
12.   fs:= TBufStream.Create(AFilename, NMode, 64*1024);
13.   try
14.     fs.Size:= 0; //important to avoid tail from old contents
15.     fs.Seek(0, soFromBeginning);
16.     SaveToStream(fs, FEncoding, IsSavingWithSignature);
17.   finally
18.     FreeAndNil(fs);
19.   end;
20.  
21.   DoFinalizeSaving;
22. end;  
23.  

Problems when ATSynEdit saves 30M big log file

- before, saving took 10-20 seconds with TFileStream, or 1-2 seconds with TMemoryStream+TFileStream. Now it is forever. ATSynedit saving progressbar stops. Disk file grows to 20G and app freezes.

- you always add fmCreate to FileMode so NTFS file streams are not preserved

[wp]

I cannot reproduce the issue with the following minimal project:

Code: Pascal  [Select][+][-]

1. program project1;
2. uses
3.   SysUtils, Classes, fpsStreams;
4. const
5.   N = 8*1024*1024;  // there are N*4 Bytes in the file --> 32 MB
6. var
7.   stream: TStream;
8.   data: array[0..N-1] of DWord;
9.   i: Integer;
10.   t: TDateTime;
11.   fn: String;
12. begin
13.   for i := 0 to N-1 do
14.     data[i] := i;
15.   fn := ExtractFilepath(ParamStr(0)) + 'test.dat';
16.  
17.   t := Now();
18.   stream := TBufStream.Create(fn, fmCreate, 64*1024);  // --> 0.09 seconds
19.   //stream := TFileStream.Create(fn, fmCreate);        // --> 19 seconds
20.   //stream := TMemoryStream.Create;                      // --> 0.22 seconds
21.   try
22.     for i := 0 to N-1 do
23.       stream.Write(data[i], SizeOf(DWord));
24.     if stream is TMemoryStream then TMemoryStream(stream).SaveToFile(fn);
25.   finally
26.     stream.Free;
27.   end;
28.   t := Now() - t;
29.  
30.   WriteLn('Done ', FormatDateTime('s.zzz', t), ' seconds.');
31.   ReadLn;
32. end.

[AlexTP]

Maybe you can repro on the maximal project, ie do as I wrote in the 1st post and save the 30M file using ATSynEdit demo.

[wp]

Sorry, I promised myself that I will not try to fix bugs occuring only in third-party software.

[AlexTP]

If i run this example, i get the file test.log with content

Code: [Select]

line one;
line two;
line end;which is not expected.
see the code, test.log must have

Code: [Select]

first.
Code

Code: Pascal  [Select][+][-]

1. {$mode objfpc}
2. {$h+}
3.  
4. uses sysutils, classes, fpsStreams;
5.  
6. procedure Save(const filename: string; L: TStringList);
7. var
8.   fs: TBufStream;
9.   NMode: word;
10.   buf: string;
11.   i: integer;
12. begin
13.   NMode:= fmOpenReadWrite or fmShareDenyWrite;
14.   //don't set fmCreate for existing file, to keep NTFS file streams
15.   if not FileExists(filename) then
16.     NMode:= NMode or fmCreate;
17.  
18.   fs:= TBufStream.Create(filename, NMode);
19.   try
20.     fs.Size:= 0;
21.     for i:= 0 to L.Count-1 do
22.     begin
23.       buf:= L[i]+#10;
24.       fs.WriteBuffer(buf[1], length(buf));
25.     end;
26.   finally
27.     FreeAndNil(fs);
28.   end;
29. end;
30.  
31. var
32.   L: TStringList;
33. begin
34.   L:= TStringList.Create;
35.   L.Add('line one;');
36.   L.Add('line two;');
37.   L.Add('line end;');
38.   Save('test.log', L);
39.  
40.   L.Clear;
41.   L.Add('first.');
42.   Save('test.log', L);
43.  
44.   L.Free;
45.  
46. end.
47.  
48.  

[AlexTP]

(Linux x64, FPC 3.2.3)

[wp]

OK, now I understand. Never had thought of such a requirement to set the Size of the TBufStream...

Please test the new version at ccr.

However, having to consider both buffer state and existing file size, I think it will be very complicated to set the Size to an arbitrary value. Therefore, I decided to allow only a new size of zero; any other value raises an exception. For simplification, there is also a TBufStream.Clear method. (Why does the abstract class TStream declare Size to be public?)