Recent

Author Topic: Possible memory leak in TCSVDataset standard class  (Read 2485 times)

kveroneau

  • Full Member
  • ***
  • Posts: 119
Possible memory leak in TCSVDataset standard class
« on: March 28, 2020, 10:48:38 pm »
I have been trying to eliminate this memory leak heaptrc keeps complaining about in my code.  I have tried almost everything I can think of, and refactored my code multiple times this afternoon to no avail.  I am starting to think that either I am not doing something I am suppose to, or there is an issue with TCSVDataset where it is constantly leaking 726 bytes of memory.  The interesting part too, is if I call this procedure from a Button instead of a menu, it no longer says the leak is from any of my modules...  So, I suspect the TCSVDataset class is not correctly freeing something.  Here's my code where the issue is originating from:

Code: Pascal  [Select][+][-]
  1. procedure TScheduleForm.FetchScheduleClick(Sender: TObject);
  2. begin
  3.   FFileName:=InputBox('Fetch Schedule from Server...', 'Remote Schedule to fetch?', FFileName);
  4.   if FFileName = '' then
  5.     Exit;
  6.   if CSVDataset.Active then
  7.     CSVDataset.Close;
  8.   if FCSVStream <> Nil then
  9.     FCSVStream.Clear
  10.   else
  11.     FCSVStream:=TMemoryStream.Create;
  12.   if not GetSchedule(FCSVStream, FFileName) then
  13.   begin
  14.     ShowMessage('Unable to perform fetch!');
  15.     StatusBar.SimpleText:='Fetch failed for: '+FFileName;
  16.     SpecialMenu.Enabled:=False;
  17.     SendSchedule.Enabled:=False;
  18.     Exit; { If this is called, and I just exit the program, no memory leaks are reported. }
  19.   end;
  20.   CSVDataset.LoadFromCSVStream(FCSVStream); { Memory leak only occurs if this is called. }
  21.   StatusBar.SimpleText:='Working with Schedule: '+FFileName;
  22.   SpecialMenu.Enabled:=True;
  23.   SendSchedule.Enabled:=True;
  24. end;
  25.  
  26. procedure TScheduleForm.FormDestroy(Sender: TObject);
  27. begin
  28.   WriteLn('Running Destroy...');
  29.   if CSVDataset.Active then
  30.     CSVDataset.Close;
  31.   if FCSVStream <> Nil then
  32.     FCSVStream.Free;
  33. end;
  34.  
  35. function TScheduleForm.GetSchedule(dst: TMemoryStream; fname: String): Boolean;
  36. begin
  37.   with TFPHTTPClient.Create(Nil) do
  38.     Try
  39.       Get(ENDPOINT_URL, dst);
  40.       if dst.Size = 3 then
  41.         Result:=False
  42.       else
  43.         Result:=True;
  44.     finally
  45.       Free;
  46.     end;
  47. end;
  48.  

The memory leak only happens if I use the LoadFromCSVStream procedure call.  The loading seems to work as expected, as my TDataSource, and my TDBGrid are perfectly able to display the CSV data.

I should note that this CSV file is headless, meaning there are no Column headers in the file.  I have the following options set on TCSVDataset:

Code: Pascal  [Select][+][-]
  1.   object CSVDataset: TCSVDataset
  2.     FieldDefs = <    
  3.       item
  4.         Name = 'Time'
  5.         Attributes = [faRequired]
  6.         DataType = ftString
  7.         Precision = -1
  8.         Size = 6
  9.       end    
  10.       item
  11.         Name = 'Type'
  12.         Attributes = [faRequired]
  13.         DataType = ftString
  14.         Precision = -1
  15.         Size = 5
  16.       end    
  17.       item
  18.         Name = 'Message'
  19.         Attributes = [faRequired]
  20.         DataType = ftString
  21.         Precision = -1
  22.         Size = 255
  23.       end>
  24.     CSVOptions.FirstLineAsFieldNames = False
  25.     CSVOptions.DefaultFieldLength = 255
  26.     CSVOptions.Delimiter = ','
  27.     CSVOptions.QuoteChar = '"'
  28.     CSVOptions.LineEnding = #10
  29.     CSVOptions.IgnoreOuterWhitespace = False
  30.     CSVOptions.QuoteOuterWhitespace = True
  31.     left = 32
  32.     top = 80
  33.   end
  34.  

Here is the full stacktrace from heaptrc output when I use the menu item selection mode:

Code: Pascal  [Select][+][-]
  1. Running Destroy...
  2. Heap dump by heaptrc unit
  3. 231177 memory blocks allocated : 14243823/14729128
  4. 231165 memory blocks freed     : 14243097/14728384
  5. 12 unfreed memory blocks : 726
  6. True heap size : 327680
  7. True free heap : 325056
  8. Should be : 325400
  9. Call trace for block $00007FFFE17C9440 size 33
  10.   $0000000000649554 line 83 of include/menuitem.inc
  11.   $0000000000649EAA line 293 of include/menuitem.inc
  12.   $0000000000435C6F
  13.   $0000000000649554 line 83 of include/menuitem.inc
  14.   $0000000000649EAA line 293 of include/menuitem.inc
  15.   $0000000000435C6F
  16.   $0000000000602D3F line 4959 of include/wincontrol.inc
  17.   $0000000000602B0A line 4871 of include/wincontrol.inc
  18. Call trace for block $00007FFFF7E02900 size 27
  19.   $0000000000649554 line 83 of include/menuitem.inc
  20.   $0000000000649EAA line 293 of include/menuitem.inc
  21.   $0000000000435C6F
  22.   $0000000000649EAA line 293 of include/menuitem.inc
  23.   $0000000000435C6F
  24.   $000000000056AD08 line 434 of include/lclintf.inc
  25.   $0000000000706032 line 338 of include/messagedialogs.inc
  26.   $000000000070608E line 344 of include/messagedialogs.inc
  27. Call trace for block $00007FFFE17A90C0 size 192
  28.   $0000000000649554 line 83 of include/menuitem.inc
  29.   $0000000000649EAA line 293 of include/menuitem.inc
  30.   $0000000000435C6F
  31.   $0000000000682C4C line 67 of include/buttonglyph.inc
  32.   $00000000006816A2 line 24 of include/bitbtn.inc
  33.   $000000000070EFAA line 177 of buttonpanel.pas
  34.   $0000000000710133 line 539 of buttonpanel.pas
  35.   $000000000070F0BC line 194 of buttonpanel.pas
  36. Call trace for block $00007FFFF7E5BC00 size 80
  37.   $000000000079F95B
  38.   $0000000000649554 line 83 of include/menuitem.inc
  39.   $0000000000649EAA line 293 of include/menuitem.inc
  40.   $0000000000435C6F
  41.   $00000000004677E8 line 71 of sched.pas
  42.   $0000000000649554 line 83 of include/menuitem.inc
  43.   $0000000000649EAA line 293 of include/menuitem.inc
  44.   $0000000000435C6F
  45. Call trace for block $00007FFFF7E5CAE0 size 80
  46.   $000000000079F95B
  47.   $0000000000649554 line 83 of include/menuitem.inc
  48.   $0000000000649EAA line 293 of include/menuitem.inc
  49.   $0000000000435C6F
  50.   $0000000000467892 line 80 of sched.pas
  51.   $0000000000649554 line 83 of include/menuitem.inc
  52.   $0000000000649EAA line 293 of include/menuitem.inc
  53.   $0000000000435C6F
  54. Call trace for block $00007FFFE17C4940 size 32
  55.   $000000000079F95B
  56.   $0000000000649554 line 83 of include/menuitem.inc
  57.   $0000000000649EAA line 293 of include/menuitem.inc
  58.   $0000000000435C6F
  59.   $00000000007B66C5
  60.   $0000000000467892 line 80 of sched.pas
  61.   $0000000000649554 line 83 of include/menuitem.inc
  62.   $0000000000649EAA line 293 of include/menuitem.inc
  63. Call trace for block $00007FFFF7E5C920 size 80
  64.   $000000000079F95B
  65.   $0000000000649554 line 83 of include/menuitem.inc
  66.   $0000000000649EAA line 293 of include/menuitem.inc
  67.   $0000000000435C6F
  68.   $0000000000467892 line 80 of sched.pas
  69.   $0000000000649554 line 83 of include/menuitem.inc
  70.   $0000000000649EAA line 293 of include/menuitem.inc
  71.   $0000000000435C6F
  72. Call trace for block $00007FFFE17C4AC0 size 24
  73.   $0000000000649554 line 83 of include/menuitem.inc
  74.   $0000000000649EAA line 293 of include/menuitem.inc
  75.   $0000000000435C6F
  76.   $00000000007B643E
  77.   $00000000007B66C5
  78.   $0000000000467892 line 80 of sched.pas
  79.   $0000000000649554 line 83 of include/menuitem.inc
  80.   $0000000000649EAA line 293 of include/menuitem.inc
  81. Call trace for block $00007FFFF7F6EFC0 size 88
  82.   $0000000000649554 line 83 of include/menuitem.inc
  83.   $0000000000649EAA line 293 of include/menuitem.inc
  84.   $0000000000435C6F
  85.   $00000000007B643E
  86.   $00000000007B66C5
  87.   $0000000000467892 line 80 of sched.pas
  88.   $0000000000649554 line 83 of include/menuitem.inc
  89.   $0000000000649EAA line 293 of include/menuitem.inc
  90. Call trace for block $00007FFFF7E09B80 size 32
  91.   $000000000049960F
  92.   $0000000000498C82
  93.   $000000000049842D
  94.   $0000000000496F03
  95.   $000000000049842D
  96.   $00000000004977B9
  97.   $000000000049787C
  98.   $0000000000499275
  99. Call trace for block $00007FFFF7E09700 size 29
  100.   $000000000049960F
  101.   $0000000000498C82
  102.   $000000000049842D
  103.   $0000000000496F03
  104.   $000000000049842D
  105.   $00000000004977B9
  106.   $000000000049787C
  107.   $0000000000499275
  108. Call trace for block $00007FFFF7E09C40 size 29
  109.   $000000000049960F
  110.   $0000000000498C82
  111.   $000000000049842D
  112.   $0000000000496F03
  113.   $000000000049842D
  114.   $00000000004977B9
  115.   $000000000049787C
  116.   $0000000000499275
  117.  

For completion, in case anyone asks, here are the lines in my code which are being referenced by the stacetrace:

Line 71: FFileName:=InputBox('Fetch Schedule from Server...', 'Remote Schedule to fetch?', FFileName);

Line 80: if not GetSchedule(FCSVStream, FFileName) then

Again, I want to reiterate that if I call the FetchScheduleClick method from a button instead of my menu, the stacktrace no longer references my code, and otherwise the stracktrace is the same:

Code: Pascal  [Select][+][-]
  1. Running Destroy...
  2. Heap dump by heaptrc unit
  3. 41039 memory blocks allocated : 2235068/2339352
  4. 41027 memory blocks freed     : 2234342/2338608
  5. 12 unfreed memory blocks : 726
  6. True heap size : 327680
  7. True free heap : 325056
  8. Should be : 325400
  9. Call trace for block $00007FFFE17C9C80 size 33
  10.   $0000000000612D22 line 2913 of include/control.inc
  11.   $0000000000641ADA line 55 of include/buttoncontrol.inc
  12.   $0000000000642323 line 169 of include/buttons.inc
  13.   $00000000006419C6 line 21 of include/buttoncontrol.inc
  14.   $0000000000435C6F
  15.   $0000000000603F7D line 5419 of include/wincontrol.inc
  16.   $000000000068F506 line 5747 of qt/qtwidgets.pas
  17.   $0000000000690419 line 6120 of qt/qtwidgets.pas
  18. Call trace for block $00007FFFF7E03080 size 27
  19.   $0000000000612D22 line 2913 of include/control.inc
  20.   $0000000000641ADA line 55 of include/buttoncontrol.inc
  21.   $0000000000642323 line 169 of include/buttons.inc
  22.   $00000000006419C6 line 21 of include/buttoncontrol.inc
  23.   $0000000000435C6F
  24.   $0000000000603F7D line 5419 of include/wincontrol.inc
  25.   $000000000068F506 line 5747 of qt/qtwidgets.pas
  26.   $0000000000690419 line 6120 of qt/qtwidgets.pas
  27. Call trace for block $00007FFFE17BA0C0 size 192
  28.   $0000000000612D22 line 2913 of include/control.inc
  29.   $0000000000641ADA line 55 of include/buttoncontrol.inc
  30.   $0000000000642323 line 169 of include/buttons.inc
  31.   $00000000006419C6 line 21 of include/buttoncontrol.inc
  32.   $0000000000435C6F
  33.   $0000000000603F7D line 5419 of include/wincontrol.inc
  34.   $000000000068F506 line 5747 of qt/qtwidgets.pas
  35.   $0000000000690419 line 6120 of qt/qtwidgets.pas
  36. Call trace for block $00007FFFF7E5C220 size 80
  37.   $000000000079F95B
  38.   $0000000000612D22 line 2913 of include/control.inc
  39.   $0000000000641ADA line 55 of include/buttoncontrol.inc
  40.   $0000000000642323 line 169 of include/buttons.inc
  41.   $00000000006419C6 line 21 of include/buttoncontrol.inc
  42.   $0000000000435C6F
  43.   $0000000000603F7D line 5419 of include/wincontrol.inc
  44.   $000000000068F506 line 5747 of qt/qtwidgets.pas
  45. Call trace for block $00007FFFF7E5D100 size 80
  46.   $000000000079F95B
  47.   $0000000000612D22 line 2913 of include/control.inc
  48.   $0000000000641ADA line 55 of include/buttoncontrol.inc
  49.   $0000000000642323 line 169 of include/buttons.inc
  50.   $00000000006419C6 line 21 of include/buttoncontrol.inc
  51.   $0000000000435C6F
  52.   $0000000000603F7D line 5419 of include/wincontrol.inc
  53.   $000000000068F506 line 5747 of qt/qtwidgets.pas
  54. Call trace for block $00007FFFE17C4F40 size 32
  55.   $000000000079F95B
  56.   $0000000000612D22 line 2913 of include/control.inc
  57.   $0000000000641ADA line 55 of include/buttoncontrol.inc
  58.   $0000000000642323 line 169 of include/buttons.inc
  59.   $00000000006419C6 line 21 of include/buttoncontrol.inc
  60.   $0000000000435C6F
  61.   $0000000000603F7D line 5419 of include/wincontrol.inc
  62.   $000000000068F506 line 5747 of qt/qtwidgets.pas
  63. Call trace for block $00007FFFF7E5CF40 size 80
  64.   $000000000079F95B
  65.   $0000000000612D22 line 2913 of include/control.inc
  66.   $0000000000641ADA line 55 of include/buttoncontrol.inc
  67.   $0000000000642323 line 169 of include/buttons.inc
  68.   $00000000006419C6 line 21 of include/buttoncontrol.inc
  69.   $0000000000435C6F
  70.   $0000000000603F7D line 5419 of include/wincontrol.inc
  71.   $000000000068F506 line 5747 of qt/qtwidgets.pas
  72. Call trace for block $00007FFFE17C50C0 size 24
  73.   $0000000000612D22 line 2913 of include/control.inc
  74.   $0000000000641ADA line 55 of include/buttoncontrol.inc
  75.   $0000000000642323 line 169 of include/buttons.inc
  76.   $00000000006419C6 line 21 of include/buttoncontrol.inc
  77.   $0000000000435C6F
  78.   $0000000000603F7D line 5419 of include/wincontrol.inc
  79.   $000000000068F506 line 5747 of qt/qtwidgets.pas
  80.   $0000000000690419 line 6120 of qt/qtwidgets.pas
  81. Call trace for block $00007FFFF7F6F3C0 size 88
  82.   $0000000000612D22 line 2913 of include/control.inc
  83.   $0000000000641ADA line 55 of include/buttoncontrol.inc
  84.   $0000000000642323 line 169 of include/buttons.inc
  85.   $00000000006419C6 line 21 of include/buttoncontrol.inc
  86.   $0000000000435C6F
  87.   $0000000000603F7D line 5419 of include/wincontrol.inc
  88.   $000000000068F506 line 5747 of qt/qtwidgets.pas
  89.   $0000000000690419 line 6120 of qt/qtwidgets.pas
  90. Call trace for block $00007FFFF7E09DC0 size 32
  91.   $000000000049960F
  92.   $0000000000498C82
  93.   $000000000049842D
  94.   $0000000000496F03
  95.   $000000000049842D
  96.   $00000000004977B9
  97.   $000000000049787C
  98.   $0000000000499275
  99. Call trace for block $00007FFFF7E09C40 size 29
  100.   $000000000049960F
  101.   $0000000000498C82
  102.   $000000000049842D
  103.   $0000000000496F03
  104.   $000000000049842D
  105.   $00000000004977B9
  106.   $000000000049787C
  107.   $0000000000499275
  108. Call trace for block $00007FFFF7E09E80 size 29
  109.   $000000000049960F
  110.   $0000000000498C82
  111.   $000000000049842D
  112.   $0000000000496F03
  113.   $000000000049842D
  114.   $00000000004977B9
  115.   $000000000049787C
  116.   $0000000000499275
  117.  

Thank you so much for any help you can provide.  Oh, I am using FPC 3.0.4 with Lazarus 2.0.0.

eljo

  • Sr. Member
  • ****
  • Posts: 468
Re: Possible memory leak in TCSVDataset standard class
« Reply #1 on: March 29, 2020, 06:02:22 am »
Could you please provide a simple (the smallest possible) sample that shows the memory leak?
Your current trace logs show blocks in windget files but I see nowhere  the csvdocument units mentioned which makes think that the problem is not in the TCSVDataset at all.

To test for that create a simple console application create a TCSVDataset dynamical through code load any data you have tested so far and the destroy the TCSVDataset and exit to see the the trace log.


avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: Possible memory leak in TCSVDataset standard class
« Reply #2 on: March 29, 2020, 12:59:20 pm »
You could also try QT with trunk FPC/LAZ, try with something other then QT, and report FPC and LAZ versions used in each try.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

kveroneau

  • Full Member
  • ***
  • Posts: 119
Re: Possible memory leak in TCSVDataset standard class
« Reply #3 on: March 29, 2020, 10:33:42 pm »
Could you please provide a simple (the smallest possible) sample that shows the memory leak?
Your current trace logs show blocks in windget files but I see nowhere  the csvdocument units mentioned which makes think that the problem is not in the TCSVDataset at all.

To test for that create a simple console application create a TCSVDataset dynamical through code load any data you have tested so far and the destroy the TCSVDataset and exit to see the the trace log.

I created a basic console application using TCustomApplication, I assigned the Owner of the created TCSVDataset to the application class.  I copied and partially modified both my GetSchedule and FetchSchedule methods only removing the GUI dependent portions.  At first I thought there was no memory leak, but then I realized that I entered the wrong filename to fetch...  After using the correct filename I did indeed see a similar exception.  I will post the entire test program here minus any endpoints of course.

New stacktrace:
Code: Pascal  [Select][+][-]
  1. Remote Schedule to fetch?  * CSV Data loaded successfully.
  2. Running Destroy...
  3. Heap dump by heaptrc unit
  4. 516 memory blocks allocated : 146833/149216
  5. 513 memory blocks freed     : 146581/148952
  6. 3 unfreed memory blocks : 252
  7. True heap size : 196608
  8. True free heap : 195872
  9. Should be : 195960
  10. Call trace for block $00007FFFF7FEBD80 size 33
  11.   $0000000000400A4F line 90 of leaktest.lpr
  12.   $00000000004765E9
  13.   $000000000047D0AE
  14.   $000000000047D335
  15.   $0000000000400804 line 62 of leaktest.lpr
  16.   $0000000000400A4F line 90 of leaktest.lpr
  17.   $00000000004765E9
  18.   $00000000004765E9
  19. Call trace for block $00007FFFF7FEBCC0 size 27
  20.   $0000000000400A4F line 90 of leaktest.lpr
  21.   $00000000004765E9
  22.   $000000000047C932
  23.   $000000000047D0AE
  24.   $000000000047D335
  25.   $0000000000400804 line 62 of leaktest.lpr
  26.   $0000000000400A4F line 90 of leaktest.lpr
  27.   $00000000004765E9
  28. Call trace for block $00007FFFF7F6C220 size 192
  29.   $0000000000400A4F line 90 of leaktest.lpr
  30.   $00000000004765E9
  31.  

Here is the code here, should be pretty simple to upload a CSV file somewhere or spin up a local web server via python2 -m SimpleHTTPServer to try the test on your own.

Code: Pascal  [Select][+][-]
  1. program leaktest;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. uses
  6.   {$IFDEF UNIX}{$IFDEF UseCThreads}
  7.   cthreads,
  8.   {$ENDIF}{$ENDIF}
  9.   Classes, SysUtils, CustApp, csvdataset, fphttpclient
  10.   { you can add units after this };
  11.  
  12. type
  13.  
  14.   { TMyApplication }
  15.  
  16.   TMyApplication = class(TCustomApplication)
  17.   private
  18.     CSVDataset: TCSVDataset;
  19.     FCSVStream: TMemoryStream;
  20.     function GetSchedule(dst: TMemoryStream; fname: String): Boolean;
  21.     procedure FetchSchedule;
  22.   protected
  23.     procedure DoRun; override;
  24.   public
  25.     constructor Create(TheOwner: TComponent); override;
  26.     destructor Destroy; override;
  27.     procedure WriteHelp; virtual;
  28.   end;
  29.  
  30. { TMyApplication }
  31.  
  32. function TMyApplication.GetSchedule(dst: TMemoryStream; fname: String): Boolean;
  33. begin
  34.   with TFPHTTPClient.Create(Nil) do
  35.     Try
  36.       AddHeader('X-Filename', fname);
  37.       Get('*Redacted*', dst);
  38.       if dst.Size = 3 then
  39.         Result:=False
  40.       else
  41.         Result:=True;
  42.     finally
  43.       Free;
  44.     end;
  45. end;
  46.  
  47. procedure TMyApplication.FetchSchedule;
  48. var
  49.   fname: string;
  50. begin
  51.   Write('Remote Schedule to fetch? ');
  52.   {ReadLn(fname);}
  53.   fname:='*Redacted*';
  54.   if fname = '' then
  55.     Exit;
  56.   if CSVDataset.Active then
  57.     CSVDataset.Close;
  58.   if FCSVStream <> Nil then
  59.     FCSVStream.Clear
  60.   else
  61.     FCSVStream:=TMemoryStream.Create;
  62.   if not GetSchedule(FCSVStream, fname) then
  63.   begin
  64.     WriteLn(' ** Unable to perform fetch!'); { Again, the memory does not occur if it cannot fetch the CSV file. }
  65.     Exit;
  66.   end;
  67.   CSVDataset.LoadFromCSVStream(FCSVStream);
  68.   WriteLn(' * CSV Data loaded successfully.');
  69. end;
  70.  
  71. procedure TMyApplication.DoRun;
  72. var
  73.   ErrorMsg: String;
  74. begin
  75.   // quick check parameters
  76.   ErrorMsg:=CheckOptions('h', 'help');
  77.   if ErrorMsg<>'' then begin
  78.     ShowException(Exception.Create(ErrorMsg));
  79.     Terminate;
  80.     Exit;
  81.   end;
  82.  
  83.   // parse parameters
  84.   if HasOption('h', 'help') then begin
  85.     WriteHelp;
  86.     Terminate;
  87.     Exit;
  88.   end;
  89.  
  90.   FetchSchedule;
  91.  
  92.   // stop program loop
  93.   Terminate;
  94. end;
  95.  
  96. constructor TMyApplication.Create(TheOwner: TComponent);
  97. begin
  98.   inherited Create(TheOwner);
  99.   StopOnException:=True;
  100.   CSVDataset:=TCSVDataset.Create(Self);
  101. end;
  102.  
  103. destructor TMyApplication.Destroy;
  104. begin
  105.   WriteLn('Running Destroy...');
  106.   if CSVDataset.Active then
  107.     CSVDataset.Close;
  108.   if FCSVStream <> Nil then
  109.     FCSVStream.Free;
  110.   inherited Destroy;
  111. end;
  112.  
  113. procedure TMyApplication.WriteHelp;
  114. begin
  115.   { add your help code here }
  116.   writeln('Usage: ', ExeName, ' -h');
  117. end;
  118.  
  119. var
  120.   Application: TMyApplication;
  121. begin
  122.   Application:=TMyApplication.Create(nil);
  123.   Application.Title:='My Application';
  124.   Application.Run;
  125.   Application.Free;
  126. end.
  127.  

I will continue to investigate further myself and post an update here if I find anything new out.  Thank you so far for your assistance.

UPDATE: So, I tried to compile my FCL with debugging enabled so I can run my program and see additional output, and now when I run my program I get a weird Runtime Error 201, which to my understanding is a range check error.  However, it is occurring in the most oddest place, within the ssockets unit in the following line:

Code: Pascal  [Select][+][-]
  1. addr.sin_addr.s_addr := HostToNet(a.s_addr);
  2.  

I did what most might do and did a full Step through all the running code, and in my Local Variables window, I do not see any issues with any of the variables in this specific stack frame.  Really odd.  I'll keep investigating.

UPDATE 2: So, this weird RunTime 201/Range Check Error only occurs if I compile the packages directory in my source code with make debug to compile the FCL with debugging enabled, not sure why, if I recompile the same source after doing a clean using make clean && make, the Runtime 201/Range Check Error disappears.  Is there a particular reason why I cannot use the ssockets unit when built with DEBUG enabled?

UPDATE 3: So, I managed to get this test project to compile with a DEBUG enabled RTL, and now the stacktrace is pointing to something related to an ansistring:

Code: Pascal  [Select][+][-]
  1. Remote Schedule to fetch?  * CSV Data loaded successfully.
  2. Running Destroy...
  3. Heap dump by heaptrc unit
  4. 516 memory blocks allocated : 146833/149216
  5. 513 memory blocks freed     : 146581/148952
  6. 3 unfreed memory blocks : 252
  7. True heap size : 196608
  8. True free heap : 195872
  9. Should be : 195960
  10. Call trace for block $00007FFFF7FEBD80 size 33
  11.   $000000000041DC82 line 342 of ../inc/heap.inc
  12.   $000000000040F8CF line 787 of ../inc/astrings.inc
  13.   $000000000040E779 line 266 of ../inc/astrings.inc
  14.   $000000000047FC29 line 76 of ../objpas/sysutils/sysstr.inc
  15.   $00000000004DB5AD
  16.   $00000000004DB3B5
  17.   $00000000004DBB66
  18.   $00000000004ACCF1
  19. Call trace for block $00007FFFF7FEBCC0 size 27
  20.   $000000000041DA5E line 276 of ../inc/heap.inc
  21.   $000000000040E33E line 115 of ../inc/astrings.inc
  22.   $000000000040F816 line 771 of ../inc/astrings.inc
  23.   $000000000040F00A line 510 of ../inc/astrings.inc
  24.   $00000000004DAED1
  25.   $00000000004DB02B
  26.   $00000000004DB6D2
  27.   $00000000004AD204
  28. Call trace for block $00007FFFF7F6C220 size 192
  29.   $000000000041DA5E line 276 of ../inc/heap.inc
  30.   $000000000041724D line 355 of ../inc/objpas.inc
  31.   $00000000004DB68B
  32.   $00000000004AD204
  33.   $00000000004D77AF
  34.   $00000000004D0AFE
  35.   $00000000004DF471
  36.   $00000000004E0A37
  37.  

As noted, it is still showing the same sizes of memory being leaked.  I am starting to make some slight progress in figuring out where this leak is coming from.  I will try and press on and not lose faith.  I am going to try and see if I can replicate without using any Networking, so I can use a DEBUG FCL to get a full stracktrace for you guys.  :D

UPDATE 4: And may I present a full stacktrace of my use and leaking of TCSVDataset from FPC 3.0.4!  I will continue investigating as far as my abilities go, but I made it this far, I'm sure I can figure out where this leak is now.  ;D

Code: Pascal  [Select][+][-]
  1. Remote Schedule to fetch?  * CSV Data loaded successfully.
  2. Running Destroy...
  3. Heap dump by heaptrc unit
  4. 261 memory blocks allocated : 354082/356968
  5. 258 memory blocks freed     : 353058/355936
  6. 3 unfreed memory blocks : 1024
  7. True heap size : 491520
  8. True free heap : 489984
  9. Should be : 490104
  10. Call trace for block $00007FFFF7FBD5B0 size 805
  11. Found addr = $0000000000400000
  12.   $000000000041D822 line 342 of ../inc/heap.inc
  13. Found addr = $0000000000400000
  14.   $000000000040F46F line 787 of ../inc/astrings.inc
  15. Found addr = $0000000000400000
  16.   $000000000040E319 line 266 of ../inc/astrings.inc
  17. Found addr = $0000000000400000
  18.   $000000000047F7C9 line 76 of ../objpas/sysutils/sysstr.inc
  19. Found addr = $0000000000400000
  20.   $00000000004DC1FD line 409 of fcl-base/src/csvreadwrite.pp
  21. Found addr = $0000000000400000
  22.   $00000000004DBFE4 line 367 of fcl-base/src/csvreadwrite.pp
  23. Found addr = $0000000000400000
  24.   $00000000004DC968 line 515 of fcl-base/src/csvreadwrite.pp
  25. Found addr = $0000000000400000
  26.   $00000000004AD9C1 line 134 of fcl-db/src/base/csvdataset.pp
  27. Call trace for block $00007FFFF7FADE80 size 27
  28. Found addr = $0000000000400000
  29.   $000000000041D5FE line 276 of ../inc/heap.inc
  30. Found addr = $0000000000400000
  31.   $000000000040DEDE line 115 of ../inc/astrings.inc
  32. Found addr = $0000000000400000
  33.   $000000000040F3B6 line 771 of ../inc/astrings.inc
  34. Found addr = $0000000000400000
  35.   $000000000040EBAA line 510 of ../inc/astrings.inc
  36. Found addr = $0000000000400000
  37.   $00000000004DBA0C line 282 of fcl-base/src/csvreadwrite.pp
  38. Found addr = $0000000000400000
  39.   $00000000004DBB90 line 295 of fcl-base/src/csvreadwrite.pp
  40. Found addr = $0000000000400000
  41.   $00000000004DC347 line 421 of fcl-base/src/csvreadwrite.pp
  42. Found addr = $0000000000400000
  43.   $00000000004AE028 line 185 of fcl-db/src/base/csvdataset.pp
  44. Call trace for block $00007FFFF7F7B220 size 192
  45. Found addr = $0000000000400000
  46.   $000000000041D5FE line 276 of ../inc/heap.inc
  47. Found addr = $0000000000400000
  48.   $0000000000416DED line 355 of ../inc/objpas.inc
  49. Found addr = $0000000000400000
  50.   $00000000004DC300 line 420 of fcl-base/src/csvreadwrite.pp
  51. Found addr = $0000000000400000
  52.   $00000000004AE028 line 185 of fcl-db/src/base/csvdataset.pp
  53. Found addr = $0000000000400000
  54.   $00000000004D633F line 3250 of fcl-db/src/base/bufdataset.pas
  55. Found addr = $0000000000400000
  56.   $00000000004CB7B9 line 1240 of fcl-db/src/base/bufdataset.pas
  57. Found addr = $0000000000400000
  58.   $00000000004E13EF line 405 of fcl-db/src/base/dataset.inc
  59. Found addr = $0000000000400000
  60.   $00000000004E36A5 line 960 of fcl-db/src/base/dataset.inc
  61.  

So to my understanding on how to check which line is leaking is to check the final file in each reported leak, correct?  So, there should be 192 bytes being leaked from fcl-db/src/base/dataset.inc on line number 960?

Update 5: It seems that the TCSVParser might not be getting freed correctly, I can see line 185 in csvdataset being referenced multiple times in the above heaptrc trace.  This is line 185 from my local sources:

Code: Pascal  [Select][+][-]
  1. FParser:=TCSVParser.Create;
  2.  

I am going to modify the TCSVDataPacketReader class to FreeAndNil FParser... and that worked.  So, yeah, there is officially a memory leak here in the csvdataset unit as I thought there was.  I am not sure how to submit patches or official bug reports.  But here is what I did in the TCSVDataPacketReader.Destroy destructor to resolve the leak:

Code: Pascal  [Select][+][-]
  1. destructor TCSVDataPacketReader.Destroy;
  2. begin
  3.   If FOwnsOptions then
  4.     FreeAndNil(FOPtions);
  5.   FreeAndNil(Fline);
  6.   if FParser <> Nil then
  7.     FreeAndNil(FParser);
  8.   inherited Destroy;
  9. end;
  10.  

Hopefully all this troubleshoot is helpful to someone who might also be facing an odd memory leak within the Free Component Library.  I'm actually surprised nobody noticed this leak until now.  Perhaps I am doing something wrong in my code to properly free FParser, if I am, please let me know how I should be using this class to free the parser.

FINAL UPDATE(I swear):  So, I was correct that there was a known memory leak in TCSVDataset, well more like a memory leak in a class it uses, the TCSVPacketReader class.  This leak which I mentioned here is resolved in the latest release candidate 3.2.0 FPC version.  Here is their updated code to resolve the leak:

Code: Pascal  [Select][+][-]
  1. destructor TCSVDataPacketReader.Destroy;
  2. begin
  3.   FreeAndNil(FCreateFieldDefs);
  4.   If FOwnsOptions then
  5.     FreeAndNil(FOPtions);
  6.   FreeAndNil(Fline);
  7.   FreeAndNil(FParser);
  8.   inherited Destroy;
  9. end;
  10.  

Wish I knew this was an issue and will be resolved in the next FPC release, or I would not have wasted an entire afternoon troubleshooting it...  At least I learned a lot more about the internals of FPC, and how to build the RTL and FCL in debug modes.
« Last Edit: March 30, 2020, 02:25:57 am by kveroneau »

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: Possible memory leak in TCSVDataset standard class
« Reply #4 on: March 30, 2020, 11:56:21 am »
I did not go through compilation and testing of your code, but at first look I wonder if leak would disappear if you simply move creation of FCSVStream into constructor.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: Possible memory leak in TCSVDataset standard class
« Reply #5 on: March 30, 2020, 12:04:48 pm »
Wish I knew this was an issue and will be resolved in the next FPC release, or I would not have wasted an entire afternoon troubleshooting it...  At least I learned a lot more about the internals of FPC, and how to build the RTL and FCL in debug modes.

That's why we usually suggest to check against trunk when reproducing a bug. :-X In this case the corresponding bug report would have been hard to find as well cause it originally occurred together with fcl-report.

kveroneau

  • Full Member
  • ***
  • Posts: 119
Re: Possible memory leak in TCSVDataset standard class
« Reply #6 on: March 30, 2020, 07:03:27 pm »
That's why we usually suggest to check against trunk when reproducing a bug. :-X In this case the corresponding bug report would have been hard to find as well cause it originally occurred together with fcl-report.

Just read the bug report, thank you.  Seems to have been reported back in 2018.  I guess FPC does not release incremental patch versions?  May I suggest separating out the versions of both the compiler/RTL and the FCL into separately versioned and released components?  This will allow bug fixes in the FCL to be released before a newer compiler/RTL version.  Because, currently my options are to get around this bug, is to either start using a potentially unstable version of FPC(3.2.0), or to patch the bug myself in my local FCL.  I did patch the FCL myself before when I noticed a bug in the Blowfish implementation.  The encrypted data was not properly using the host processor's Endianness.  I never opened a bug report for this, nor submitted my patched unit file to a developer.  The patch is however available on the forums, if you are curious on what I did, and if the issue has been resolved.

 

TinyPortal © 2005-2018