Could you please provide a simple (the smallest possible) sample that shows the memory leak?
Your current trace logs show blocks in windget files but I see nowhere the csvdocument units mentioned which makes think that the problem is not in the TCSVDataset at all.
To test for that create a simple console application create a TCSVDataset dynamical through code load any data you have tested so far and the destroy the TCSVDataset and exit to see the the trace log.
I created a basic console application using
TCustomApplication, I assigned the Owner of the created
TCSVDataset to the application class. I copied and partially modified both my
GetSchedule and
FetchSchedule methods only removing the GUI dependent portions. At first I thought there was no memory leak, but then I realized that I entered the wrong filename to fetch... After using the correct filename I did indeed see a similar exception. I will post the entire test program here minus any endpoints of course.
New stacktrace:
Remote Schedule to fetch? * CSV Data loaded successfully.
Running Destroy...
Heap dump by heaptrc unit
516 memory blocks allocated : 146833/149216
513 memory blocks freed : 146581/148952
3 unfreed memory blocks : 252
True heap size : 196608
True free heap : 195872
Should be : 195960
Call trace for block $00007FFFF7FEBD80 size 33
$0000000000400A4F line 90 of leaktest.lpr
$00000000004765E9
$000000000047D0AE
$000000000047D335
$0000000000400804 line 62 of leaktest.lpr
$0000000000400A4F line 90 of leaktest.lpr
$00000000004765E9
$00000000004765E9
Call trace for block $00007FFFF7FEBCC0 size 27
$0000000000400A4F line 90 of leaktest.lpr
$00000000004765E9
$000000000047C932
$000000000047D0AE
$000000000047D335
$0000000000400804 line 62 of leaktest.lpr
$0000000000400A4F line 90 of leaktest.lpr
$00000000004765E9
Call trace for block $00007FFFF7F6C220 size 192
$0000000000400A4F line 90 of leaktest.lpr
$00000000004765E9
Here is the code here, should be pretty simple to upload a CSV file somewhere or spin up a local web server via
python2 -m SimpleHTTPServer to try the test on your own.
program leaktest;
{$mode objfpc}{$H+}
uses
{$IFDEF UNIX}{$IFDEF UseCThreads}
cthreads,
{$ENDIF}{$ENDIF}
Classes, SysUtils, CustApp, csvdataset, fphttpclient
{ you can add units after this };
type
{ TMyApplication }
TMyApplication = class(TCustomApplication)
private
CSVDataset: TCSVDataset;
FCSVStream: TMemoryStream;
function GetSchedule(dst: TMemoryStream; fname: String): Boolean;
procedure FetchSchedule;
protected
procedure DoRun; override;
public
constructor Create(TheOwner: TComponent); override;
destructor Destroy; override;
procedure WriteHelp; virtual;
end;
{ TMyApplication }
function TMyApplication.GetSchedule(dst: TMemoryStream; fname: String): Boolean;
begin
with TFPHTTPClient.Create(Nil) do
Try
AddHeader('X-Filename', fname);
Get('*Redacted*', dst);
if dst.Size = 3 then
Result:=False
else
Result:=True;
finally
Free;
end;
end;
procedure TMyApplication.FetchSchedule;
var
fname: string;
begin
Write('Remote Schedule to fetch? ');
{ReadLn(fname);}
fname:='*Redacted*';
if fname = '' then
Exit;
if CSVDataset.Active then
CSVDataset.Close;
if FCSVStream <> Nil then
FCSVStream.Clear
else
FCSVStream:=TMemoryStream.Create;
if not GetSchedule(FCSVStream, fname) then
begin
WriteLn(' ** Unable to perform fetch!'); { Again, the memory does not occur if it cannot fetch the CSV file. }
Exit;
end;
CSVDataset.LoadFromCSVStream(FCSVStream);
WriteLn(' * CSV Data loaded successfully.');
end;
procedure TMyApplication.DoRun;
var
ErrorMsg: String;
begin
// quick check parameters
ErrorMsg:=CheckOptions('h', 'help');
if ErrorMsg<>'' then begin
ShowException(Exception.Create(ErrorMsg));
Terminate;
Exit;
end;
// parse parameters
if HasOption('h', 'help') then begin
WriteHelp;
Terminate;
Exit;
end;
FetchSchedule;
// stop program loop
Terminate;
end;
constructor TMyApplication.Create(TheOwner: TComponent);
begin
inherited Create(TheOwner);
StopOnException:=True;
CSVDataset:=TCSVDataset.Create(Self);
end;
destructor TMyApplication.Destroy;
begin
WriteLn('Running Destroy...');
if CSVDataset.Active then
CSVDataset.Close;
if FCSVStream <> Nil then
FCSVStream.Free;
inherited Destroy;
end;
procedure TMyApplication.WriteHelp;
begin
{ add your help code here }
writeln('Usage: ', ExeName, ' -h');
end;
var
Application: TMyApplication;
begin
Application:=TMyApplication.Create(nil);
Application.Title:='My Application';
Application.Run;
Application.Free;
end.
I will continue to investigate further myself and post an update here if I find anything new out. Thank you so far for your assistance.
UPDATE: So, I tried to compile my FCL with debugging enabled so I can run my program and see additional output, and now when I run my program I get a weird Runtime Error 201, which to my understanding is a range check error. However, it is occurring in the most oddest place, within the
ssockets unit in the following line:
addr.sin_addr.s_addr := HostToNet(a.s_addr);
I did what most might do and did a full Step through all the running code, and in my
Local Variables window, I do not see any issues with any of the variables in this specific stack frame. Really odd. I'll keep investigating.
UPDATE 2: So, this weird RunTime 201/Range Check Error only occurs if I compile the
packages directory in my source code with
make debug to compile the FCL with debugging enabled, not sure why, if I recompile the same source after doing a clean using
make clean && make, the Runtime 201/Range Check Error disappears. Is there a particular reason why I cannot use the
ssockets unit when built with
DEBUG enabled?
UPDATE 3: So, I managed to get this test project to compile with a DEBUG enabled RTL, and now the stacktrace is pointing to something related to an ansistring:
Remote Schedule to fetch? * CSV Data loaded successfully.
Running Destroy...
Heap dump by heaptrc unit
516 memory blocks allocated : 146833/149216
513 memory blocks freed : 146581/148952
3 unfreed memory blocks : 252
True heap size : 196608
True free heap : 195872
Should be : 195960
Call trace for block $00007FFFF7FEBD80 size 33
$000000000041DC82 line 342 of ../inc/heap.inc
$000000000040F8CF line 787 of ../inc/astrings.inc
$000000000040E779 line 266 of ../inc/astrings.inc
$000000000047FC29 line 76 of ../objpas/sysutils/sysstr.inc
$00000000004DB5AD
$00000000004DB3B5
$00000000004DBB66
$00000000004ACCF1
Call trace for block $00007FFFF7FEBCC0 size 27
$000000000041DA5E line 276 of ../inc/heap.inc
$000000000040E33E line 115 of ../inc/astrings.inc
$000000000040F816 line 771 of ../inc/astrings.inc
$000000000040F00A line 510 of ../inc/astrings.inc
$00000000004DAED1
$00000000004DB02B
$00000000004DB6D2
$00000000004AD204
Call trace for block $00007FFFF7F6C220 size 192
$000000000041DA5E line 276 of ../inc/heap.inc
$000000000041724D line 355 of ../inc/objpas.inc
$00000000004DB68B
$00000000004AD204
$00000000004D77AF
$00000000004D0AFE
$00000000004DF471
$00000000004E0A37
As noted, it is still showing the same sizes of memory being leaked. I am starting to make some slight progress in figuring out where this leak is coming from. I will try and press on and not lose faith. I am going to try and see if I can replicate without using any Networking, so I can use a DEBUG FCL to get a full stracktrace for you guys.
UPDATE 4: And may I present a full stacktrace of my use and leaking of TCSVDataset from FPC 3.0.4! I will continue investigating as far as my abilities go, but I made it this far, I'm sure I can figure out where this leak is now.
Remote Schedule to fetch? * CSV Data loaded successfully.
Running Destroy...
Heap dump by heaptrc unit
261 memory blocks allocated : 354082/356968
258 memory blocks freed : 353058/355936
3 unfreed memory blocks : 1024
True heap size : 491520
True free heap : 489984
Should be : 490104
Call trace for block $00007FFFF7FBD5B0 size 805
Found addr = $0000000000400000
$000000000041D822 line 342 of ../inc/heap.inc
Found addr = $0000000000400000
$000000000040F46F line 787 of ../inc/astrings.inc
Found addr = $0000000000400000
$000000000040E319 line 266 of ../inc/astrings.inc
Found addr = $0000000000400000
$000000000047F7C9 line 76 of ../objpas/sysutils/sysstr.inc
Found addr = $0000000000400000
$00000000004DC1FD line 409 of fcl-base/src/csvreadwrite.pp
Found addr = $0000000000400000
$00000000004DBFE4 line 367 of fcl-base/src/csvreadwrite.pp
Found addr = $0000000000400000
$00000000004DC968 line 515 of fcl-base/src/csvreadwrite.pp
Found addr = $0000000000400000
$00000000004AD9C1 line 134 of fcl-db/src/base/csvdataset.pp
Call trace for block $00007FFFF7FADE80 size 27
Found addr = $0000000000400000
$000000000041D5FE line 276 of ../inc/heap.inc
Found addr = $0000000000400000
$000000000040DEDE line 115 of ../inc/astrings.inc
Found addr = $0000000000400000
$000000000040F3B6 line 771 of ../inc/astrings.inc
Found addr = $0000000000400000
$000000000040EBAA line 510 of ../inc/astrings.inc
Found addr = $0000000000400000
$00000000004DBA0C line 282 of fcl-base/src/csvreadwrite.pp
Found addr = $0000000000400000
$00000000004DBB90 line 295 of fcl-base/src/csvreadwrite.pp
Found addr = $0000000000400000
$00000000004DC347 line 421 of fcl-base/src/csvreadwrite.pp
Found addr = $0000000000400000
$00000000004AE028 line 185 of fcl-db/src/base/csvdataset.pp
Call trace for block $00007FFFF7F7B220 size 192
Found addr = $0000000000400000
$000000000041D5FE line 276 of ../inc/heap.inc
Found addr = $0000000000400000
$0000000000416DED line 355 of ../inc/objpas.inc
Found addr = $0000000000400000
$00000000004DC300 line 420 of fcl-base/src/csvreadwrite.pp
Found addr = $0000000000400000
$00000000004AE028 line 185 of fcl-db/src/base/csvdataset.pp
Found addr = $0000000000400000
$00000000004D633F line 3250 of fcl-db/src/base/bufdataset.pas
Found addr = $0000000000400000
$00000000004CB7B9 line 1240 of fcl-db/src/base/bufdataset.pas
Found addr = $0000000000400000
$00000000004E13EF line 405 of fcl-db/src/base/dataset.inc
Found addr = $0000000000400000
$00000000004E36A5 line 960 of fcl-db/src/base/dataset.inc
So to my understanding on how to check which line is leaking is to check the final file in each reported leak, correct? So, there should be
192 bytes being leaked from
fcl-db/src/base/dataset.inc on line number
960?
Update 5: It seems that the
TCSVParser might not be getting freed correctly, I can see line 185 in
csvdataset being referenced multiple times in the above heaptrc trace. This is line 185 from my local sources:
FParser:=TCSVParser.Create;
I am going to modify the
TCSVDataPacketReader class to FreeAndNil
FParser... and that worked. So, yeah, there is officially a memory leak here in the
csvdataset unit as I thought there was. I am not sure how to submit patches or official bug reports. But here is what I did in the
TCSVDataPacketReader.Destroy destructor to resolve the leak:
destructor TCSVDataPacketReader.Destroy;
begin
If FOwnsOptions then
FreeAndNil(FOPtions);
FreeAndNil(Fline);
if FParser <> Nil then
FreeAndNil(FParser);
inherited Destroy;
end;
Hopefully all this troubleshoot is helpful to someone who might also be facing an odd memory leak within the Free Component Library. I'm actually surprised nobody noticed this leak until now. Perhaps I am doing something wrong in my code to properly free
FParser, if I am, please let me know how I should be using this class to free the parser.
FINAL UPDATE(I swear): So, I was correct that there was a known memory leak in
TCSVDataset, well more like a memory leak in a class it uses, the
TCSVPacketReader class. This leak which I mentioned here is resolved in the latest release candidate 3.2.0 FPC version. Here is their updated code to resolve the leak:
destructor TCSVDataPacketReader.Destroy;
begin
FreeAndNil(FCreateFieldDefs);
If FOwnsOptions then
FreeAndNil(FOPtions);
FreeAndNil(Fline);
FreeAndNil(FParser);
inherited Destroy;
end;
Wish I knew this was an issue and will be resolved in the next FPC release, or I would not have wasted an entire afternoon troubleshooting it... At least I learned a lot more about the internals of FPC, and how to build the RTL and FCL in debug modes.
Recent
