[Solved]Memory Leak and/or program corruption on MacOs

[dbannon]

Well spotted Ron, I had forgotten about that development.

The setting used are also included in pics and interestingly I already had those setting enabled in my development environment but no line numbers as the "wrong" versions used.

Not so much 'wrong' as not new enough ?  I also suspect that Lazarus 3.4 would be new enough, the important thing is FPC trunk.

The post you referred me to spoke of having to use the trunk version(s).  I tried to install the trunks of both Laz & FPC using FPCUpDeluxe but found that anchor docking although installed did not do any docking so that test was abandoned so I cannot say whether the current trunk would show line numbers or not. 

I'd suggest that using trunk-trunk for development work, especially for release might be a bit brave. Just keep the bleeding edge kit for leak detection. Have your anchor docking in the fpc-fixes/lazarus-3.4 tool chain.

I do find it surprising that this subject has been spoken of many times and that there IS a solution but it remains something of a mystery

Its only a mystery because no one has documented the process (other than a forum posts).  Right now, you appear to be the resident expert on the topic, please consider documenting what you find.

In addition there are many unfreed blocks that do not refer to any of my code.  These I do not understand at all.  Next I shall review my code in the light of the line numbered dumps to see if I can reduce the magnitude of the problem.  Maybe some more posts will follow as I try to make sense of it.

They are possibly fpc units called from your code. Do you have the rtl/fcl compiled with debug information ?  I suspect fpupdeluxe won't do that ? My approach, and only because its my way of thinking, is to start with a fpc322 (probably from tarball but it does not particularly matter) and use that to build both a debug trunk FPC and a trunk Lazarus.

The real innovation of that other post was processing the dump outside of lazarus. Its not particularly startling, its just having all that stuff happening within Lazarus has made us soft !

If you are getting even some line numbers, and they are correct, then please consider a step by step documentation on the wiki. I'm happy to help there necessary. The others who follow will thank you !

Davo

[BrunoK]

If you dont show more code, nobody is going to be able to help you.

It is the same for the other poster who has difficulties with a tiny 3500 records database in sqlite3.
Sqlite3 like any other db's is Not that easy to starts with.
What is important is to get a first working application.

[TRon]

So the simple answer to "can I get heaptrc dump with line numbers" is yes and no.

Thank you very much for trying and sharing your findings with us Wilko500.

I have taken the liberty to post your findings verbatim into this post in this thread so that all information is gathered into one place and not derail this thread off-topic any further. If you have any objection to that then please let me know and I'll remove that post (In which case I will link to the post with your findings instead, hopefully with your blessing in that case).

I am aware that perhaps using trunk/trunk is not a solution for your current situation but it is encouraging enough to perhaps be able to advise people and keep an eye on things and see when these changes make it into a mainstream release.

Well spotted Ron, I had forgotten about that development.

Thank you Davo. Especially for all your grunt research that you did.

Hopefully the negative answer we need to keep giving is able to turn into a positive one.

[Wilko500]

Small additional hint: If you use leak trace then use it right from the start of your development. I personally also sometimes forget to do that from the get go and it is much harder to backtrack introduced leaks later on.

If you do it right from the start then whenever you introduced a leak it will show up immediately, indirectly telling you that what you just did is not the way you should implemented things (or that the current implementation requires additional work).

Ofc. all this in hindsight and with 2 cents.

A very useful tip.  And thank you for posting my comment of the other Mac post. I think this forum is truly amazing resource, with useful info and helpful contributors. I just wish I was smart enough to contribute more.  Maybe one day

If you dont show more code, nobody is going to be able to help you.

It is the same for the other poster who has difficulties with a tiny 3500 records database in sqlite3.
Sqlite3 like any other db's is Not that easy to starts with.
What is important is to get a first working application.

I fully accept your comment but in this case I didn't think there was any value is providing more code since it seemed very clear to me that it was quite likely that there was a combination of causes and that it was necessary to try to limit the scope of the issues thus improving the likelihood of finding a solution.

So an update.  The memory leak is partially resolved.  The search for it was difficult and long winded.  It revealed several issues with my code and perhaps more importantly my lack of experience in this sort of debugging!  While making small changes to the code to try to identify the leak source I made things worse and created an access violation that took me ages to find.  An interesting side line was that I also discovered an unhandled data error.  On reading the input data file I carry out many validations before attempting to write to the database.  One of those tests was for blank lines and the simple test length of line =0 failed to find a few lines with a single space in column one.  This particular problem has not been observed in the VB6 version of my upload program using the same test and has been running for more than 9 years. However, the main source of the leaks was to do with SQLite, or perhaps more accurately, how I was opening and closing the database. In particular the close was lacking.  What I have now is a closeDb Procedure.  Im not certain that the two Closes's are necessary. I added the boolean flag to ensure that I don't try to close a non existent database.

Code: Pascal  [Select][+][-]

1. Procedure CloseDb();
2. Begin
3.     If IsDbConnected = True Then
4.         //Database connected, close
5.         Begin
6.         dBQuery.Close;
7.         dBConn.Close;
8.         dBTrans.Free;
9.         dBConn.Free;
10.         dBQuery.Free;
11.         //Set flag to say database closed
12.         IsDbConnected:=False;
13.         End
14.      Else
15.         Begin
16.         //Database not connected, do nothing
17.         WriteLn('Database not connected, do nothing');
18.      End;{If}
19. End;{Procedure}

I also found a few TSTringList's that were not explicitly free'ed but called hundreds of times .  I still have a small memory leak and I will post that in a separate thread since it seems that it has a different cause.  I removed ALL my code and that small leak remains.

So now my stress test of processing some 10,000 upload records works without fault.  I am happy with that and can now proceed to the more complicated upload to web process.  So I thank you all for your comments and help and will mark this thread as solved.