Recent

Author Topic: [Solved]Memory Leak and/or program corruption on MacOs  (Read 5757 times)

Wilko500

  • Full Member
  • ***
  • Posts: 180
[Solved]Memory Leak and/or program corruption on MacOs
« on: August 23, 2024, 11:22:33 pm »
I am unable to get LeakViewer to display line numbers on MacOs.  I have attempted to follow instructions found in various forum posts without success. My app reads data files, parses then and writes data to SQLite database.  It fails with try-except error "unable to open database" after processing some 6500 records.  the data file has been excluded as the cause because it fails when re-running without the failing data file simply causes it to fail on the next data file.
The database error is most likely NOT the cause but a consequence of some other problem, maybe memory leak.  I say this because just before failure the app reports incorrect values in its processing log eg 1349206016 duplicate records skipped.  In addition two command buttons fail to act, both being controlled by a Boolean variable.  Looks like many many variables have become corrupted including whatever is related to database connections.

The attached pic is of my heap trace AFTER I have clicked resolve and pointed to the app executable.  In settings I have enabled -gl and -gh, set debugger to "Standard LLDB [LLDB debugger alpha] with fpdebug.

I have included the failing procedure code

Have I missed something or could this be an issue with MacOs?

Two questions, can a severe memory leak cause internal corruption and is there a way to view line numbers on MacOs.  The pic shows a simple test not the full app failure. 


Code: Pascal  [Select][+][-]
  1. //------------------------------------------------------------------------------
  2. //Procedure : DataBaseUpdate
  3. //Author    : Richard
  4. //Date      : 20/05/2024
  5. //Purpose   : To read data from the -.DA.dat file and upload the data
  6. //          : to the PvUpload database
  7. //          : fDate in format yyyy-mm-dd is used for file operations only
  8. //          : sDate, derived from fDate in format yyyy/mm/dd is used for date comparisons
  9. //------------------------------------------------------------------------------
  10. Procedure UpDateDataBase(fDate: String; VAR iDup, iUpd, iZero: Int32; sComplete, sSysName, sModel, sSysDate:String);
  11. Var
  12.   i:           Integer;             //Work var
  13.   fPath:       String;              //For datafile access
  14.   fNo:         Text;
  15.   strLine:     String;
  16.   arData:      TStringArray;        //For reading and splitting data line
  17.   arDataLen:   Int32;
  18.   iNewRecs:    Int32;               //Record counts
  19.   iDupRecs:    Int32;
  20.   iZeroRecs:   Int32;
  21.   sSystemName: String;              //Returned by ReadInFile
  22.   sSystemDate: String;
  23.   sColNames:   String;              //returned by ReadUnFile
  24.   sUnits:      String;
  25.   sTimeOn:     String;              //Returned by ReadSyFile
  26.   sTimeOff:    String;
  27.   sEnergyToday:Single;
  28.   sEnergyLife: Single;
  29.   arInp:       ar19;                //An array ofelements used for writing a new Time record
  30.   sUnitId:     Int32;
  31.   sDayId:      Int32;               //Work var
  32.   sDate:       String;              //Date in format yyyy/mm/dd
  33.   iInvId:      Int32;               //Returned InverterId
  34.   Complete:    String;              //Day's data completed status
  35.   sTime:       String;              //Work var
  36.   InverterID:  Int32;               //InverterID returned by GetInverterId
  37.   DayId:       Int32;               //Work var
  38.   UpLoaded:    String;              //Day uploaded status
  39.   sMPPT:       String;              //Work var
  40.   theLine:     String;              //Debug, holds a line indicator
  41.  
  42. Begin
  43. Try
  44.  
  45. //Need to get Model & MPPT now so we can call ImnverterID, required for GetDayID
  46. //So open -DA.dat file, get it and close
  47. theLine:='AA';
  48. GetModelMPPT(fDate);  //Gets gSys. Model & MPPT
  49.  
  50. //Extract relevant information from temp files -In, -Un, -Sy
  51.  
  52. //First Information file -In
  53. //ReadInFile(fDate, sSystemName, sSystemDate); //Gets gSys. SystemName & SystemDate
  54. ReadInFile(fDate); //Gets gSys. SystemName & SystemDate
  55.  
  56. //Second Units file -Un
  57. ReadUnFile(fDate, sColNames, sUnits); //Gets gSys ColNames & ColUnits
  58.  
  59. //Third System file -Sy
  60. //Gets gSys. SysTimeOn & SysTimeOff & EnergyToday & EnergyLife
  61. //ReadSyFile(fDate, sTimeOn, sTimeOff, sEnergyToday, sEnergyLife);
  62. ReadSyFile(fDate);
  63.  
  64. //But we also do need ColNames & ColUnits for GetInverterID later in code
  65. //sUnitId:=GetUnitId(sColNames, sUnits);  //Gets gSys ColNames & ColUnits
  66. sUnitId:=GetUnitId;  //Gets gSys ColNames & ColUnits
  67.  
  68. //And InverterID for call to GetDayId
  69. //Because it comes from reading the -DA.dat file
  70. //InverterID:=GetInverterId(gSys.Model, sSystemName, gSys.MPPT, sColNames, sUnits);
  71. InverterID:=GetInverterId;
  72.  
  73. //And get DayId for database inserts
  74. theLine:='BB';
  75. sDayId:=GetDayId(fDate, gSys.SysTimeOn, gSys.SysTimeOff, gSys.EnergyToday, gSys.EnergyLife, InverterID);
  76. //sDayId:=GetDayId(fDate, sTimeOn, sTimeOff, sEnergyToday, sEnergyLife, 99); //iInvId not known yet
  77.  
  78. //Now loop through all data lines in -DA.dat file and process
  79.   iDupRecs:=0;
  80.   iNewRecs:=0;
  81.   iZeroRecs:=0;
  82.   //Assign & open data file
  83.   fPath:=g.TempPath + '/' + fDate + '-DA.dat';
  84.   Assign(fNo, fPath);
  85.   Reset(fNo);
  86.   While Not Eof(fNo) Do
  87.       Begin
  88.       ReadLn(fNo, strLine);
  89.       //Turn this string into a record of time,  need new function
  90.       //08:02;2;PVI-3.6-OUTD-UK;D;131.4;0.0;0.0;92.8;0.0;0.0;238.8;0.8;0.0;18.9;17.8;0.0;14.1;0;0;
  91.       //Ignore trailing ';' thus 19 elements running 0 to 18 in arData[i]
  92.       If RightStr(strLine, 1) = ';' Then strLine:= MidStr(strLine,1, Length(strLine)-1);
  93.       arData:=strLine.Split([';']);
  94.       arDataLen:=length(arData);   //TSTringArray from -DA.dat file
  95.       //Create array[1..19]  of string for input values
  96.       For i:= 0 to arDataLen - 1 Do   //Remember length is number of elements, they start at zero
  97.           Begin
  98.           arInp[i+1]:= arData[i];
  99.       End;{For}
  100.       //WriteLn(strLine);
  101.       If Not IsRecAllZero(arInp) Then
  102.           //Ensure that record is not all zero's
  103.           Begin
  104.           //Use this record
  105.           sTime:= arInp[1];   //arInp[1] = time
  106.               //Does time exist in table times
  107.               If DoesTimeExist( sDayId, sTime) = False Then
  108.                   Begin
  109.                   //Ok to insert new record in table Times but first
  110.                   //We need InverterId for SQL insert plus items returned by GetInverterID
  111.                   sMPPT:=arInp[4];
  112.                   sModel:= arInp[3];
  113.                   DayId:=sDayId;  //Is returned by function call DoesTimeExist
  114.                   UpLoaded:='N';  //As this is a new record, always = 'N'
  115.                   //InverterID:=GetInverterId(sModel, sSystemName, sMPPT, sColNames, sUnits);
  116.                   //is this necessary                InverterID:=GetInverterId;
  117.                   dbTrans.Active:=True;
  118.                   theLine:='CC';
  119.                   dbQuery.SQL.Text:='INSERT INTO TIMES (DayID, Time, Uploaded, Model, Address, MPPT, ' +
  120.                                     'VDC1, IDC1, PDC1, VDC2, IDC2, PDC2, VAC, ' +
  121.                                     'IAC, PAC, TINV, TINT, Energy, RISO, ILEAK, GenFreq) ' +
  122.                                     'VALUES (' + IntToStr(DayID) + ', ' +   //DayId
  123.                                     QuotedStr(arInp[1]) + ', ' +            //Time
  124.                                     QuotedStr(Uploaded) + ', ' +            //Uploaded
  125.                                     IntToStr(InverterID) + ', ' +           //Model
  126.                                     arInp[2] + ', ' +                       //Address
  127.                                     QuotedStr(arInp[4]) + ', ' +            //MPPT
  128.                                     arInp[5] + ', ' +                       //VCD1
  129.                                     arInp[6] + ', ' +                       //IDC1
  130.                                     arInp[7] + ', ' +                       //PDC1
  131.                                     arInp[8] + ', ' +                       //VDC2
  132.                                     arInp[9] + ', ' +                       //IDC2
  133.                                     arInp[10] + ', ' +                      //PDC2
  134.                                     arInp[11] + ', ' +                      //VAC
  135.                                     arInp[12] + ', ' +                      //IAC
  136.                                     arInp[13] + ', ' +                      //PAC
  137.                                     arInp[14] + ', ' +                      //TINV
  138.                                     arInp[15] + ', ' +                      //TINT
  139.                                     arInp[16] + ', ' +                      //Energy
  140.                                     arInp[17] + ', ' +                      //RISO
  141.                                     arInp[18] + ', ' +                      //ILEAK
  142.                                     arInp[19] + ');';                       //GENFREQ
  143.                   dbQuery.ExecSQL;
  144.                   dbTrans.Commit;
  145.                   dbQuery.Free;
  146.                   dbTrans.Free;
  147.                   Inc(iNewRecs);
  148.                   //WriteLn('Insert ' + IntToStr(iNewRecs));
  149.                   End
  150.               Else
  151.                   Begin
  152.                   //Time already exists
  153.                   Inc(IdupRecs);
  154.                   //WriteLn('Duplicate time ignore : ' + arInp[1]);
  155.               End;{If DoesTimeExist}
  156.           End
  157.       Else
  158.           Begin
  159.           //A zero record, ignore
  160.           Inc(iZeroRecs);
  161.       End;{If IsRecAllZero}
  162.   End;{While}
  163.  
  164.   If iNewRecs >= 0 Then iUpd:=iNewRecs;
  165.   If iDupRecs >= 0 Then iDup:=iDupRecs;
  166.   If iZeroRecs >= 0 Then iZero:=iZeroRecs;
  167.   Close(fNo);
  168.  
  169. Except
  170. On E: EDataBaseError Do
  171.   Begin
  172.   //Needs Db in Uses statement
  173.   WriteLn(theLine + '-- ' + 'Raised Exception: ' + E.ClassName + ' With message: ' + E.Message);
  174.   End;
  175. End;{Try}
  176. End;{Procedure UpDateDataBase}
« Last Edit: September 07, 2024, 09:55:30 pm by Wilko500 »
MacBook Pro mid 2015 with OS Monterey 12.7.6
FPC 3.2.3 Lazarus 3.7
FPC 3.2.2 Lazarus 3.4

dbannon

  • Hero Member
  • *****
  • Posts: 3744
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Memory Leak and/or program corruption on MacOs
« Reply #1 on: August 24, 2024, 02:11:34 am »
Its an old story I am afraid Wilko, Leakview does not work on the Mac, if you search the forum there was some discussion about it some years ago, it just does not work.

Unless your code contains Mac specific things, you can take it to another platform and debug the leaks there ?  I'm thinking of a Virtual Machine with, perhaps, Linux as a client. Debugging is always easier under Linux.

As far as a leak causing memory corruption ?  I really don't know but I have never seen it and it does seem unlikely to me. But that's not authoritative.

Davo
Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

BrunoK

  • Hero Member
  • *****
  • Posts: 766
  • Retired programmer
Re: Memory Leak and/or program corruption on MacOs
« Reply #2 on: August 24, 2024, 04:39:47 am »
Lines 145, 146.
Code: Pascal  [Select][+][-]
  1.   dbQuery.Free;
  2.   dbTrans.Free;
Maybe next interation needs them. It is not clear where you recreate them.

Wilko500

  • Full Member
  • ***
  • Posts: 180
Re: Memory Leak and/or program corruption on MacOs
« Reply #3 on: August 24, 2024, 06:36:07 am »
Its an old story I am afraid Wilko, Leakview does not work on the Mac, if you search the forum there was some discussion about it some years ago, it just does not work.

Unless your code contains Mac specific things, you can take it to another platform and debug the leaks there ?  I'm thinking of a Virtual Machine with, perhaps, Linux as a client. Debugging is always easier under Linux.

As far as a leak causing memory corruption ?  I really don't know but I have never seen it and it does seem unlikely to me. But that's not authoritative.

Davo
Thank you for confirming that.  Its kind of what I was expecting. And yes I did see some old posts referring to this but hoped that there might have been an update.
Sadly my code does contain some Mac specific stuff like where to store files and configurations, the nature of developing on Mac.  They can be changed or adapted for cross platform but I was hoping to avoid that
MacBook Pro mid 2015 with OS Monterey 12.7.6
FPC 3.2.3 Lazarus 3.7
FPC 3.2.2 Lazarus 3.4

Wilko500

  • Full Member
  • ***
  • Posts: 180
Re: Memory Leak and/or program corruption on MacOs
« Reply #4 on: August 24, 2024, 06:39:04 am »
Lines 145, 146.
Code: Pascal  [Select][+][-]
  1.   dbQuery.Free;
  2.   dbTrans.Free;
Maybe next interation needs them. It is not clear where you recreate them.
Thank you for your suggestion. These were not in original code, added after I read that everything created should be freed!  With or without makes no difference.
MacBook Pro mid 2015 with OS Monterey 12.7.6
FPC 3.2.3 Lazarus 3.7
FPC 3.2.2 Lazarus 3.4

BrunoK

  • Hero Member
  • *****
  • Posts: 766
  • Retired programmer
Re: Memory Leak and/or program corruption on MacOs
« Reply #5 on: August 24, 2024, 10:14:34 am »
Thank you for your suggestion. These were not in original code, added after I read that everything created should be freed!  With or without makes no difference.
It doesn't work exactly like that. Created instances must be freed before the program ends or as soon as they are not used anymore. If they are used during a run, well, their existence is required.

You are likely having a case of USE AFTER FREE error, that is, you did Something.FREE but further in you code you reference the thing you freed. It may work but there is a strong possibility the the memory used by "Something" has been attributed to another variable/string thus garbaging other variables.

I suggest you search for all ????.Free (Search in files '.free') in your project and look if you prematuredly free objects that should still exist.

Why I note the 2 free's mentionned above is that if you created as components on the form, you should let the form free them on application terminate.

TRon

  • Hero Member
  • *****
  • Posts: 4377
Re: Memory Leak and/or program corruption on MacOs
« Reply #6 on: August 24, 2024, 10:21:48 am »
Small additional hint: If you use leak trace then use it right from the start of your development. I personally also sometimes forget to do that from the get go and it is much harder to backtrack introduced leaks later on.

If you do it right from the start then whenever you introduced a leak it will show up immediately, indirectly telling you that what you just did is not the way you should implemented things (or that the current implementation requires additional work).

Ofc. all this in hindsight and with 2 cents.
Today is tomorrow's yesterday.

Wilko500

  • Full Member
  • ***
  • Posts: 180
Re: Memory Leak and/or program corruption on MacOs
« Reply #7 on: August 24, 2024, 11:35:12 am »
Thank you TRon.  That is very smart! 

This app was originally created in VB6 and is still running (24 - 7) and has been for more than 10 years.  I am thus quite confident of the overall design and logic.  The purpose of moving it to FPC was to get me away from depending on Windoze. Has been significantly more difficult than I expected.  I am obviously doing something in a way that the compiler doesn't like very much as my memory leaks are not just a few but on my 6,500 record stress test many 10's of thousands.

It would have been good to have had a heads up right at the beginning so I could have corrected whatever it is I'm not getting right.
MacBook Pro mid 2015 with OS Monterey 12.7.6
FPC 3.2.3 Lazarus 3.7
FPC 3.2.2 Lazarus 3.4

cdbc

  • Hero Member
  • *****
  • Posts: 2687
    • http://www.cdbc.dk
Re: Memory Leak and/or program corruption on MacOs
« Reply #8 on: August 24, 2024, 11:42:15 am »
Hi
Then it's pretty safe to assume, that your leak is situated somewhere repetitive...
So, scrutinize those areas first, with so many, you're bound to stumble upon something 'fishy'  %)
Just my "Nickel's Worth"
Regards Benny
If it ain't broke, don't fix it ;)
PCLinuxOS(rolling release) 64bit -> KDE6/QT6 -> FPC Release -> Lazarus Release &  FPC Main -> Lazarus Main

jamie

  • Hero Member
  • *****
  • Posts: 7610
Re: Memory Leak and/or program corruption on MacOs
« Reply #9 on: August 24, 2024, 03:39:03 pm »
Lines 145, 146.
Code: Pascal  [Select][+][-]
  1.   dbQuery.Free;
  2.   dbTrans.Free;
Maybe next interation needs them. It is not clear where you recreate them.
Thank you for your suggestion. These were not in original code, added after I read that everything created should be freed!  With or without makes no difference.

Regardless, if it makes no difference by not freeing the components, you can't do that unless you created them locally in your code.

If these are components you dropped on the form, then you need to ensure they stay alive.

The only other idea that comes to mind is closing the query but not freeing the component in that code, because the next time around that component will be no more.

you most likely have other parts of the program that is doing the same and it's compounding the problem.


The only true wisdom is knowing you know nothing

Wilko500

  • Full Member
  • ***
  • Posts: 180
Re: Memory Leak and/or program corruption on MacOs
« Reply #10 on: August 25, 2024, 01:07:00 pm »
One small step!!!!
First thanks for all comments and suggestions. They remind me that I have such a lot to learn.
As it happens I still have my virtual Win7Pro on my development MacBook so I thought I’d see if I can run my code on that. And so I can and get heaptrc dump with line numbers. At least as long as the main form opens and is closed before I click the start button. Getting it to run is going to be a challenge. Stuff I expected to work doesn’t eg anchordocking is problematic and my TProcess call crashes Lazarus. That being said my attempt to bypass Mac specific code is quick and dirty intended only to confirm that I can get heaptrc line numbers.

Now I have to start over with a more structured approach to compiling my Mac code in Windows. Perhaps I’ll try to make the code cross compilable. Might take a while though.

On the successful, although limited heaptrc with line numbers I find two classes created in form activation that are not freed. Not the primary cause of my memory leak but an indication of where to look. I will post again when I have made more progress.
Thank you all
MacBook Pro mid 2015 with OS Monterey 12.7.6
FPC 3.2.3 Lazarus 3.7
FPC 3.2.2 Lazarus 3.4

TRon

  • Hero Member
  • *****
  • Posts: 4377
Re: Memory Leak and/or program corruption on MacOs
« Reply #11 on: August 25, 2024, 01:41:00 pm »
@wilko500:
On a side note: not that long ago someone seem to have figured out how to create stack-traces with line-numbers on a mac. See here

There seem to be some specific conditions that have to be met so please don't keep your hopes up too high (there is also nothing wrong with debugging on another platform)

If you have the time it would be nice if you would be able to verify these findings (I am not sure if it is bound to specific architecture or otherwise and unfortunately can't check myself).
« Last Edit: August 25, 2024, 01:43:10 pm by TRon »
Today is tomorrow's yesterday.

Wilko500

  • Full Member
  • ***
  • Posts: 180
Re: Memory Leak and/or program corruption on MacOs
« Reply #12 on: August 25, 2024, 01:51:00 pm »
TRon, thanks for that. I’ve bookmarked that post and will look at it. Sure would be cool if it works. I’ll post the outcome.
MacBook Pro mid 2015 with OS Monterey 12.7.6
FPC 3.2.3 Lazarus 3.7
FPC 3.2.2 Lazarus 3.4

TRon

  • Hero Member
  • *****
  • Posts: 4377
Re: Memory Leak and/or program corruption on MacOs
« Reply #13 on: August 25, 2024, 02:00:23 pm »
You're more than welcome Wilko500.

Would be nice if you could do that for us (at your convenience). If you post the results then please use that same thread and share as much details as possible.

It is a longstanding issue and get asked about a lot and there is almost no information available to be able to backup any claims. Till that thread it was assumed, by grunt work from others, that it was not possible to accomplish for a mac. Your findings would be able to provide more information on the matter and be able to help out others with the same problem.

TIA.
Today is tomorrow's yesterday.

Wilko500

  • Full Member
  • ***
  • Posts: 180
Re: Memory Leak and/or program corruption on MacOs
« Reply #14 on: August 25, 2024, 11:10:43 pm »
So the simple answer to "can I get heaptrc dump with line numbers" is yes and no. It appears to depend on what versions of FPC and Lazarus are being used.  My project development environment is Laz/Fpc =34/322 and that does NOT produce line numbers.  I also have a test environment of Laz/FPC = 399/331 that DOES produce line numbers.   Both were installed using FPCUpDeluxe (about screens attached.

The setting used are also included in pics and interestingly I already had those setting enabled in my development environment but no line numbers as the "wrong" versions used.

The post you referred me to spoke of having to use the trunk version(s).  I tried to install the trunks of both Laz & FPC using FPCUpDeluxe but found that anchor docking although installed did not do any docking so that test was abandoned so I cannot say whether the current trunk would show line numbers or not. 

I do find it surprising that this subject has been spoken of many times and that there IS a solution but it remains something of a mystery as to what exactly must be installed to make it happen. So my best guess at present is to install Laz/FPC = 399/331 using FPCUpDeluxe and then you should get line numbers in your heaptrc dump. Naturally if you have the necessary skills to download source and compile then there may be other options.  At present I do not have those skills.

Now, the subject of my original post about memory leaks and program corruption I have no answer, YET.  The dumps with line numbers are very long and I have had just a very quick look.  The results are interesting to say the least.  I have easily identified that two classes I created are not freed.  Ok, and easy fix and a learning point for me.  More interesting is a repeating set of references to a routine that opens an SQLite database does a lookup and closes the database (perhaps 1000's of times).   For this I thought that having opened a database then closing it would be enough.  Apparently not.

In addition there are many unfreed blocks that do not refer to any of my code.  These I do not understand at all.  Next I shall review my code in the light of the line numbered dumps to see if I can reduce the magnitude of the problem.  Maybe some more posts will follow as I try to make sense of it.

A huge thank you to all who have commented on this post. If I can assist with further tests (on Mac stuff) then yes, I'm up for that. I would be very pleased to contribute to an amazing product.
MacBook Pro mid 2015 with OS Monterey 12.7.6
FPC 3.2.3 Lazarus 3.7
FPC 3.2.2 Lazarus 3.4

 

TinyPortal © 2005-2018