[Solved] My program freezes after running for several days

[Wilko500]

I have been doing an extended test on my solar PV upload program.  Just when I thought it was all OK the program froze after 3 days.  No errors, No warnings.  I have been able to recreate this by force quoting the program and re-starting, and waiting another 4 days!!!  The program is not doing very much, checking for the existence of a text file (<200 lines), parsing it and updating a SQLite database with a few fields. The actual web upload is not active yet.  The various processing tasks are  controlled by a series of 6 timers (TTimer) and all are frozen as wells the GUI interface.  My program was composed on macOS Monterey and tested on older Catalina

Looking at the program details in Mac Activity Monitor I can see some 255 open file references. I don't really understand the stuff in Activity Monitor but it does provide a clue.  The filename referred to is archived, so the program says, so should not be referred to again, but seemingly is.

It does seem that there is a coding error in the conversion from VB6.  The VB6 program is running without this error on the same data and on the same Mac (in virtual windows).  Investigations continue but it is rather time consuming.

My question is somewhat philosophical.  Does FPC/Lazarus have a maximum open file limit?  And if it does why no error, or crash when it is exceeded?

Code: Pascal  [Select][+][-]

1.  launchctl limit maxfiles
2.         maxfiles    256            unlimited    

This from the Mac is soft limit, hard limit (I think)

Any thoughts?

Lazarus 3-7 FPC 3-3-3
macOS Monterey/Catalina

[Thaddy]

On Mac, the maximum number of open files per process is indeed 256 (which is extremely st*p** to keep so many files open for weeks, unless it is a system monitor type of process and that should be a Daemon.)
You can increase it by using ulimit.
On Linux it is - usually - 1024, which is a silly amount over weeks, and can also be increased by using ulimit.
On windows, it is not the handles, but the capacity of the filepaths, 260ísh, which can be set in the registry or the group policy.

Why would you want so many files actually open per process? That is rare.

You can try cat /proc/sys/fs/file-max to see what is the total number the system allows overall.(trillions)

[edit] Better set/getrlimit, because ulimit seems to be obsoleted in the future. Still works, though.

Close files when not needed or waiting, cache them and opening them when approached/needed.

Al this is, I hope explained, OS specific.

[Laksen]

How often are you opening the file? How do you open it?

[Wilko500]

Just for clarity I do not intentionally hold any files open. Every 5 minutes I check if a log file exists. If it does I open it read I, parse the data into 4 other files then close them all. That processing is probably less than a second so most of the time my program is just waiting.

@Thady,  thank you for your explanation. It is helpful. I have no wish to increase the open file limit but If my program has exceeded the limit causing a complete freeze would have expected some form of error.

It is highly likely that I have a coding/logic error but there is no point in posting code until I can identify a suspect proc or function.

My best guess for now is that there is a single data file that does not conform to the expected format and this could be due to a power cut affecting the RS485 data transfer.

And yes, I do validation when the file is parsed but apparently I have not anticipated every eventuality. Investigations continue and I will post again if/when I discover the solution.

[Josh]

hi
create a logfile and after every open and close of a file write to the logfile what file is open or closed and where it was called in your app, you should see mismatches pretty quick.

[dseligo]

Just for clarity I do not intentionally hold any files open. Every 5 minutes I check if a log file exists. If it does I open it read I, parse the data into 4 other files then close them all. That processing is probably less than a second so most of the time my program is just waiting.

@Thady,  thank you for your explanation. It is helpful. I have no wish to increase the open file limit but If my program has exceeded the limit causing a complete freeze would have expected some form of error.

It is highly likely that I have a coding/logic error but there is no point in posting code until I can identify a suspect proc or function.

I propose that you decrease open file limit until you find error in your code - it will freeze your app faster so you don't have to wait several days to see if it's working.

[Wilko500]

Good advice and I do have a log file where I log errors and debug stuff.

I have a debug run in progress. Be a few days for next fail

[Aruna]

Code: Pascal  [Select][+][-]

1.  launchctl limit maxfiles
2.         maxfiles    256            unlimited    

This from the Mac is soft limit, hard limit (I think)

Any thoughts?

Hi @Wilko500, from what you have described, the program's accumulation of open file references suggests that files aren’t being closed after each access. This could be a classic case of resource leakage, especially with file handles.

• Increase maxfiles Temporarily: Use launchctl to increase the maxfiles limit temporarily and check if the program runs longer. This can confirm that an open file cap is indeed the issue.
• Ensure Explicit File Closure: After each file access (both for reading and writing), confirm the files are closed immediately.
• Ensure that SQLite connections and transactions are also properly managed. Open connections should close after each operation to avoid cumulative open file references.

I am not certain but I think why FPC/Lazarus doesn’t throw errors or crash is because as you noted, FPC does not inherently check for file descriptor limits unless an explicit file-open operation fails. Many programs only handle general exceptions, so once the file descriptor limit is met, subsequent file operations can silently fail if not individually checked for errors.

So to resolve this I would:

• Ensure file closures after each access, especially in recurring operations.
• Temporarily increase maxfiles as a test to validate the open file limit as the cause.
• Check SQLite connection handling, as these can contribute to the open file count.

Hope this helps

[Wilko500]

I propose that you decrease open file limit until you find error in your code - it will freeze your app faster so you don't have to wait several days to see if it's working.

What a good idea, thank you

[Thaddy]

@Thady,  thank you for your explanation. It is helpful. I have no wish to increase the open file limit but If my program has exceeded the limit causing a complete freeze would have expected some form of error.

Well, I actually expected that, because if a process is out of its allocated resources, there is no - easy- way to recover unless you have pre-allocated an exception object for that situation beforehand. (Similar to how FPC and Delphi handle EOutOfMemory: with a pre-allocated object)
It is simply a fatal exception. Unix like OS's handle that on process level, whereas Windows handles it on system level.(the latter is worse, because you can't recover at all)
Keep a simple map with a max of, say 200 handles, and close them when not needed or exceeded. That is per process on Mac. And... really...256 open handles at one time and in the same process?
Since everything is a handle on Unix, you also risk losing all your devices....

[MarkMLl]

How often are you opening the file? How do you open it?

More to the point, how do you close it?

How are you parsing the directory? Does this require opening a directory which you aren't closing?

How are you talking to the RS485 port? Are you explicitly closing it after opening it?

Are you doing anything that involves sockets- UDP as much as TCP? Is every open accompanied by a close?

MarkMLl

[Wilko500]

Ok. This version of the program is not directly access the RS485, yet.  That will follow in due course.  This is how I open & close files

Code: Pascal  [Select][+][-]

1.   //Check for existing temp files, delete if found
2.   DatFiles:= TStringList.Create;
3.   FindAllFiles(DatFiles, TempPath, '*.dat', False); //find all temp data files
4.   iDat:=datFiles.Count;
5.   If iDat > 0 Then
6.       Begin
7.       //Files found to delete
8.       For i:= 0 To iDat - 1 Do
9.           Begin
10.           DeleteFile(DatFiles[i]);
11.       End;{For}
12.   End;{If}
13.   //Ceate & open files for input and output
14.   Assign(fLog, TempPath + '/' + fDate + '.log'); //Input log file
15.   Assign(fDat, TempPath + '/' + fDate + '-DA.dat'); //Output inverter info file
16.   Assign(fIn, TempPath + '/' + fDate + '-IN.dat'); //Output inverter info file
17.   Assign(fSys, TempPath + '/' + fDate + '-SY.dat'); //Output inverter info file
18.   Assign(fUn, TempPath + '/' + fDate + '-UN.dat'); //Output inverter info file
19.   Reset(fLog);
20.   ReWrite(fDat);
21.   ReWrite(fIn);
22.   ReWrite(fSys);
23.   ReWrite(fUn);
24.   While Not EOF(fLog) Do
25.       Begin
26.       //Parse file one line at a time  
27.       //do stuff with lines from file
28.   //And close files
29.   Close(fLog);
30.   Close(fDat);
31.   Close(fIn);
32.   Close(fSys);
33.   Close(fUn);
34.   DatFiles.Free;
35.  

Very simple coding.

So I started a new debug run with additional debug line in the suspected procedure and the outcome was,  disappointing,  the processing worked as it was supposed to!  Further, watching the Activity Monitor there was no accumulation of open files as I had seen on previous occasions.  I did not change the existing code just added a few WriteLn's and then ran in terminal mode.

Tomorrow at daylight my inverter will start provide live data and I will see what happens.  I'll run it for a few days, might even try a power failure to see how it recovers. 

It is beginning to look like a case of bad data caused by event unknown (but probably a power failure) that is not being handled correctly.  I'll post again when I get more information from another extended test.

[MarkMLl]

Ok. This version of the program is not directly access the RS485, yet.  That will follow in due course.  This is how I open & close files

That's not "a version of the program", it's an incomplete code fragment.

Where's FindAllFiles defined?

MarkMLl

[Thaddy]

A power failure does not keep files open after a restart.
A restart initializes all processes and resources from scratch, which means any previously open file handles are closed.
So this is unlikely unless you have some recovery mechanism built-in that tries to open too many files at once.
Damaged files because of a power failure is another matter, but also does not use handles.
Also, long running systems need a PSU and these are not expensive anymore.

[Josh]

Hi

I do not see any IOERROR checking on any of the file operations.
It is not wise to assume that the operations have succeeded.

if its a gui app, create a global variable MyOpenFileCount, set it to zero, then after every successful open of file increment it,after every successful close decrement it, create an idle timer and display its value somewhere on your gui, it should match watch the system reports. You can then spot a run away situation, and if one starts you can handle it gracefully.