Recent

Author Topic: ZEOSLIB increasing memory usage inside TThread  (Read 998 times)

nmra

  • Newbie
  • Posts: 2
ZEOSLIB increasing memory usage inside TThread
« on: July 21, 2022, 09:34:23 pm »
Hello all,

I tried to run a SELECT SQL to retrieve some data from MySql database. For this, I had created a TZQuery inside a TThreadn to avoid freezing the GUI of the application.

But, after the end of each consult (tested with 100,000+ rows, just to "force" a freeze) the memory was increasing instead of returning to the initial values (even after FreeAndNil(ZQuery) and FreeAndNil(TThread)) increasing ~2mb to each query executed (verified using Windows taskman.)

The test:

1 - Click button1 to start thread and query

2 - Wait callback

3 - Click button2 to finish and free query

Code:
Code: Pascal  [Select][+][-]
  1.     { TForm }
  2.    
  3.     procedure TForm1.Button1Click(Sender: TObject);
  4.     begin
  5.       //DBQuery: TSubThreadTest;
  6.       DBQuery:= TSubThreadTest.Create(SQLStatement, true, ZCon, @onQueryCallBack);
  7.       DBQuery.Start;
  8.     end;
  9.    
  10.     procedure TForm1.Button2Click(Sender: TObject);
  11.     begin
  12.       FreeAndNil(DBQuery);
  13.  
  14.      //Tried using WaitFor + Free, but not difference.
  15.     end;
  16.    
  17.     procedure TForm1.onQueryCallBack(fail: boolean; query: TZQuery);
  18.     begin
  19.       lLastExec.Caption:= FormatDateTime('hh:mm:ss', Now);
  20.     end;  
  21.    
  22.     { TSubThreadTest }
  23.    
  24.     constructor TSubThreadTest.Create(SQLStatement: string; toExec: boolean;
  25.       connector: TZConnection; EOnConclude: TSubThreadTestCallBack);
  26.     begin
  27.       inherited Create(true);
  28.    
  29.       //main: TZQuery;
  30.    
  31.       main:= TZQuery.Create(nil);
  32.       main.Connection:= connector;
  33.       main.SQL.Text:= SQLStatement;
  34.    
  35.       isToExec:= toExec;
  36.    
  37.       OnConclude:= EOnConclude;
  38.    
  39.       FreeOnTerminate:= False;
  40.     end;
  41.    
  42.    
  43.     procedure TSubThreadTest.callback;
  44.     begin
  45.       if assigned(OnConclude) then
  46.         OnConclude(true, main);
  47.     end;
  48.    
  49.     procedure TSubThreadTest.Execute;
  50.     begin
  51.       if isToExec then
  52.         main.ExecSQL
  53.       else
  54.         main.Open;
  55.    
  56.       Synchronize(@callback);
  57.    
  58.       FreeAndNil(main);
  59.     end;
  60.  

First thread create and call: 24mb memory usage

Second thread create and call: 26,5mb memory usage

Etc...

malcome

  • Jr. Member
  • **
  • Posts: 80
Re: ZEOSLIB increasing memory usage inside TThread
« Reply #1 on: July 22, 2022, 12:33:11 pm »
So what?
Mysql is waiting your next order with the cache thing, i suppose.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 10291
  • FPC developer.
Re: ZEOSLIB increasing memory usage inside TThread
« Reply #2 on: July 22, 2022, 01:17:50 pm »
This is normal. If you allocate large blocks, the memory will be returned to the OS on deallocated.

Smaller allocations are pooled and reused, and that is probably what tdataset/zeos does.

nmra

  • Newbie
  • Posts: 2
Re: ZEOSLIB increasing memory usage inside TThread
« Reply #3 on: July 22, 2022, 01:59:20 pm »
Thanks for the replies.

But, this don't cause memory leak after?

malcome

  • Jr. Member
  • **
  • Posts: 80
Re: ZEOSLIB increasing memory usage inside TThread
« Reply #4 on: July 22, 2022, 09:37:27 pm »
You have some ways.
  • Stop using Tthread
  • Stop using Zeos
  • Stop using Mysql
  • Stop using Lazarus
If you find a person who caused memory leak and report it, then lots of users say you thanks!

rvk

  • Hero Member
  • *****
  • Posts: 4948
Re: ZEOSLIB increasing memory usage inside TThread
« Reply #5 on: July 22, 2022, 09:55:47 pm »
There isn't really a leak.
There is only a leak if it is reported when you enable heaptrc in project options.
Until then this is just normal behavior.

Memory is just used and sometimes during a programs lifetime even more memory is used. At some point this should stabilize. All memory will be automatically released at end of program.
Here is a similar discussion (about Delphi but the priciple is the same)
https://stackoverflow.com/questions/2569661/why-does-my-delphi-programs-memory-continue-to-grow

@nmra what happens if you click the button 10 times? And 60 times? Does it keep increasing with 24MB? Taskmanager isn't really the tool to check for memory leaks.

BTW, I would use WaitFor + Free instead of directly freeing the thread.

Also... I'm not entirely sure what you are doing (using a TZConnection from the main thread inside a thread) is thread safe. It could be that each thread would need it's own  TZConnection. But maybe someone with more ZEOS experience can tell us that.
« Last Edit: July 22, 2022, 10:03:23 pm by rvk »

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1086
Re: ZEOSLIB increasing memory usage inside TThread
« Reply #6 on: July 25, 2022, 09:23:11 am »
The amount of memory your application allocates has no direct relation to the amount of memory it actually uses. The OS uses paging, which means that pages are only inserted when written. And blocks of memory that aren't used can be discarded or written to file, when needed.

If you look in the task manager (Windows), under memory, you see that there are many different kinds of usage and that you generally have about twice as much memory committed (allocated) than is actually in use (paged in).

Thaddy

  • Hero Member
  • *****
  • Posts: 11916
Re: ZEOSLIB increasing memory usage inside TThread
« Reply #7 on: July 25, 2022, 01:01:26 pm »
The memory is free'd when you use SomeThread.waitfor in the main thread. Just before program ends.
Don't kill a thread to soon. That gets you into trouble.
Actually in your example it does not matter too much becase the OS will release the memory anyway.
If you use waitfor, the application ends without leaks.
(On a side note: I still not understand why people are using zeoslib, it has a sick/broken architecture and does not have any advantages above the core db components which have a very sound architecture. Asking for trouble. In my opinion it should be removed from the distribution to prevent users using it.)
« Last Edit: July 25, 2022, 01:08:51 pm by Thaddy »
Black themes should be banned.

rvk

  • Hero Member
  • *****
  • Posts: 4948
Re: ZEOSLIB increasing memory usage inside TThread
« Reply #8 on: July 25, 2022, 01:24:54 pm »
(On a side note: I still not understand why people are using zeoslib, it has a sick/broken architecture and does not have any advantages above the core db components which have a very sound architecture. Asking for trouble. In my opinion it should be removed from the distribution to prevent users using it.)
Is Zeos in the distibution?
I thought it was only downloadable in the OPM.
(But I would also prefer SQLdb over Zeos)

And if you want it off OPM, well, there might be more candidates for that  ;)

Thaddy

  • Hero Member
  • *****
  • Posts: 11916
Re: ZEOSLIB increasing memory usage inside TThread
« Reply #9 on: July 25, 2022, 04:00:06 pm »
Indeed, sorry, OPM. And your second point is also right: there are more packages that are remove/dissolve/disappear candidates. And some that - because of quality code but lack of maintenance - should be marked as legacy.
Black themes should be banned.

Thaddy

  • Hero Member
  • *****
  • Posts: 11916
Re: ZEOSLIB increasing memory usage inside TThread
« Reply #10 on: July 25, 2022, 07:25:02 pm »
You have some ways.
  • Stop using Tthread
  • Stop using Zeos
  • Stop using Mysql
  • Stop using Lazarus
If you find a person who caused memory leak and report it, then lots of users say you thanks!
The only valid entry is 2.
Black themes should be banned.

marsupilami79

  • New Member
  • *
  • Posts: 29
Re: ZEOSLIB increasing memory usage inside TThread
« Reply #11 on: July 26, 2022, 10:47:54 pm »
And some that - because of quality code but lack of maintenance - should be marked as legacy.
How come you are qualified to talk about Zeos maintenance or code quality? When did you even try to do someting there? What is your big contribution where to see how much better you are?
Zeos developer

marsupilami79

  • New Member
  • *
  • Posts: 29
Re: ZEOSLIB increasing memory usage inside TThread
« Reply #12 on: July 26, 2022, 10:52:23 pm »
Also... I'm not entirely sure what you are doing (using a TZConnection from the main thread inside a thread) is thread safe. It could be that each thread would need it's own  TZConnection. But maybe someone with more ZEOS experience can tell us that.
Zeos is thread safe as long as one TZConnection only gets used from one thread at a time. Multiple threads can each use their own TZConnection. We do have a user who does big operations in a background thread and connects GUI components to the data sets after fetching the data is finished. See https://zeoslib.sourceforge.io/viewtopic.php?f=40&t=83751&p=212402#p212086 for more information.
Zeos developer

 

TinyPortal © 2005-2018