Recent

Author Topic: Windows Threads Total Limit  (Read 9657 times)

Nick N.

  • New member
  • *
  • Posts: 9
Windows Threads Total Limit
« on: July 23, 2021, 10:41:38 pm »
Hi all

I have A simple test, creating and terminating lot og threads. On my PC this test working without errors. On several clints PC code fails after 100K - 500K cycles.

Thread creation error: Not enough memory resources are available to process this command.

Code:

Code: Pascal  [Select][+][-]
  1. procedure Empty;
  2. begin
  3.   //
  4. end;
  5.  
  6. procedure TForm1.Button1Click(Sender: TObject);
  7. Var s : string;
  8.     tc, i, j, no : integer;
  9.     thread : TThread;
  10.     list : TList;
  11. begin
  12.  
  13.  
  14.   no := 0;
  15.   List := TList.Create;
  16.  
  17.   s := '1000000';
  18.   if not InputQuery('Enter number of queries', '', s) then exit;
  19.   tc := StrToInt(s);
  20.  
  21.   for i := 1 to tc div 500 do
  22.   begin
  23.  
  24.     Caption := 'STRESS TEST: ' + inttostr(no);
  25.     Application.ProcessMessages;
  26.  
  27.     List.Clear;
  28.  
  29.     for j := 1 to 500 do
  30.     begin
  31.       inc(no);
  32.  
  33.       Thread := TThread.CreateAnonymousThread(@empty);
  34.  
  35.       thread.FreeOnTerminate := false;
  36.       thread.Start;
  37.       List.Add(thread)
  38.     end;
  39.  
  40.     while list.Count > 0 do
  41.     begin
  42.      TThread(List[0]).WaitFor;
  43.      TThread(List[0]).Free;
  44.      List.Delete(0);
  45.     end;
  46.  
  47.   end;
  48.  
  49. end;
  50.  
  51.  

Does Windows have a maximum limit on the total number of threads created? Can I disable it ?

Tnx

skalogryz

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2770
    • havefunsoft.com
Re: Windows Threads Total Limit
« Reply #1 on: July 23, 2021, 11:02:13 pm »
Thread creation error: Not enough memory resources are available to process this command.
are you trying 32-bit or 64-bit? something to read

Nick N.

  • New member
  • *
  • Posts: 9
Re: Windows Threads Total Limit
« Reply #2 on: July 24, 2021, 07:35:08 am »
Thread creation error: Not enough memory resources are available to process this command.
are you trying 32-bit or 64-bit? something to read

I use 32bit. Yes, I read this article too.

I still can't figure out why the error occurs on some PC. In my case, threads are created and destroyed, there can be no more than 500 threads at the same time. Everything works for me locally without failures, on the test machine after a while the error of creating new threads starts to appear

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 1090
  • Professional amateur ;-P
Re: Windows Threads Total Limit
« Reply #3 on: July 24, 2021, 04:42:00 pm »
Hey Nick,

Thread creation error: Not enough memory resources are available to process this command.

On the risk of seeming pedantic, and my apologies if you read it like that, it's not my intention, but the error message mentions memory.

Can you please provide us with the RAM amount for both the test machine and the client machine?

If both have the same amount, then please ignore my post.
But if the amount differs by a big amount, well, that could be your problem.

Also, are both machines running the same version of Windows? And I mean the same, cuz Windows Home and Windows Pro have some differences that can affect memory handling., I think...

Cheers,
Gus
Lazarus 3.99(main) FPC 3.3.1(main) Ubuntu 23.10 64b Dark Theme
Lazarus 3.0.0(stable) FPC 3.2.2(stable) Ubuntu 23.10 64b Dark Theme
http://github.com/gcarreno

Nick N.

  • New member
  • *
  • Posts: 9
Re: Windows Threads Total Limit
« Reply #4 on: July 24, 2021, 06:51:25 pm »
Can you please provide us with the RAM amount for both the test machine and the client machine?

I have Windows 10 Pro, 16 GB ram. Test PC is virtual, windows server 2019, 8GB ram.

I tested the code on different computers, somewhere it works, somewhere it doesn't. There are enough system resources, 100 threads are 1GB of system memory, there is enough of this everywhere.

This code also not working in Delphi XE3 and 10.4.2 (latest)

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11351
  • FPC developer.
Re: Windows Threads Total Limit
« Reply #5 on: July 24, 2021, 07:16:07 pm »
I tested the code on different computers, somewhere it works, somewhere it doesn't. There are enough system resources, 100 threads are 1GB of system memory, there is enough of this everywhere.

So 500 threads are 5GB of memory. So how much memory can a 32-bit app use (depending on windows edition and tuning, 2,3 or 4 GB, but certainly not more)

440bx

  • Hero Member
  • *****
  • Posts: 3921
Re: Windows Threads Total Limit
« Reply #6 on: July 24, 2021, 09:09:38 pm »
I tested the code on different computers, somewhere it works, somewhere it doesn't.
and that is normal.  the address space layout will not be identical in all versions of Windows even for the same program.  ASLR alone is going to make them vary not to mention the configuration of the Windows installation.

There are enough system resources, 100 threads are 1GB of system memory, there is enough of this everywhere.
It isn't just about how much memory is available.  Memory isn't the only resource consumed by threads. 

Use something like Process Hacker and/or Process Explorer to track the consumption of various system resources as more threads are created.  Doing that will likely point you in the right direction.  Simply concluding that memory is plentiful, therefore it should work, is much too simplistic.

The really important question is, do you really need hundreds of threads to solve the problem (whatever it may be) ?  if you do, a PC class machine isn't the ideal machine to solve the problem (understatement of the day.)

HTH.
(FPC v3.0.4 and Lazarus 1.8.2) or (FPC v3.2.2 and Lazarus v3.2) on Windows 7 SP1 64bit.

Nick N.

  • New member
  • *
  • Posts: 9
Re: Windows Threads Total Limit
« Reply #7 on: July 24, 2021, 09:43:57 pm »
So 500 threads are 5GB of memory. So how much memory can a 32-bit app use (depending on windows edition and tuning, 2,3 or 4 GB, but certainly not more)

I was wrong. 1 Thread is 1MB, 100 is 100MB, 1000 is 1GB

The really important question is, do you really need hundreds of threads to solve the problem (whatever it may be) ?  if you do, a PC class machine isn't the ideal machine to solve the problem (understatement of the day.)

This test is the localization of the problem. The multithreading app stops working after a certain amount of time, may be 1 day, may be week.  App connecting to database and making queries by requests of hardware stations. No heavy load, normal number of threads, randomly crashed on several pc

Final test on Delphi/lazarus:

Code: Pascal  [Select][+][-]
  1. program ttDelphi;
  2.  
  3. {$APPTYPE CONSOLE}
  4.  
  5. {$R *.res}
  6.  
  7. uses
  8.   System.SysUtils, Classes;
  9.  
  10.   Var part, j, no : integer;
  11.       list : array of TThread;
  12.       Thread : TThread;
  13.  
  14. begin
  15.  
  16.   try
  17.  
  18.     part := 1000;
  19.     SetLength(list, part);
  20.  
  21.     no := 0;
  22.  
  23.     while true do
  24.     begin
  25.  
  26.       for j := 1 to part do
  27.       begin
  28.         Thread := TThread.CreateAnonymousThread(
  29.         procedure
  30.         begin
  31.           sleep(100);
  32.         end
  33.         );
  34.  
  35.         inc(no);
  36.  
  37.         Thread.FreeOnTerminate := false;
  38.         Thread.Start;
  39.         list[j - 1] := Thread;
  40.  
  41.       end;
  42.  
  43.       WriteLn( Format('Created %d threads', [no]) );
  44.  
  45.       for j := 1 to part do
  46.       begin
  47.         List[j - 1].WaitFor;
  48.         List[j - 1].Free;
  49.       end;
  50.  
  51.     end;
  52.  
  53.   except
  54.     on E: Exception do
  55.     begin
  56.       Writeln(E.ClassName, ': ', E.Message);
  57.       Readln;
  58.     end;
  59.   end;
  60.  
  61. end.
  62.  

Test in Visual Studio 2019 (also crashed) :
Code: Pascal  [Select][+][-]
  1.  
  2. #include <iostream>
  3. #include <stdio.h>
  4. #include <windows.h>
  5. #include <conio.h>
  6. #include <system_error>
  7.  
  8. DWORD CALLBACK ThreadProc(void*)
  9. {
  10.         Sleep(100);
  11.         return 0;
  12. }
  13.  
  14. int __cdecl main(int argc, const char* argv[])
  15. {
  16.  
  17.         int count = 0;
  18.         while (true) {
  19.  
  20.                 HANDLE hh[1000];
  21.  
  22.                 for (int j = 0; j < 1000; j++)
  23.                 {
  24.                         DWORD id;
  25.                         hh[j] = CreateThread(NULL, 0, ThreadProc, NULL, 0, &id);
  26.                         if (hh[j] == NULL) {
  27.                                 printf("%s", std::system_category().message(GetLastError()));
  28.                                 getchar();
  29.                                 return 0;
  30.                         }
  31.                         count++;
  32.                 }
  33.  
  34.                 printf("Created %d threads\n", count);
  35.  
  36.                 for (int j = 0; j < 1000; j++)
  37.                 {
  38.                         WaitForSingleObject(hh[j], INFINITE);
  39.                         CloseHandle(hh[j]);
  40.                 }
  41.  
  42.         }
  43.  
  44. }
  45.  
  46.  

I think this is a system problem, the primitive code should work everywhere. May be related to antivirus.

skalogryz

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2770
    • havefunsoft.com
Re: Windows Threads Total Limit
« Reply #8 on: July 25, 2021, 02:42:32 am »
I was wrong. 1 Thread is 1MB, 100 is 100MB, 1000 is 1GB
Why would 1 thread be equal to 1Mb?
I thought the default stack size of a thread is 4Mb.
So we're looking into 500 threads for 32-bit (2Gb max)  machine.

(TThread class has a StackSize parameter. You might want to play with it in order to see how far you can go)

This test is the localization of the problem...
The multithreading app stops working after a certain amount of time, may be 1 day, may be week.  App connecting to database and making queries by requests of hardware stations. No heavy load, normal number of threads, randomly crashed on several pc
Why would you suspect the number of threads then? If the number of threads used in the app are "normal"?
You might need to add extra logs entries into the app.

I still can't figure out why the error occurs on some PC. In my case, threads are created and destroyed, there can be no more than 500 threads at the same time. Everything works for me locally without failures, on the test machine after a while the error of creating new threads starts to appear
It is possible that your experiencing the different context-switching time.
The more threads you have the more time is wasted to switch between the different threads.
I'm assuming your machine is fast enough to create and dispose the necessary amount.
The test machine might be lagging on executing the launched threads at some point.

I think if you want to text the maximum number of threads you need to modify your test.
Instead of "Sleep(100)" all every launched thread should wait for some event to be signaled.
In this case you're not risking the context switching lag.

Once your main thread cannot spawn new threads anymore you simply singal the even and let all the threads to finish. In this case, I'm expecting the "max number" of threads to be identical on your and the test machine.

Code: Pascal  [Select][+][-]
  1. program project1;
  2. {$ifndef fpc}{$APPTYPE CONSOLE}{$endif}
  3.  
  4. uses
  5.   SysUtils, Classes, syncobjs;
  6.  
  7. type
  8.   TTestThread = class(TThread)
  9.     procedure  Execute; override;
  10.   end;
  11.  
  12. var
  13.   showStopper : TEvent;
  14.  
  15. procedure TTestThread.Execute;
  16. begin
  17.   showStopper.WaitFor(INFINITE);
  18. end;
  19.  
  20.  
  21. procedure RunTest(count: integer; stackSize: integer);
  22. var
  23.   list : TList;
  24.   Thread : TThread;
  25.   j, no: integer;
  26. begin
  27.   list := TList.Create;
  28.   showStopper := TEvent.Create(nil, true, false,'');
  29.   try
  30.     no := 0;
  31.     try
  32.       for j := 1 to count do
  33.       begin
  34.         Thread := TTestThread.Create(true, stackSize);
  35.         list.Add(thread);
  36.         inc(no);
  37.         Thread.FreeOnTerminate := false;
  38.         Thread.Start;
  39.         if (j mod 10 = 0) then writeln(j,' threads created')
  40.       end;
  41.     except
  42.       on x: exception do
  43.         writeln('failure while creating a thread: ', j,' ',x.message);
  44.     end;
  45.   finally
  46.     writeln('cleaning up');
  47.     showStopper.SetEvent;
  48.     for j := 0 to list.Count-1 do begin
  49.       TThread(List[j]).WaitFor;
  50.       TThread(List[j]).Free;
  51.     end;
  52.     list.Free;
  53.     showStopper.Free;
  54.     writeln('clean is done');
  55.   end;
  56. end;
  57.  
  58.  
  59. var
  60.   part     : integer;
  61.   stackSize : integer;
  62.   st : double;
  63. begin
  64.   try
  65.     part := 1000;
  66.     stackSize := DefaultStackSize;
  67.     if ParamCount>0 then part := StrToIntDef(ParamStr(1), 0);
  68.     if ParamCount>1 then begin
  69.        st := StrToFloatDef(ParamStr(2), -1);
  70.        if st>0 then stackSize := Round(1024*1024*st)
  71.        else stackSize := DefaultStackSize;
  72.     end;
  73.  
  74.     writeln('Threads   = ', part);
  75.     writeln('stackSize = ', stackSize/1024/1024:0:2,'Mb');
  76.     RunTest(part, stackSize);
  77.   except
  78.     on E: Exception do
  79.     begin
  80.       Writeln(E.ClassName, ': ', E.Message);
  81.       Readln;
  82.     end;
  83.   end;
  84.  
  85. end.

If you run this app like this:
Code: [Select]
test 1000 1
(spawning 1000 threads with 1Mb stack size) all of them should be created.
« Last Edit: July 25, 2021, 03:04:16 am by skalogryz »

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1311
    • Lebeau Software
Re: Windows Threads Total Limit
« Reply #9 on: July 25, 2021, 11:59:54 pm »
Why would 1 thread be equal to 1Mb?
I thought the default stack size of a thread is 4Mb.

The default size is whatever the program's PE header specifies, which can be set via the {$MINSTACKSIZE} and {$MAXSTACKSIZE} directives.
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

skalogryz

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2770
    • havefunsoft.com
Re: Windows Threads Total Limit
« Reply #10 on: July 26, 2021, 12:59:25 am »
The default size is whatever the program's PE header specifies, which can be set via the {$MINSTACKSIZE} and {$MAXSTACKSIZE} directives.
The setting only applies to the main thread (as the thread is launched by the system).
The stack size of a thread launched by the application is defined in run-time.
WinAPI CreateThread has dwStackSize parameter.

FPC TThread.Create has a parameter to the stack size which is defaulted to DefaultStackSize.
The value of DefaultStackSize doesn't depends on {$MAXSTACKSIZE}

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1311
    • Lebeau Software
Re: Windows Threads Total Limit
« Reply #11 on: July 26, 2021, 06:34:06 pm »
The setting only applies to the main thread (as the thread is launched by the system).

The setting applies as the default for ANY thread where a stack size is not specified explicitly.

The stack size of a thread launched by the application is defined in run-time.

It CAN be specified at runtime, but if it is NOT specified then the default specified in the PE header is used instead.

WinAPI CreateThread has dwStackSize parameter.

Yes, and here is what the documentation says about it:

CreateThread

Quote
dwStackSize

The initial size of the stack, in bytes. The system rounds this value to the nearest page. If this parameter is zero, the new thread uses the default size for the executable. For more information, see Thread Stack Size.

Thread Stack Size

Quote
The default size for the reserved and initially committed stack memory is specified in the executable file header. Thread or fiber creation fails if there is not enough memory to reserve or commit the number of bytes requested. The default stack reservation size used by the linker is 1 MB. To specify a different default stack reservation size for all threads and fibers, use the STACKSIZE statement in the module definition (.def) file. The operating system rounds up the specified size to the nearest multiple of the system's allocation granularity (typically 64 KB). To retrieve the allocation granularity of the current system, use the GetSystemInfo function.

To change the initially committed stack space, use the dwStackSize parameter of the CreateThread, CreateRemoteThread, or CreateFiber function. This value is rounded up to the nearest page. Generally, the reserve size is the default reserve size specified in the executable header. However, if the initially committed size specified by dwStackSize is larger than or equal to the default reserve size, the reserve size is this new commit size rounded up to the nearest multiple of 1 MB.

To change the reserved stack size, set the dwCreationFlags parameter of CreateThread or CreateRemoteThread to STACK_SIZE_PARAM_IS_A_RESERVATION and use the dwStackSize parameter. In this case, the initially committed size is the default size specified in the executable header. For fibers, use the dwStackReserveSize parameter of CreateFiberEx. The committed size is specified in the dwStackCommitSize parameter.

So, the size in the PE header can apply to ANY thread, not just the main thread, depending on the caller's desire when creating a new thread.

FPC TThread.Create has a parameter to the stack size which is defaulted to DefaultStackSize.
The value of DefaultStackSize doesn't depends on {$MAXSTACKSIZE}

Sure, and according to FPC's documentation, DefaultStackSize is 4MB by default.  On the other hand, nothing stops a user from creating a TThread object with StackSize=0 to use the default from the PE header.  Or even creating a thread without going through TThread at all (maybe by calling CreateThread() directly, or maybe by using a 3rd party library/DLL or a system API that creates its own threads, etc).
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

skalogryz

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 2770
    • havefunsoft.com
Re: Windows Threads Total Limit
« Reply #12 on: July 26, 2021, 06:38:42 pm »
The setting applies as the default for ANY thread where a stack size is not specified explicitly.
...
It CAN be specified at runtime, but if it is NOT specified then the default specified in the PE header is used instead.

... On the other hand, nothing stops a user from creating a TThread object with StackSize=0 to use the default from the PE header
TThread always specifies it explicitly (from WinAPI prespective).
Unless the code using TThread explicitly specifies NOT to use it (by passing stack size as zero).

I did posted a sample above. You can recompile it with a different application stack size, yet the Maximum number of threads will remain the same, unless the thread stack size is modified.

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1311
    • Lebeau Software
Re: Windows Threads Total Limit
« Reply #13 on: July 26, 2021, 09:10:15 pm »
TThread always specifies it explicitly (from WinAPI prespective).

By default yes, since its StackSize constructor parameter is not 0 by default.  But the user COULD set DefaultStackSize=0, or simply call TThread.Create(..., 0).  I checked the RTL's source code, the StackSize parameter gets passed as-is to CreateThread(), there is no default size used if 0 is passed in.

Unless the code using TThread explicitly specifies NOT to use it (by passing stack size as zero).

Exactly.  Or, like I said, by simply not using TThread to begin with.
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

 

TinyPortal © 2005-2018