Utilising Multiple Processors with Single Jobs from a Large Batch

[molly]

works great!!

Good 

BTW:

Line 93 and similarily 101 :

wouldn't compile as they were. From what I can gather, the second property passed to QueueAsyncCall has to be a pointer to an Integer (ptrInt). So I changed it to the following which compiled:

The change you made there is ok, but please be warned that it isn't a pointer to an integer. See documentation:

Ptrint is a signed integer type which has always the same size as a pointer. Ptrint is considered harmfull and should almost never be used in actual code, because pointers are normally unsigned.

[Gizmo]

Good "pointer" (ha ha). I have changed to PtrUInt instead. Thanks Molly.

The introduction of the ptrint type was a mistake. Please use ptruint instead (https://www.freepascal.org/docs-html/rtl/system/ptruint.html).

[Gizmo]

One snag...I seem to be getting "Thread creation error: Not enough storage available to process this command" when pointing the demo project at any folder containing more than a few dozen files.

I have Googled that, and it seems to point towards the Stack Size or the 'IRPStackSize' registry value (https://support.microsoft.com/en-us/help/106167/error-message-not-enough-server-storage-is-available-to-process-this-command), but perhaps more significantly the 'DefaultStackSize' value of the Create constructor for TThread, which seems to be 4 x 1024 x 1024 which is 4 billion (4Gb I think). So I'm guessing a 32-bit limit of some sort is being reached very quickly in this example?

Fixes for this are not overly clear though. The nearest relevant suggestion I found was here (http://forum.lazarus.freepascal.org/index.php?topic=26441.0) where the OP talks of :

...finding a bug in my code that was limiting the number of threads to ~244. solved. I could reach about 22000 threads in my cache.

except, unhelpfully, he doesn't go on to explain what the code was he was using and what he changed it to .

What I'd like is some reasonable way of limiting the program to X number of threads so that the executable, when used on anyones computer, will say "Ah, this user has X CPU's and Y cores, so I will launch Z threads" : Z being say 50% of Y. I've worked out how to list the processor count (with if TThread.IsSingleProcessor then and TThread.ProcessorCount) but I have no idea how to then decide on suitable stack sizes given the count of processors. 

[marcov]

4 x 1024 x 1024 = 4 MB, not 4 GB

[taazz]

What I'd like is some reasonable way of limiting the program to X number of threads so that the executable, when used on anyones computer, will say "Ah, this user has X CPU's and Y cores, so I will launch Z threads" : Z being say 50% of Y. I've worked out how to list the processor count (with if TThread.IsSingleProcessor then and TThread.ProcessorCount) but I have no idea how to then decide on suitable stack sizes given the count of processors.

you might have better lack if you search for thread pool instead of thread manager.

[Gizmo]

Guys

I'm hoping one of you can help me with this perculiar Thread Creation error and lack of free storage error? I have Googled it for several days, found one partial mention on this forum (as linked above) but all the suggestions seem to relate to changing computer settings and registry entries, which can't be right. I've also done some further code refinements over the last few evenings, as Taazz advised, by looking for memory leaks using HeapTrace. I noticed from that, that I wasn't freeing the file stream handle (fs.free), for one thing. I also noticed I was still calling the filename string from outside the Execute method. I also noticed that my "big file" loop was updating Form1 directly, still. So I've removed that for now. And I'm using  Dispose(PInfoData(Data)); instead of Free(Data). And I have specifically set the stack size in the project options to 100000, as I read that was one solution on the Lazarus bug tracker.

And yet, I still get the error.

I was hoping one of you might download the demo project, and see if you get the same error by selecting a folder with quite a lot of files in it, and seeing if you get it as well? You might not the first time round, or the second, but on my PC, I usually get it whenever I point it at a file with more than about 50 files in it. If you do get the error, are you able to see where the problem might be, and if you are, can you explain to me how you find it for future reference, because using the debugger is not helping me and I don't know how to use the stacktrace thingy? The error is thrown by

Code: [Select]

constructor TFileHashingWorkerThread.Create(aFilename : string;const FinishEvent:TDataEvent);
begin
    inherited Create(False);   <------- This is what throws the error
...

The full project is zipped here : https://my.pcloud.com/publink/show?code=XZvaiaZz8YjMOUg8fXaiGTorzs9BpExWCXV created with latest version of Lazarus and FPC.

In order to compile it, you'll need to use the Package Manager to add the HashLib4Pascal library (lpk package file comes with it), which is included in the folder, under Packages\FPC.

Any help appreciated. I just can't seem to debug what is causing the error.
 [/code]

[balazsszekely]

Hi Gizmo,

Please read this: https://blogs.technet.microsoft.com/markrussinovich/2009/07/05/pushing-the-limits-of-windows-processes-and-threads/

[Gizmo]

GetMem

Great article! Makes lots of sense and has helped with a lot of understanding. I was getting near the end and wondering "Ok, but how do I actually solve this issue" and there doesn't appear to be an obvious answer there. One statement the writer does make is

For instance, the general goal for a scalable application is to keep the number of threads running equal to the number of CPUs

which re-itterates one of my earlier replies. My program can lookup the number of CPU's (it does it with a FormCreate event), and so if there are, say 4 CPU's reported, it would be helpful to limit the number of threads being created to just 4. My program has existsed since 2011 with no multi-threading at all, so although 4 threads seems horribly less than the several thousand that could be incorporated, this feature request improvement to my program is just that - a feature request. It's not something that I need to do to make me rich or anything. So if I can get my program to just use 4 threads wherever the count of CPU's is 4 or more, that would be enough I think? Or do you guys disagree? I don't necessarily want to make it so that it throttles the backside out of the users PC. I just want to make it better than it is, which is basically a single process utility at the moment. 

So I assume there's a way to say "If TThread.ActiveThreadCount = TThread.ProcessorCount then wait for thread availability"? Though I notice here (https://www.freepascal.org/docs-html/rtl/classes/tthread.html) that there is no obvious way to ask about the number of running threads?

[balazsszekely]

Hashing is a CPU intensive process. Creating more then a few threads it won't help speeding up your application, since you're limited by the number of available CPUs. More over, beyond 8-10 threads(case of 4 processors) it will definitely kill performance and slow down your program. However switching to a threaded solution it has it's benefits, like a more responsive GUI.