I am going to assume for the moment you are using windows.Doh! Sorry, I broke rule number 1: State your OS, version information etc.
Also remember to move the Form.handle to a normal variable somewhere, do this in the main thread before starting the secondary thread. From the secondary thread you use this reference for the form.handle when using PostMessageI haven't done that before. I'm sure there is good reason to do so, but I don't know why. Could you briefly elaborate (or point me in the direction of more info)? Thanks.
This process almost works 100% perfectly, and I have no thread variable contentions of any sort, as the data always makes sense (as part of my evaluation procedure, I am testing the data in the buffer, as well as testing for buffer overruns). However I have discovered one caveat: WriteAsyncQueue executes the triggered event as a critical section, which means in effect my worker thread is momentarily frozen while the event is processed in the main loop. This means I lose up to about 15ms of valuable data, as the worker thread fails to pick up the data every (approximately) 1.2ms. The data in question is coming from an I2C Analogue to Digital converter.
With more investigating, I came across the TThread.Queue method (as found here: https://www.freepascal.org/docs-html/rtl/classes/tthread.queue.html )
This method is not process blocking like TThread.Synchronize, however that doesn't necessarily mean it isn't running a critical section internally, similar to WriteAsyncQueue, which would have the same negative result in my case. With that said, I would like to use it, however I have been completely unsuccessful in finding a working example online, of a worker thread that triggers an event in the main loop using TThread.Queue.
I am going to assume for the moment you are using windows.Instead of proposing non-portable code, I'd rather suggest RTLEvents.
... postMessage(MainThreadWindowHandle, ...
Thanks for confirming what I suspected (as far as TThread.Queue using a critical section in its method). I don't believe I will have too many difficulties getting my buffer to be thread safe once I have a lockless option up and running, as I have literally spent weeks carrying out many tests transferring variables of various types across the thread barrier and monitoring/mitigating variable contention. I wrote a good program to test this, thrashing 2 CPU's at 100% for hours at a time. Tested on both my Intel I7 based machine as well as the ARM based Raspberry Pi 3 board.
TThread.Queue works like Synchronize though you need to keep the data available until the Queue call returns. However for your purpose this wouldn't work as the queue mechanism (which is shared by Queue and Synchronize) as you suspected internally uses a critical section. What you need is a lockless threadsafe queue (https://www.schneems.com/2017/06/28/how-to-write-a-lock-free-queue/) (with a good implementation of such maybe I'd adjust the internals of TThread as well...)
Instead of proposing non-portable code, I'd rather suggest RTLEvents.Thanks Sash, being a big Linux fan, I definitely prefer to learn options that are ideally platform/OS agnostic. I have developed software before that relies heavily on Windows API's etc. and while they certainly hold some attraction, code that I can simply cross compile etc. with zero changes is by far the most appealing option. :D
WriteAsyncQueue executes the triggered event as a critical section, which means in effect my worker thread is momentarily frozen while the event is processed in the main loop. This means I lose up to about 15ms of valuable data, as the worker thread fails to pick up the data every (approximately) 1.2ms. The data in question is coming from an I2C Analogue to Digital converter.
I really do not see how could anything there block for 15ms.
On a normal OS, nothing is realtime, any thread (including your data collector) can be interrupted for any amount of time...Truer words were never spoken :)
A quick question on the circular buffer: Do you use an array for the buffer?A pre-allocated array is the most efficient way, particularly when the reading and writing of elements is time sensitive as in the case you've described. Basically, in a time sensitive process, you don't want time spent allocating new memory blocks.
I think this should be safe (as long as I am not reading/writing the same array element at the same time across threads of course) but would like to confirm, thanks.Correctly implemented it is safe. That means, as long as the variables "ReadPos" and "WritePos" are _always_ written by different threads then it should be fine. IOW, each variable is "owned" (written to) by one and only one thread. The one concern is, in a lockless implementation, the buffer must be large enough to ensure data is always read faster than it is written, to ensure there is always an "already read" slot in the array available for a new element. A lockless implementation is basically a race condition between reader and writer where the winner of the race is "guaranteed" (hopefully, the "reader" thread.)
Linux decides that servicing another process or routine is more important, and puts my application on pause.While you can not stop Linux from interrupting your code, you can make it behave in a deterministic way:
With regard to QueueAsyncCall I made this part of my main loop, called by the worker thread. This does not seem very safe to me (having the worker thread calling a main loop procedure), however it compiles and works just fine.QueueAsyncCall simply puts in a serialized thread safe way 2 pointers (1 for method and 1 for data) into a queue. Then just before time comes for GUI update (either by OS or forced with Application.ProcessMessages) your main application thread stops execution of your code and calls all methods from the queue. Nothing unsafe there. Only one thread has GUI access, so no problems at all.
I followed the instructions found here:Forum example is very similar because record based thread safe GUI update example in the wiki was added by me after that forum discussion. I gave you forum link because unlike wiki it shows some discussion on the topic. It doesn't matter where WriteAsyncQueue is located (although there might be some good and some bad practices). What matters is that WriteAsyncQueue gets called from the main thread at the right time to allow thread safe GUI access. TMemoChannel is not a form. It is a class that I have implemented for MultiLog and it was used to add thread safe logging to TMemo located in some user application form. You can find full implementation here: https://github.com/blikblum/multilog/blob/master/memochannel.pas. Thanks to that I have a 24/7 industrial application running for years now, having several threads logging into the same memo for visualization and into the same archive log file without issues.
https://wiki.lazarus.freepascal.org/Asynchronous_Calls
From the example (very similar to the example in your earlier link too) it shows both QueueAsyncCall and WriteAsyncQueue to be part of the same class or structure (presumably TMemoChannel is the main Form of the application). Am I wrong? as WriteAsyncQueue is the triggered event, and contains visual components that you would expect in the main loop/GUI it seems sensible that these are main loop procedures. Should they both instead be part of my worker thread?