Need help with Indy IdTCPServer

[Thaddy]

Yes, it is a great help in this particular case but also in general.

[TheMouseAUS]

Thank for all your input, I am trying to figure out how I share my code, its has a lot happening in it, normal running is around 45-55 threads. I will try to outline it to start with :-

Threads are :-
Main thread - runs a couple or routines and syncs from PLC threads for any error logs. This also updates a string holding current date and time used in all logs.
Threads that communicate to industrial controllers (PLC's) using libopen62541 (OPC)
Threads that communicate to industrial controllers (PLC's) using libplctag (allen bradley)
Indy TCP threads for clients.

The PLC threads simply write data from the controller to variables in a TRecord. Once initialized the variables dont change except there value.

The clients can request any log data  (held in TStringLists) or the variables stored in TRecords.

The system log which holds and connection issues generally seems to be the one I had most issues with, now I have written that side must better so that should be fine moving forwards.

The interesting thing the crash I caught the other night pointed to the  current date and time string as it was being written to, the TCP Exception was trying to read it. I didnt think this would do it but once I put it into a critical section, it seems to have solved it.

I do have a question about Critical sections, many of the string lists are being read from and only written to occasionally, is putting these in a critical section the best as I want clients to be able to read them all at once unless it is being written to as I dont want reads to lock the section.


My app is 6000 lines and TBH i am a little embarrassed sharing as I am no expert (self taught, I read as much as I can) but if you want me to I can.

[LeP]

Like I wrote, writing FROM 1 "point" at time (this means that only one THREAD can write the shared resource and NO CONCURRENCES IN WRITING SHOULD BE DONE !!!!) and reading it is not an issue but if you are reading a structure like a record, you must syncronize of course that read with write.

If you are writing 30 words (for example) on you record and you read that concurrently how you can be sure that you are reading all words updated ?

You must use something to lock the read until write is done. That is the TCriticalSection perfect use (there is not only that, other thecs can be used of course).

With PLC is the same: except for OPC (but this depends on implementations too) when you send 30 words how you can be sure that ALL 30 words are wrote when the PLC is cycling on those ? It's possbile that in one cycle scan only 20 word were write and other 10 will be written on the next cycle scan (it's an example).

Not all protocols write "ALL in ONCE" on the PLC memory, until this is described in the documentation.

[Thaddy]

Have you looked at https://www.freepascal.org/docs-html/rtl/sysutils/tmultireadexclusivewritesynchronizer.html
That is actually what @LeP suggests. It is very simple to use and much better (and faster) than a critical section.
Despite its frighteningly long name it is relatively lightweight.
(I always declare an alias for it, otherwise I go even more crazy than I already am:)

Code: Pascal  [Select][+][-]

1. type
2.   TMrsw = TMultiReadExclusiveWriteSynchronizer;

[TheMouseAUS]

Yes I did originally use this, until I came across a thread on a forum that said it was slow, buggy and on linux just just a wrapper for TCriticalSection.

If this is not the case I will got back to it as it wont hard to do.

Have you looked at https://www.freepascal.org/docs-html/rtl/sysutils/tmultireadexclusivewritesynchronizer.html
That is actually what @LeP suggests. It is very simple to use and much better (and faster) than a critical section.
Despite its frighteningly long name it is relatively lightweight.
(I always declare an alias for it, otherwise I go even more crazy than I already am:)

Code: Pascal  [Select][+][-]

1. type
2.   TMrsw = TMultiReadExclusiveWriteSynchronizer;

[Thaddy]

Who wrote that? It is not true, although a cs is used for only the write operation.
It is not slow for reads and that is for you the most important thing, much faster than a cs.
And Windows or Linux does not make any difference.
Especially on Linux which has a much, much faster threading model than Windows in the first place you can ignore it.

Since Freepascal comes with all sourcecode it is easy to verify that claim is wrong.
I knew that, but I just looked again: it is a wrong statement.

[TheMouseAUS]

Like I wrote, writing FROM 1 "point" at time (this means that only one THREAD can write the shared resource and NO CONCURRENCES IN WRITING SHOULD BE DONE !!!!) and reading it is not an issue but if you are reading a structure like a record, you must syncronize of course that read with write.

If you are writing 30 words (for example) on you record and you read that concurrently how you can be sure that you are reading all words updated ?

You must use something to lock the read until write is done. That is the TCriticalSection perfect use (there is not only that, other thecs can be used of course).

With PLC is the same: except for OPC (but this depends on implementations too) when you send 30 words how you can be sure that ALL 30 words are wrote when the PLC is cycling on those ? It's possbile that in one cycle scan only 20 word were write and other 10 will be written on the next cycle scan (it's an example).

Not all protocols write "ALL in ONCE" on the PLC memory, until this is described in the documentation.

The clients are updating once a second, if the variable being read is not up-to-date it really wont hurt, for example if its a temperature it will only be a maximum of one sec behind. All of the PLC data/tag handling has been working extremely reliable for the past three years with out any issues. My problems only started two weeks ago when i switched the clients to TCP from UDP. This was done for many reasons (sending data as a memory stream for one), order of receiving packets (udp sometimes screwed it up).

I have now wrapped all writes in critical sections so no concurrent writes can happen, It has stabilized a lot now from where I started two weeks ago. It does mean my theory was wrong though as I thought reading the date-time string caused the crash as the main thread was only thread the ever writes to it.

[TheMouseAUS]

That will teach me for believing what i read without fact checking.

So to make sure I understand, lets say I have a TStringList, each thread always reads it every second, only when writing to it should be wrapped in a cs.
The other threads trying to read it while a write takes place wont be an issue?

Who wrote that? It is not true, although a cs is used for only the write operation.
It is not slow for reads and that is for you the most important thing, much faster than a cs.
And Windows or Linux does not make any difference.
Especially on Linux which has a much, much faster threading model than Windows in the first place you can ignore it.

Since Freepascal comes with all sourcecode it is easy to verify that claim is wrong.
I knew that, but I just looked again: it is a wrong statement.

[LeP]

The clients are updating once a second, if the variable being read is not up-to-date it really wont hurt, for example if its a temperature it will only be a maximum of one sec behind. All of the PLC data/tag handling has been working extremely reliable for the past three years with out any issues.

One seconds is very slow, and "tag" normally means simple data, I don't expect issues ... it would probably be different if you read or wrote an entire DB from Siemens.

All of the PLC data/tag handling has been working extremely reliable for the past three years with out any issues. My problems only started two weeks ago when i switched the clients to TCP from UDP. This was done for many reasons (sending data as a memory stream for one), order of receiving packets (udp sometimes screwed it up).

You had issues with UDP, so not all athings worked on past.

I have now wrapped all writes in critical sections so no concurrent writes can happen, It has stabilized a lot now from where I started two weeks ago. It does mean my theory was wrong though as I thought reading the date-time string caused the crash as the main thread was only thread the ever writes to it.

Concurrent writes can occurs ONLY using different CONTEXT ... I mean that you cannot use the global IOHandler of IdTCPServer to write but you MUST use the IOHandler of the CONTEXT (that is related to a single "connection").

An you MUST NOT use the IoHandler of specific CONTEXT to write concurrently from more "points" of your server application.

This means that you can write concurrently form multiple connections, but not more than one for every connection.

[TheMouseAUS]

You had issues with UDP, so not all athings worked on past.

No I never had issues with UDP at all or the server as a whole until the switch

[TheMouseAUS]

I have cleaned up my critical sections. I have one last issue, now only there when SSL is enabled (Tried with both TauruaTLS and IndySec both crash out). I cannot capture an exception so cannot handle the error.

The test server runs command line (in this instance using IndySec) , test client(uses TaurusTLS due to IndySec timeout issue)

FPC 3.2.2
x86_64-linux-qt6

[LeP]

Try that, with multiple clients too.

Hope this help you

RUN WITHOUT IDE TO TEST (of course after first run or compilation)!!!

EDIT: I MODIFY THE SOURCES TO WORKS WITH WINDOWS .... REINSERT THE UNIT USED WITh LINUX THAT I HAVE REMOVED.

[TheMouseAUS]

Thanks for helping me improve, the issue still exists server side though.

Code: Pascal  [Select][+][-]

1.       IdTCPClient1.IOHandler.WriteLn('Hello');
2.       idTCPClient1.Disconnect; // this will crash the server if ssl enabled;
3.  

Doing this in the client still crashes the server, even with the mods you made to the server code.
The fact its that easy to crash the server I find concerning. I know in real life you would never do this but if you wanted to you could if your broke into network, setup an app to do this and the server will never run! I realize in this situation you will probably have bigger problems but still its a worry. Running it in debug shows an error in IdSecOpenSSLHeaders_ssl

Code: Pascal  [Select][+][-]

1.  
2. function Load_SSL_accept(ssl: PSSL): TOpenSSL_C_INT; cdecl;
3. begin
4.   // this breaks
5.   SSL_accept := LoadLibSSLFunction('SSL_accept');
6.   if not assigned(SSL_accept) then
7.     EOpenSSLAPIFunctionNotPresent.RaiseException('SSL_accept');
8.   Result := SSL_accept(ssl);
9. end;
10.  
11.  

I dont know if its right or not, as logging shows nothing the server app just dies!

Try that, with multiple clients too.

Hope this help you

RUN WITHOUT IDE TO TEST (of course after first run or compilation)!!!

EDIT: I MODIFY THE SOURCES TO WORKS WITH WINDOWS .... REINSERT THE UNIT USED WITh LINUX THAT I HAVE REMOVED.

[LeP]

Thanks for helping me improve, the issue still exists server side though.

Code: Pascal  [Select][+][-]

1.       IdTCPClient1.IOHandler.WriteLn('Hello');
2.       idTCPClient1.Disconnect; // this will crash the server if ssl enabled;
3.  

Doing this in the client still crashes the server, even with the mods you made to the server code.

For me not.

Are you sure you implemented like I attached ? ALL ?

[TheMouseAUS]

I complied your server code just changed a couple of uses for linux, the client can still crash it.

This picked up no exception. just instant crash

Code: Pascal  [Select][+][-]

1. procedure TTestServer.IdTCPServer1Exception(AContext: TIdContext;
2.   AException: Exception);
3. begin
4.   WriteLn(AException.Message);
5.   AContext.Connection.Disconnect;
6. end;
7.