Need help with Indy IdTCPServer

[TheMouseAUS]

I have a server that is using idTCPServer. Documentation on Indy is pretty much non existent anymore and there seems no alternative. I  was using UDP but the limitations of UDP make it not attractive in my use case.

The issue I have is the server app stability is shot since changing to TCP and I have no answers/ideas how to improve it. IdTCPServer seems very particular in its operation/connections. I have used TCriticalSections on anything that has multiple clients accessing and that side seems ok, the biggest issue I have is around the connection itself. The clients are using IdTCPClient. If they crash its no big deal but I cant have the server side crashes, its not good. The server has a real issue with the client dropping the connection. When the IdTCPOnExecute event is fired, I read the clients message request, process the request and write the reply back to client, but if that connection has dropped for whatever reason it breaks the server. As a test I got the client to send a request and instantly disconnect, the server crashed immediately. Can any one shed any light on how to handle the exception and stabilize the server side?

Thanks for reading :-)

[LeP]

How you tried with TaurusTLS as IoHnadleer ?
Have you assigned the Exception Event of Indy TCPServer ?

The server should not crash. I use it in Delphi in situations worse than yours and it never crashed.
I never saw it crashed, and I use TCPServer in industrial environment, with tens of thousands transaction per seconds per single connection. Never ending comunications, years without stop the application.

Can you post a minimal example of client / server, so I can try it with Lazarus ?

[Thaddy]

What do you mean "No alternative"?
fcl-net/web, synapse, just to name two.
I consider Indy legacy and ported everything that does not need Delphi compatibility to the standard fcl packages.

Anyway: You should handle the EIdConnClosedGracefully  exception if the client closed the connection itself.
If a connection is lost because of other things: The server receives something like a EidConnReset (not sure about the name here).
The first exception should be handled automatically in Indy servers, so which exception do you get?

Anyway: your session management seems to lack. Usually Indy is very stable.

[Remy Lebeau]

I'm the maintainer of Indy...

The issue I have is the server app stability is shot since changing to TCP and I have no answers/ideas how to improve it.

TIdTCPServer is a very mature and stable component. So any instability is more likely due to issues in your code. But, you didn't provide any details about your code, your server configuration, your environment, etc.

I have used TCriticalSections on anything that has multiple clients accessing and that side seems ok

That depends on what "anything" is. For instance, you can't use a CriticalSection to protect concurrent access to UI elements, you must Synchronize with the UI thread instead.

The server has a real issue with the client dropping the connection. When the IdTCPOnExecute event is fired, I read the clients message request, process the request and write the reply back to client, but if that connection has dropped for whatever reason it breaks the server.

Define "breaks", exactly. What is the actual problem you are experiencing? You need to be very specific.

As a test I got the client to send a request and instantly disconnect, the server crashed immediately.

"Crashed" how, exactly? Are we talking just a regular Indy/socket exception (that's normal behavior), or an actual application failure? Indy does raise exceptions on errors, and trying to read from/write to a client that has disconnected is a normal error condition that raises an exception.

Can any one shed any light on how to handle the exception

Just let the server handle the exception, so it can close its side of the connection, and cleanup the thread that triggered the event.

and stabilize the server side?

What do you consider to be "unstable", exactly. Please show your actual code, and provide details about the actual exception.

[TheMouseAUS]

Thank you for taking the time to respond. Indy has been around a long time and I have no doubts that it is stable. the issue will definitely be in my code and I am out of my depth. I reached out because I have looked at so many examples but with out proper documentation it can be very difficult to figure out as many examples I see are based around a httpserver.

Indy is a beast so without real knowledge / experience it can be a mine field.

If everything closes gracefully all is ok. I am trying to harden against when things dont. Be aware I am being really aggressive to create this error and I dont think I would ever see it in real life unless someone was being a bad actor. Dodgy wifi etc all seems to be handling ok

The server is a command-line app, no visual components and memory access to data is wrapped in TCriticalSection's. Overall everything is running well, and I have no GPF.

I did look at fcl-net from what i read its a lot more hands on and at this stage not quite ready to dip into it - yet

"Crashed" how, exactly? Are we talking just a regular Indy/socket exception (that's normal behavior), or an actual application failure? Indy does raise exceptions on errors, and trying to read from/write to a client that has disconnected is a normal error condition that raises an exception.

Just let the server handle the exception, so it can close its side of the connection, and cleanup the thread that triggered the event.

This is what i need guidance on, do you use the server exception event? At the moment I do get application failure but I am assuming ( probably incorrectly) that is caused by me not handling an exception correctly.

So if Server Exception event is fired do I call disconnect and then free?

[TheMouseAUS]

My management does indeed lack, I really want to learn to do it properly, just not sure how, I have search many places and docs are no where to be found.

I did check out synapse a long time ago, I have learnt a lot in that time, I will check it out again.

What do you mean "No alternative"?
fcl-net/web, synapse, just to name two.
I consider Indy legacy and ported everything that does not need Delphi compatibility to the standard fcl packages.

Anyway: You should handle the EIdConnClosedGracefully  exception if the client closed the connection itself.
If a connection is lost because of other things: The server receives something like a EidConnReset (not sure about the name here).
The first exception should be handled automatically in Indy servers, so which exception do you get?

Anyway: your session management seems to lack. Usually Indy is very stable.

[Thaddy]

Better check out the - come as standard - fcl-web and fcl-net packages and examples. They are family of synapse, but more advanced and better maintained. (and come with lots of examples and tests)

But what Remy wrote is true: Indy is very, very stable. Any instability must come from how you use Indy.
If you want to program using components, Indy is still a very good option.

To check what's wrong, like Remy wrote, we need some code that reproduces the problem.
Indy is used in very,very,very,very large commercial applications all over the world.
And not one, but hundreds, maybe even thousands, very large ones. (500.000+ client users? Yes)

[marcov]

I consider Indy legacy and ported everything that does not need Delphi compatibility to the standard fcl packages.

I consider the fcl packages have a way to high mutation count to be usable. fcl-web alone 420 mutations not merged to fixes (and even more relative to FPC 3.2.2).

[LeP]

The server has a real issue with the client dropping the connection. When the IdTCPOnExecute event is fired, I read the clients message request, process the request and write the reply back to client, but if that connection has dropped for whatever reason it breaks the server. As a test I got the client to send a request and instantly disconnect, the server crashed immediately. Can any one shed any light on how to handle the exception and stabilize the server side?

Thanks for reading :-)

I tested exactly that (I do this time to time) in every possible situation, with LAN, WIFI, BLE (NETWORK SIMULATION), with component, with thread, like console app with thread and I never found an issue.

Please, report the code you refer to test here.

[Thaddy]

I consider the fcl packages have a way to high mutation count to be usable. fcl-web alone 420 mutations not merged to fixes (and even more relative to FPC 3.2.2).

Well, the equivalent is fcl-net, not fcl-web, when we are talking networking. And that is very stable.
I just happen to use fcl-web units too. That is geared towards different tasks.

[TheMouseAUS]

I think I maybe onto the issue, when the app was crashing out, it appears, Lazarus debug was loosing the app. I would have to 'kill' the Lazarus application as it would sit then and do nothing. Except one crash last night it did pickup a GPF fault to one variable I had missed putting in a Critical Section, it was being written by the main thread at the same time IdTCPServer was trying to read it. I think I need look at all my locks again maybe. Its strange that doing this test would generate this fault yet most of time I had managed to get away with it!

[Thaddy]

https://en.wikipedia.org/wiki/Heisenbug

Happens all the time, especially with sometimes hard to debug multithreaded applications.
And running under a debugger may/does alter the behaviour of the application.
Things may not be as they seem.
Hence extensive logging and the use of assertions are a good way to avoid the debugger.
In your case both may help, since you are debugging a program that has real-time throughput, which makes a debugger your enemy to some extend. For example when setting a breakpoint (or crash) does not mean the socket stops receiving data: it doesn't, the other side doesn't know you have a breakpoint set or crashed and the socket is not closed when you crash or have set a breakpoint. That alone is sufficient to crash under the debugger or even crash the debugger itself.
EAssert exceptions can be used to close the socket, so you can debug gracefully.

BTW: your use of locks and cs looks very strange to me, not only with Indy but also in general.
We really need some code.
Indy handles those things low-level for you, so your code may actually introduce a conflict with how Indy is designed.
That is also as per Remy's comments if I understood him correctly.

[LeP]

Instead of debugging, use the console.

It's true that it introduces latency that could impact ratings, but it's still better than debugging ... not in all cases, but in many.

Build a function protected by a critical section (e.g., LogShow) and use it to plot whatever you want.

It's convenient, easy to use, and you can even use it in production if you need to test something for a long time and don't need to be there with a debugger, which, in the case of asynchronous events (like the network), still alters the behavior and related evaluations of the software.

Code: Pascal  [Select][+][-]

1. Uses DateUtils;
2.  
3. var CriticSec: TCriticalSection;
4.  
5. //Somewhere
6. TCriticalSection.Create;
7. //And
8. TCriticalSection.Free;
9.  
10. //Use, don't worry about thread
11. //EDIT1: correct typo "
12. LogShow('Err: should not be here - ': + 'something if needs, like vars values', MilliSecondOfTheDay(now));
13.  
14. procedure LogShow(Msg: String; Timems: Cardinal);
15. begin
16.   CriticSec.Acquire;
17.   writeln(Timems.ToString,' - ', Msg);
18.   CriticSec.Release;
19. end;
20.  

Edit: in the modern CPU (I speak for Intell), 1 write and more read concurrently (ONLY ONE WRITE !!!) a memory area is not an issue, it does not generate AV or deadlock. The issue is that this read may be related to old or new value and no one can know about (this is called RACE condition).
I speak about normal READ and WRITE like read a variable and write a variable, if the software use partially LOCK or other primitives than the things are differents.

[Thaddy]

@LeP
Creative use of double quotes, so it creatively doesn't work  (Though I know what you mean, it is a stub to replace, I use <> for that)
But logging can be done with the features of Lazarus itself, which does basically the same.
I am a great fan of logging (and assertions).

[LeP]

@LeP
Creative use of double quotes, so it creatively doesn't work  (Though I know what you mean, it is a stub to replace)
But logging can be done with the features of Lazarus itself, which does basically the same.
I am a great fan of logging (and assertions).

Sorry, I wrote speedy ....

I know about logs, but since I use Delphi ... I always use that, it's a example, an help.