Recent

Author Topic: Synapse TCP/IP client and server  (Read 13016 times)

GetMem

  • Hero Member
  • *****
  • Posts: 3352
Synapse TCP/IP client and server
« on: February 25, 2020, 08:41:12 pm »
Since there are very few examples out there, I would like to share a TCP/IP component based on synapse + a small demo application. I hope you will find it useful.
In order to compile the demo, you need two components:
 - Synapse trunk(http://www.ararat.cz/synapse/doku.php/download or OPM)
 - VirtualTreeView available both in 2.0.6 and Trunk. Since the VTV versions are slightly different, I attach two zip files, one for 2.0.6 and one for trunk.

The component supports:
- connection through proxy server(html, proxy4, proxy5)
- encrypted communication
- compressed streaming
- broadcasting
- resource strings which can be easily convert to other languages

The demo was mainly tested with Lazarus Trunk/FPC 3.0.4.

You can download the latest version from here: https://forum.lazarus.freepascal.org/index.php/topic,48677.msg406918.html#msg406918
« Last Edit: May 25, 2021, 12:13:36 pm by GetMem »

MoCityMM

  • Jr. Member
  • **
  • Posts: 72
Re: Synapse TCP/IP client and server
« Reply #1 on: March 09, 2020, 01:21:32 pm »
Nice! I like it. Thank you for sharing it, adding to my 'howto' repository.

-Mo

nouzi

  • Full Member
  • ***
  • Posts: 189
Re: Synapse TCP/IP client and server
« Reply #2 on: March 09, 2020, 02:27:23 pm »
very nice
my english is  bad
Lazarus 2.0.6 free pascal 3.0.4
Lazarus trunk  free pascal trunk 
System : linux mint 19.3 64bit  windows 7 64bit

chentung

  • New member
  • *
  • Posts: 9
Re: Synapse TCP/IP client and server
« Reply #3 on: January 29, 2021, 05:04:08 am »
wonderful examples !
Thank You !

GetMem

  • Hero Member
  • *****
  • Posts: 3352
Re: Synapse TCP/IP client and server
« Reply #4 on: January 29, 2021, 07:30:47 am »
wonderful examples !
Thank You !
You're welcome! I attach a slightly improved versions.

chentung

  • New member
  • *
  • Posts: 9
Re: Synapse TCP/IP client and server
« Reply #5 on: January 30, 2021, 02:58:31 pm »
Thank You !
I love it !

Guser979

  • New Member
  • *
  • Posts: 24
Re: Synapse TCP/IP client and server
« Reply #6 on: March 14, 2021, 02:53:47 am »
Thank you very much. I was not able to port forward on my router. So I used Zerotier and all the machines are able to access mine with their server.

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 535
  • Professional amateur ;-P
Re: Synapse TCP/IP client and server
« Reply #7 on: March 31, 2021, 10:02:45 pm »
Hey GetMem,

This looks very nice and quite interesting.
Definitely needs a good look.
Superb job!!

Cheers,
Gus
Lazarus 2.1.0(trunk) FPC 3.3.1(trunk) Ubuntu 21.04 64b Dark Theme
Lazarus 2.0.12(stable) FPC 3.2.2(stable) Ubuntu 21.04 64b Dark Theme
http://github.com/gcarreno

kupferstecher

  • Sr. Member
  • ****
  • Posts: 437
Re: Synapse TCP/IP client and server
« Reply #8 on: May 15, 2021, 11:32:55 pm »
Hello GetMem,

thanks for sharing! I'm studying and trying to understand your code right now, especially the data handling between the threads. Two questions:

1. Multithreading
As I understand you add tasks to the task list e.g. within the main thread, the task list is then polled by the TClientThread of the server application. I don't see any protection there when TList.Add is called in order to add a task (E.g. from TfMain.miBroadcastMessageClick via FTCPServer.SendMessage). The task list is accessed in the Execute method of the client thread and first the count property is checked and then in case it is not zero the list is copied to a local TList. That copy sequence is protected in a critical section, but as the list is thread specific, that critical section seems to protect nothing. Or is it just to guarantee a cache flush?
Additionally I wouldn't expect the access to TList.count to be thread safe, but I'm not sure. Can you say something to that data handling concept?


2. Synapse RecvPacket
You use RecvPacket several times, e.g. in TClientThread.RecvMessage (line 685 of uTCPSockets). I'm not sure about the synapse library, but as far as I understand the TCP-protokoll the incoming data is just a byte stream and not bundled in packets. So when you call RecvPacket how can you be sure that what you receive is the complete data and not only half of a message or several messages combined?

Regards~

GetMem

  • Hero Member
  • *****
  • Posts: 3352
Re: Synapse TCP/IP client and server
« Reply #9 on: May 19, 2021, 07:34:40 am »
Hello kupferstecher,

First of all, sorry for the late response, I was busy with other things.

1. Multithreading
Quote
As I understand you add tasks to the task list e.g. within the main thread, the task list is then polled by the TClientThread of the server application. I don't see any protection there when TList.Add is called in order to add a task (E.g. from TfMain.miBroadcastMessageClick via FTCPServer.SendMessage). The task list is accessed in the Execute method of the client thread and first the count property is checked and then in case it is not zero the list is copied to a local TList. That copy sequence is protected in a critical section, but as the list is thread specific, that critical section seems to protect nothing. Or is it just to guarantee a cache flush?
Additionally I wouldn't expect the access to TList.count to be thread safe, but I'm not sure. Can you say something to that data handling concept?
The idea behind TTaskList is that in order to broadcast a message, we have to lock, then loop through the server's client list. If we perform the actual broadcasting from inside the loop, the server will become unresponsive. By using a thread safe TTaskList, we offload the bulk of the work to the execute event of each client, this way minimize the amount of time the list needs to be locked.
Each TClientThread has its own task list, more over there is no data sharing between the different worker threads, so everything is thread safe, except when new task are added from the main thread. As you correctly noticed, the AddTask method should also be wrapped inside a critical section. I will fix this issue soon, but instead of my own critical sections, I will switch to a TThreadList, which is thread safe by default. I should have gone with a TThreadList in the first place. Thanks for noticing this bug!


2. Synapse RecvPacket
Quote
You use RecvPacket several times, e.g. in TClientThread.RecvMessage (line 685 of uTCPSockets). I'm not sure about the synapse library, but as far as I understand the TCP-protokoll the incoming data is just a byte stream and not bundled in packets. So when you call RecvPacket how can you be sure that what you receive is the complete data and not only half of a message or several messages combined?
I will quote from Synapse documentations:
Quote
Method reads all data waiting for read. If no data is received within
     TIMEOUT (in milliseconds) period, @link(LastError) is set to WSAETIMEDOUT.
     Methods serves for reading unknown size of data. Because before call this
     function you don't know size of received data, returned data is stored in
     dynamic size binary string. This method is preffered for reading from
     stream sockets (like TCP). It is very goot for receiving datagrams too!
     (UDP protocol)}
After a RecvPacket usually I check for errors(FBlockSocket.LastError > 0), so if something went wrong I will know it.
« Last Edit: May 19, 2021, 07:36:16 am by GetMem »

kupferstecher

  • Sr. Member
  • ****
  • Posts: 437
Re: Synapse TCP/IP client and server
« Reply #10 on: May 19, 2021, 12:11:35 pm »
Hello GetMem,

thanks for the response!

1. Multithreading
 OK, it's clear now :-)


2. Synapse RecvPacket
For that I'm still not convinced. The docu I also read, and it doesn't say anything about packets, just the available data of the byte stream.

What I mean is, if you send two messages from the client, and after both arrived then you call RecvPacket (and not earlier), then I would expect the function to return the data holding both messages as one string. And I don't see that case handled in your code. The other issue I see is if one message doesn't come as one IP-packet but is split up. If the second one is delayed and RecvPacket is called before arrival, then I expect to get returned the already received data of the first IP-packet. Which is not the complete message. In both cases the FBlockSocket.LastError should be OK / zero.

The reason I looked so close into that is that I think the above issues exist and I'm not sure how to handle them efficiently. Combining partial "packets" always includes some buffering and copying. But perhaps for messages smaller than IP-packet size this is no issue?


When I made some tests and sent a "frame" combined of a byte, a word, a data block and again a byte I checked the network traffic with wireshark. Each part was sent on its own (with fClientSocket.SendBuffer) and according to Wireshark also transfered in its own IP-packet. I would have expected that at least beginning with the second portion, the code should be faster than the network and thus the data should then be transfered combined in one IP-packet. But it wasn't. Could be due to the small data sizes or implementation dependent or I just don't understand the TCP protocol correctly.

GetMem

  • Hero Member
  • *****
  • Posts: 3352
Re: Synapse TCP/IP client and server
« Reply #11 on: May 19, 2021, 01:39:39 pm »
Quote
2. Synapse RecvPacket
For that I'm still not convinced. The docu I also read, and it doesn't say anything about packets, just the available data of the byte stream.

What I mean is, if you send two messages from the client, and after both arrived then you call RecvPacket (and not earlier), then I would expect the function to return the data holding both messages as one string. And I don't see that case handled in your code. The other issue I see is if one message doesn't come as one IP-packet but is split up. If the second one is delayed and RecvPacket is called before arrival, then I expect to get returned the already received data of the first IP-packet. Which is not the complete message. In both cases the FBlockSocket.LastError should be OK / zero.

The reason I looked so close into that is that I think the above issues exist and I'm not sure how to handle them efficiently. Combining partial "packets" always includes some buffering and copying. But perhaps for messages smaller than IP-packet size this is no issue?


When I made some tests and sent a "frame" combined of a byte, a word, a data block and again a byte I checked the network traffic with wireshark. Each part was sent on its own (with fClientSocket.SendBuffer) and according to Wireshark also transfered in its own IP-packet. I would have expected that at least beginning with the second portion, the code should be faster than the network and thus the data should then be transfered combined in one IP-packet. But it wasn't. Could be due to the small data sizes or implementation dependent or I just don't understand the TCP protocol correctly.
OK. Now I see what you meant. I will look into the issue, but please note I'm not a networking expert either,  I'm not familiar with all the intricacies of TCP/IP protocol. With that said, I do have a closed source project, with a similar structure as the attached demo from my previous post, which is running for years. Multiple clients communicates via the server, exchanging large amount of data, nobody ever complained about data loss.

kupferstecher

  • Sr. Member
  • ****
  • Posts: 437
Re: Synapse TCP/IP client and server
« Reply #12 on: May 19, 2021, 03:56:30 pm »
OK, I see. If it works it works :-)

Perhaps some networking experts around here can clarify if its safe that way.

avra

  • Hero Member
  • *****
  • Posts: 2128
    • Additional info
Re: Synapse TCP/IP client and server
« Reply #13 on: May 20, 2021, 08:10:26 am »
OK, I see. If it works it works :-)

Perhaps some networking experts around here can clarify if its safe that way.
I am not a networking expert but I have a dozen of Synapse based tcp/ip and serial applications working 24/7 in heavy industrial environments without leaks or dropped messages. Some of them are made more then 10 years ago (with Delphi 7 and 2006) and there was no reason for any changes since commissioning deployment on Windows servers, and some of them run on Linux servers or ARM boards that do not even restart for more then a year or two and have a high traffic load.

I can say that Synapse works well.  ;)
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

kupferstecher

  • Sr. Member
  • ****
  • Posts: 437
Re: Synapse TCP/IP client and server
« Reply #14 on: May 20, 2021, 01:10:54 pm »
Hello avra,

there's no doubt that Synapse is working very well. The question was if it can be relied on that a buffer/packet sent via TCP ist received as such packet using Synapse's RecvPacket. In my understanding TCP doesn't guarantee that, but only a stream of bytes in the correct order. Just like a Serial port.

Would be interesting to know, how you dealt with this issue, parsing the data stream or relying on packets?

Regards~

 

TinyPortal © 2005-2018