Recent

Author Topic: Websockets Server/Client Implementation  (Read 11877 times)

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Websockets Server/Client Implementation
« on: February 18, 2020, 10:01:33 pm »
Hey,

for a project I required a websocket server to communicate with an web application. From a first glance I wasn't really satisfied with the few results I found on google didn't really satisfy me, so I've wrote my own.

Github Page

It works fully using the fcl ssockets unit and is therefore cross plattform available without any additional packages or components (like indy or synapse) required.

Also, once the protocol was implemented for the server it was only a small step to implement also a client, so this provides an implementation for a Server and a Client.

For further details see the Readme.md on github.

For my little project for which has only to handle a few connections in a local network and everyone is nice it's now working fine, I haven't been able to test it on a larger scale. If you run into problems with it, feel free to open an issue on github or post here.
« Last Edit: February 20, 2020, 03:17:11 am by Warfley »

edwinyzh

  • New Member
  • *
  • Posts: 43
Re: Websockets Server/Client Implementation
« Reply #1 on: March 01, 2020, 04:34:56 am »
Well done!
Thanks for sharing!
Does it fully implement the Websockets protocol standard?

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: Websockets Server/Client Implementation
« Reply #2 on: March 01, 2020, 10:42:39 am »
Nicely done. I wonder whether that would work with Mycroft?

https://mycroft-ai.gitbook.io/docs/mycroft-technologies/mycroft-core/message-bus

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Zath

  • Sr. Member
  • ****
  • Posts: 391
Re: Websockets Server/Client Implementation
« Reply #3 on: March 01, 2020, 11:34:20 am »
Thanks for sharing this, very interesting.

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Websockets Server/Client Implementation
« Reply #4 on: March 05, 2020, 04:37:02 am »
Does it fully implement the Websockets protocol standard?
Mostly, it does not:
1. provide any support for extensions (i.e. no mechanism like overloading some functions or so), so any extensions need to be implemented on the source level.  I'm actually currently thinking about a nice method to implement this neatly..
2. check for the validity of UTF-8 String message. It will simply copy the bytes. The protocol requires to fail the connection in case of malformed UTF-8, which right now is currently in the responsibility of the user (i.e. if you get an error processing the messages, you need to call close explicitly). So just from a technical point of view byte messages and String messages are handled identical.
3. provide a special mechanism for  subprotocols. They need to be implemented by the User, i.e. in the TWebsocketHandler (by overriding the Accept method you also can influence the handshake), but I don't know enough about the subprotocols to know if this is sufficient.
4. Support Messages in Close-Frames like statuscodes, while this is actually not hard to implement, it's something I hadn't time to do.
5. Use websocket URI's but requires you to provide the host, port and path (including the query) directly
And lastly, using fragmented Frames, while implemented, is currently kinda pointless, as the implementation works on a message basis (for having seamless threading support), meaning the messages get buffered anyway. But controll messages are still handled, even when waiting for fragments of another message

Besides those points this should be standard compliant. But I worked more on an example basis and only superficially read through the standard and the only parts I've read in depth where the most important core protocol features.

Nicely done. I wonder whether that would work with Mycroft?
https://mycroft-ai.gitbook.io/docs/mycroft-technologies/mycroft-core/message-bus
From what I've seen this simply uses Json-Messages on top of websocket UTF-8 messages, so I don't see why it should not work. You could define a class which owns the TWebsocketCommunincator, as well as a class hierachy for the Mycroft Message types, and simply wrap it. So your wrapper class subscribes to TWebsocketCommunincator.OnRecieveMessage, recieves the messages, loads the json, and than creates the respected Mycroft message from it. Similarly it can provide a write method, which takes such a mycroft message, converts it to json and writes it to the TWebsocketCommunincator.WriteData stream.


There are also a few things I should mention: For one, it can handle SSL, but I have no idea how, as the ssockets Unit has nearly no documentation and all of my knowledge about it comes from looking at fpHttpClient. Thats why I provide ras access to the TSocketHandler, so you can still use it, if you know how. For me this isn't currently a problem, as I only need the server, which is behind an NGINX reverse proxy, who does all the HTTPS stuff.
The second thing to keep in mind with this library is that it is very conservative in terms of Threading, while it is thread safe (i.e. there can't be any problems from calling the send/recieve/close functions in parallel), it does not make any assumptions what runs on what Thread. I did this because I didn't want to rely on Synchronize or TThread.Queue, because I didn't want that library to be dependend on the user calling CheckSynchronize regularly. So if you want to lift messages to the main thread, you need to call Synchronize seperately. My application does mostly MySQL stuff, which is also thread safe, so I don't care which threads recieve and send messages. But if you are for example writing an LCL application, you might want to use synchronize or TThread.Queue


While for my purposes this is currently enough, if you wish for additional features, feel free to leave a ticket at Github, I will try to keep this library as general as possible to be able to cover many use cases, so feedback is highly appreciated, especially for cases which I haven't thought of before

mr-highball

  • Full Member
  • ***
  • Posts: 233
    • Highball Github
Re: Websockets Server/Client Implementation
« Reply #5 on: March 05, 2020, 04:36:20 pm »
Just found this and am really happy you put the time into it (and better yet, shared it). Multiple occasions I could've used a websocket library and there just wasn't any good option.

jsonnabend

  • New Member
  • *
  • Posts: 15
Re: Websockets Server/Client Implementation
« Reply #6 on: April 25, 2020, 05:30:29 pm »
Thanks for sharing this. I got it to compile in half a second and was receiving messages from the sample webpage you included.

I can't figure out how to push messages from the server back to the browser client. Any help?

Thanks again.

RG

  • Newbie
  • Posts: 5
Re: Websockets Server/Client Implementation
« Reply #7 on: July 02, 2020, 02:49:04 pm »
Thanks for sharing
I was looking for something like this. I just tested the examples one server and two clients and I noticed the CPU load goes up to 52 percent when I open the second client, when I close the clients then the load on the CPU is even greater close to 100 percent. The tests were done with Windows 10.
What could be the problem?

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Websockets Server/Client Implementation
« Reply #8 on: July 08, 2020, 02:38:47 pm »
Thanks for sharing
I was looking for something like this. I just tested the examples one server and two clients and I noticed the CPU load goes up to 52 percent when I open the second client, when I close the clients then the load on the CPU is even greater close to 100 percent. The tests were done with Windows 10.
What could be the problem?
Are you using the latest version of the master branch?

Because I "fixed" this problem (as was reported in this issue: https://github.com/Warfley/LazWebsockets/issues/9) on May 29th

Thanks for sharing this. I got it to compile in half a second and was receiving messages from the sample webpage you included.

I can't figure out how to push messages from the server back to the browser client. Any help?

Thanks again.

Check out the server example: https://github.com/Warfley/LazWebsockets/blob/ce3bcc1da0c6121236d32743018b2ba0a253a1ec/examples/chatServer.pas#L51

RG

  • Newbie
  • Posts: 5
Re: Websockets Server/Client Implementation
« Reply #9 on: July 08, 2020, 07:50:29 pm »
Quote
Are you using the latest version of the master branch?

Because I "fixed" this problem (as was reported in this issue: https://github.com/Warfley/LazWebsockets/issues/9) on May 29th

To be sure I downloaded again -> tested again -> got the same result.

Thanks


Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Websockets Server/Client Implementation
« Reply #10 on: July 09, 2020, 09:20:56 pm »
In wsutil.pas in line 462 you find a Sleep(0), replace the 0 with a positive number (e.g. 5 or 10, maybe even 100) and check it again.
You can also try to uncomment the TThread.yield() before the sleep.

Sadly I don't have much time currently at hand to do the debugging myself, but I suspect this to be the issue.

RG

  • Newbie
  • Posts: 5
Re: Websockets Server/Client Implementation
« Reply #11 on: July 10, 2020, 11:47:47 am »
Ok, I will try that.

Thank You

hgbk

  • New member
  • *
  • Posts: 9
Re: Websockets Server/Client Implementation
« Reply #12 on: February 25, 2021, 02:56:19 pm »
Not sure whether this gets attention after 1 year of silence, nevertheless:
1) great work! Thanks! Gives e the hope to solve my problem.
2) question: when I start and connect the html client several times, how can I differnetiate betwenn the 4 connections, e.g. to send different messages to each of them?

(a beginner in websockets)

hgbk

  • New member
  • *
  • Posts: 9
Re: Websockets Server/Client Implementation
« Reply #13 on: February 28, 2021, 12:56:57 pm »
Not sure whether this gets attention after 1 year of silence, nevertheless:
1) great work! Thanks! Gives e the hope to solve my problem.
2) question: when I start and connect the html client several times, how can I differnetiate betwenn the 4 connections, e.g. to send different messages to each of them?

(a beginner in websockets)

Though I still don’t know how to identify TWebsocketCommunincator during Connect, I found a workaround  which solves my specific problem:
I identify the TWebsocketCommunincator during the first message received from a specific client and assign it to an array of TWebsocketCommunincator (:= TWebsocketCommunincator(Sender)). So I can send separate messages to the different clients.

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Websockets Server/Client Implementation
« Reply #14 on: March 02, 2021, 09:29:52 pm »
First of all, just found a little bit of time to update this project again. Fixed the thread pooling issues that resulted in 100% CPU load, as well as a few other bugs.

Though I still don’t know how to identify TWebsocketCommunincator during Connect, I found a workaround  which solves my specific problem:
I identify the TWebsocketCommunincator during the first message received from a specific client and assign it to an array of TWebsocketCommunincator (:= TWebsocketCommunincator(Sender)). So I can send separate messages to the different clients.

I answered you in your issue, but here again, with a little bit more detail. Previously the library did not track the open connections, you had to do it yourself, like with the array you are using. So what you did is actually how it was intendet. That said, I changed this, now the Handler has a thread safe list of active connections.

To identify a connection you have different options. The first one would be to look at the remote address and port (ACommunication.SocketStream.RemoteAddress.Address and ACommunication.SocketStream.RemoteAddress.Port). This combination is unique for every connection.

Another possibility is to have some form of identifier, for example upon connection your client could send a message like 'NAME=XXX' or something like that and you can read that message and then just store an association between name and the TWebsocketCommunicator object somewhere. For this you can for example use a TFPGMap, TDictionary or if performance does not matter to you and you don't want to use generics, TStringList.
BUT: when you have a threaded application, remember that you need to use thread safe objects. You can look at how the thread save connection list is managed within my TWebsocketHandler class. For further information I recommend you reading this wiki article, because multithreaded applications are not so simple: https://wiki.lazarus.freepascal.org/Multithreaded_Application_Tutorial#Critical_sections

 

TinyPortal © 2005-2018