Websockets Server/Client Implementation

[hgbk]

Thanks a lot for your great efforts!!

Actually I'm still in the development and testing mode, in which I start the clients from my browser and therefore different to life situations the client instances show the same IP adress, port.

Therefore I used the alternative ("another possibility") in which I do the identification protocol in the transfer message ("Name=XXX"). This works well in my case.

Client (JavaScript) and server (Pascal) are ready now, connection via websockets is established and 1st transfer from client to server work. now it's time to debug ...

Thanks again!!!

[senglit]

First thanks for your contribution. Now I'm using this package to develop a simple app. And I got a small problem with identify different clients.
A part of my code on server side like this:

Code: Pascal  [Select][+][-]

1. procedure TSocketHandler.DoHandleCommunication(ACommunication: TWebsocketCommunicator);
2. var
3.   str: string;
4.   conn: TWSConnection;  // i store the usename combined with its TWebsocketCommunicator and some user data by a TList of TWSConnection
5.   i, msgCount: integer;
6. begin
7.   WriteLn('Connected to ', ACommunication.SocketStream.RemoteAddress.Address,
8.     ':', ACommunication.SocketStream.RemoteAddress.Port);
9.   ACommunication.OnReceiveMessage := @MessageReceived;
10.   ACommunication.OnClose := @ConnectionClosed;
11.   while ACommunication.Open do begin
12.     if not GetConnByCommunicator(ACommunication, conn) then continue;
13.     msgCount := conn.writeMsg.Count;
14.     if msgCount > 0 then begin
15.       for i := 0 to msgCount - 1 do ACommunication.WriteStringMessage(conn.writeMsg[i]);
16.       for i := 0 to msgCount - 1 do conn.writeMsg.Delete(0);
17.     end;
18.   end;
19. end;    
20.  

Now I run the server together with 2 clients on the same computer. It looks ok  but only a little problem.
this is a part of writeln message of the server:

Code: Text  [Select][+][-]

1. Message from 127.0.0.1:663 - { "token" : "C05115C4274C437C9F382291AC0C03D2", "msgtype" : "subscribe", "boxname" : "box0" }
2. Message from 127.0.0.1:663 - { "CoilOut" : "00000000", "CoilIn" : "00000000", "Humidity" : "14.5", "Temperature" : "30.2", "AC current" : "0.09", "AC voltage" : "228", "DC voltage" : "25.35", "msgtype" : "report", "token" : "3BB7235907AE4C76A432F8826623D9E7" }
3.  

The first message is sent by client1 as a controller. The second message is sent by client2 as a sensor hub. Client1 send a "subscribe" command to the sever, and the server dispatch it to client2, then client2 start to send sensor values to server every second and server will dispatch them to client1. The whole work flow is ok except: Client1 and Client2 got the same address and port!!!

I think the ports should be given by operation system and it should not be the same. but that's what I got on my screen

when I disconnect client1, the server knows and writeln 127.0.0.1:663(client1) is disconnected. client2(also 127.0.0.1:663) is still there. it's not influnced by the disconnection of client1 even they have the same ip and port. I'm really confused. Till now this problem dosen't cause any runtime problem. But I afraid of potential problem since I can't explain it.

why?

First of all, just found a little bit of time to update this project again. Fixed the thread pooling issues that resulted in 100% CPU load, as well as a few other bugs.

Though I still don’t know how to identify TWebsocketCommunincator during Connect, I found a workaround  which solves my specific problem:
I identify the TWebsocketCommunincator during the first message received from a specific client and assign it to an array of TWebsocketCommunincator (:= TWebsocketCommunincator(Sender)). So I can send separate messages to the different clients.

I answered you in your issue, but here again, with a little bit more detail. Previously the library did not track the open connections, you had to do it yourself, like with the array you are using. So what you did is actually how it was intendet. That said, I changed this, now the Handler has a thread safe list of active connections.

To identify a connection you have different options. The first one would be to look at the remote address and port (ACommunication.SocketStream.RemoteAddress.Address and ACommunication.SocketStream.RemoteAddress.Port). This combination is unique for every connection.

Another possibility is to have some form of identifier, for example upon connection your client could send a message like 'NAME=XXX' or something like that and you can read that message and then just store an association between name and the TWebsocketCommunicator object somewhere. For this you can for example use a TFPGMap, TDictionary or if performance does not matter to you and you don't want to use generics, TStringList.
BUT: when you have a threaded application, remember that you need to use thread safe objects. You can look at how the thread save connection list is managed within my TWebsocketHandler class. For further information I recommend you reading this wiki article, because multithreaded applications are not so simple: https://wiki.lazarus.freepascal.org/Multithreaded_Application_Tutorial#Critical_sections

[Warfley]

This is really quite weird, this means that something went horribly wrong (either the capturing of IP and port by the underlying ssockets lib is broken, or there is some memory management issue that both point to the same address, or some very weird caching bug), whatever it is, this should never happen.

If I have some free time I will investigate this. May I ask what OS you are using?

[senglit]

I'm using win10 lazarus2.2.0 with fpc 3.2.2

and here is the server writeln part:

Code: Pascal  [Select][+][-]

1. procedure TSocketHandler.MessageReceived(Sender: TObject);
2. var
3.   Messages: TWebsocketMessageOwnerList;
4.   m: TWebsocketMessage;
5.   Comm: TWebsocketCommunicator;
6.   jo: TJSONObject;
7.   func, utype, uname, pwd, token: string;
8.   msg, val, res: string;
9.   conn: TWSConnection;
10.   cp: PWSConnection;
11. begin
12.   Comm := TWebsocketCommunicator(Sender);
13.   Messages := TWebsocketMessageOwnerList.Create(True);
14.   jo := TJSONObject.Create;
15.   try
16.     Comm.GetUnprocessedMessages(Messages);
17.     for m in Messages do begin
18.       if m is TWebsocketStringMessage then begin
19.         msg := TWebsocketStringMessage(m).Data;
20.         WriteLn('Message from ', Comm.SocketStream.RemoteAddress.Address,
21.           ':', Comm.SocketStream.RemoteAddress.Port, ' - ', msg);
22.         jo := GetJSON(msg) as TJSONObject;
23.         func := jo.Get('msgtype');
24.         if func = 'gettoken' then begin
25.           utype := jo.Get('usertype');
26.           uname := jo.Get('uname');
27.           pwd := jo.Get('upassword');
28.           token := GenToken(utype, uname, pwd);
29.           ConfirmConn(uname, token, comm);
30.           jo.Clear;
31.           jo.Add('msgtype', 'tokenreturn');
32.           jo.Add('token', token);
33.           res := jo.AsJSON;
34.           Comm.WriteStringMessage(res);
35.         end else begin
36.           if not GetConnByCommunicator(comm, conn) then continue;
37.           if conn.utype <> 'monitor' then begin
38.             GetConnsByType('monitor', monitorList);
39.             for cp in monitorList do begin
40.               if cp^.token = '' then continue;
41.               cp^.writeMsg.Add(msg);
42.             end;
43.           end else begin
44.             if func = 'subscribe' then begin
45.               val := jo.Get('boxname');
46.               if val = null then continue;
47.               if not GetConnByUname(val, conn) then continue;
48.               jo.Clear;
49.               jo.Add('msgtype', 'subscribe');
50.               jo.Add('switch', 'on');
51.               res := jo.AsJSON;
52.               conn.writeMsg.Add(res);
53.               jo.Clear;
54.               jo.Add('msgtype', 'forcereport');
55.               res := jo.AsJSON;
56.               conn.writeMsg.Add(res);
57.             end else if func = 'unsubscribe' then begin
58.               val := jo.Get('boxname');
59.               if val = null then continue;
60.               if not GetConnByUname(val, conn) then continue;
61.               jo.Clear;
62.               jo.Add('msgtype', 'subscribe');
63.               jo.Add('switch', 'off');
64.               res := jo.AsJSON;
65.               conn.writeMsg.Add(res);
66.             end;
67.           end;
68.         end;
69.       end else begin
70.         msg := TWebsocketStringMessage(m).Data;
71.         //i don't know how to handle messages like ping pong close.... just leave it here
72.       end;
73.     end;
74.   finally
75.     Messages.Free;
76.     jo.Free;
77.   end;
78. end;
79.  

[senglit]

Today I stopped the sensor hub client and opened 2 controler clients at the same computer.

The sever says that it accepted connection for 2 times at the same ip and port. My code take them as 2 connections and saved it in monitorList(a TList with infomation of each connection). When the server needs to dispatch messges to clients, it send message to all of the connections in that TList. And then the sever raised an error.

By the way, this "Stream already closed" error always raise at connection broken. I tried to contain it in a "try except" but it still raised and the "except" part did not run. So I can't delete the broken connction from monitorList. I guess it would be a big problem.

The way the server acted is diffrent from that when I openned 1 sensor hub client and 1 controller clients.

[senglit]

The sever says that it accepted connection for 2 times at the same ip and port. My code take them as 2 connections and saved it in monitorList(a TList with infomation of each connection). When the server needs to dispatch messges to clients, it send message to all of the connections in that TList. And then the sever raised an error.

My bad. This problem is not caused by your package but my own code. I allow one user to login 2 times and when one of them logout I closed all of the connection bind with that user name. But If I login with 2 different user name and password, these 2 connections do have the same ip and port.

And  a "try except" cann't  the exception problem is still there. Does it happen because I wrote websocket server in a thread?

[Warfley]

Try except should pretty much always work, but if you are using Lazarus, it will throw the message that an exception has been risen even if the exception will be cought. Just click the continue button so the except block will be executed. This is just a little annoyance with Lazarus

[MarkMLl]

Try except should pretty much always work, but if you are using Lazarus, it will throw the message that an exception has been risen even if the exception will be cought. Just click the continue button so the except block will be executed. This is just a little annoyance with Lazarus

I'm not sure this invariably applies, but I don't think that the main program's exception handler will catch an exception thrown in a dynamically-linked library.

MarkMLl

[Warfley]

Found the problem with the identical port, I used the wrong identifier (LocalPort rather than RemotePort) pushed a fixed version to the github

[senglit]

Found the problem with the identical port, I used the wrong identifier (LocalPort rather than RemotePort) pushed a fixed version to the github

problem solved, 2 clients from the same computer got the different port now:

Code: Text  [Select][+][-]

1. box 14117 2022-03-18 11:53:02 192.168.115.212:38659
2. Connected to 127.0.0.1:25546
3. Message from 127.0.0.1:25546 - { "func" : "gettoken", "uname" : "root", "upassword" : "123456" }
4. box 14117 2022-03-18 11:53:03 192.168.115.212:38659
5. box 14117 2022-03-18 11:53:04 192.168.115.212:38659
6. Connected to 127.0.0.1:26314
7. Message from 127.0.0.1:26314 - { "func" : "gettoken", "uname" : "aaa", "upassword" : "aaa" }
8. box 14117 2022-03-18 11:53:05 192.168.115.212:38659
9.  

[senglit]

Is this a bug?

I was testing my app and wanted to see what would happen if I connect the websocket server from a web browser.

This is how I set my server:

Code: Pascal  [Select][+][-]

1.   socket := TWebSocketServer.Create(8888);
2.   socket.RegisterHandler('*', '/ws', handler, True, True);
3.  

When I connected from web browser with http://127.0.0.1/ I got an message "connection request rejected" like conneting to any other unavailable website. It's ok.

But when I connected with http://127.0.0.1:8888/ws or http://127.0.0.1:8888 the web socket server took arount 1 minute and raised an exception like the attached
image. After I click "continue" the web browser got and "Not a Websocket Request" message. If I ran my app without debugger nothing looks unnormal, the exception must be ignored automaticly.

[Warfley]

Yes, this is expected behavior, this is not an HTTP webserver, it can only handle Websocket connections, anything else will cause the Not a Websocket Request to be launched. The 1 minute delay is kinda weird though, will check it if i have time on the weekend

[senglit]

A little advise.

Do you think it's a good idea to add this into TWebsocketCommunicator? So that user can new his/her own dataset on TSocketHandler.Accept and dispose it on TSocketHandler.ConnectionClosed and I think it will be more convinient for user to handle their connection-related data than using TFPGMap.

Code: Pascal  [Select][+][-]

1. UserData:PtrInt;
2.  

Edit:
oh, there is no TWebsocketCommunicator yet when Accept. But user can new his/her dataset at TSocketHandler.DoHandleCommunication when something like "login" received.

[Warfley]

Actually, making the WebsocketCommunicator better extensible is something that was already suggested earlier. I am not so keen on simply adding a PtrInt field to it, because this is not such a clean solution. What was suggested before is to have the ability to create a custom class that inherits from TWebsocketCommunicator and can be extended with custom data or functions.

I just added this feature, see the BroadcastServer example for how to use it.

I also changed up the ConnectionList, which I previously added to the SocketHandler base class, but I think that this is too restrictive, as there might be the need for other datastructures (like a map mapping an identifier to the Object), so this is something the SocketHandler now has to manage manually (see BroadcastServer as an example).

[senglit]

Tried and worked fine.

Actually, making the WebsocketCommunicator better extensible is something that was already suggested earlier. I am not so keen on simply adding a PtrInt field to it, because this is not such a clean solution. What was suggested before is to have the ability to create a custom class that inherits from TWebsocketCommunicator and can be extended with custom data or functions.

I just added this feature, see the BroadcastServer example for how to use it.

I also changed up the ConnectionList, which I previously added to the SocketHandler base class, but I think that this is too restrictive, as there might be the need for other datastructures (like a map mapping an identifier to the Object), so this is something the SocketHandler now has to manage manually (see BroadcastServer as an example).