Lazarus

Programming => Networking and Web Programming => Topic started by: MortenB on March 02, 2021, 04:53:28 am

Title: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on March 02, 2021, 04:53:28 am
I would like to subscribe to certain feeds of data, and then display this data on my computer,
What I really would like now is to make an app that listens to the Binance Price-ticker, and simply displays the numbers, or everything which is received.

Does anybody have information regarding which WebSocket addon I would need, and an example of how to set this up?

I need to start a subscription.
Then I need to be able to read the returned stream.
And then close the subscription.
This just to get started. I learn best from relevant examples.

Thank you in advance.
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: Jurassic Pork on March 02, 2021, 06:57:48 am
hello,
you can try lazWebsockets (https://github.com/Warfley/LazWebsockets) !

Friendly, J.P
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on March 02, 2021, 08:36:32 am
I have installed this Lazarus WebSocket-solution.
The problem is a total lack of guides as in how to use it against various providers of data one can subscribe to.
I find no real examples of anything for this, except the provided example of a chat server and client.
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: Warfley on March 02, 2021, 10:33:16 pm
The library does only implement the base websocket protocol. Your subscription system must use some form of sub protocol on top of websockets, this is up to you to implement. The library just gives you the networking capabilities to establish streams and send and recieve messages.

The service you are subscribing to needs to have some form of documentation. If you find that, maybe I can help you with specific questions.
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: PierceNg on March 03, 2021, 01:21:06 am
The library does only implement the base websocket protocol. Your subscription system must use some form of sub protocol on top of websockets, this is up to you to implement. The library just gives you the networking capabilities to establish streams and send and recieve messages.

The service you are subscribing to needs to have some form of documentation. If you find that, maybe I can help you with specific questions.

OP mentioned Binance, so possibly this one: https://github.com/binance/binance-spot-api-docs/blob/master/web-socket-streams.md (https://github.com/binance/binance-spot-api-docs/blob/master/web-socket-streams.md)
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on March 03, 2021, 08:36:30 am
Thank you for your time and effort in this matter.
And yes.
I did mention Binance, and of course there is a lot of things I read from the Binance API.
Everything there seems to be in Joson-format.
And... making the Joson-queries and reading the Joson-answers is a piece of cake. I have no problems there, and I believe the API is well enough documented for me to work with that.
-
However.
What I have never done before is established a subscription.
Or sent any messages at all via the Internet.
What I really would like is:
1. An example of how to send messages/streams/strings to the service provider
2. An example of how to know if I received a reply, errormessage or anything that I can interpret.
3. An example of how to read the received stream.
-
In my simple mind, I consider the workflow to be like:
1. Subscribe to DataStream
2. Check for errormessage.
3. If no errormessage, then check if new data is available on/in the stream 3–4 times every second.
4. If data is available, read and display information.
and then just cycle 3 and 4.
-
-
It doesn't have to be using the Lazarus WebSockets.
I also see that Indy has all the functionality for WebSockets, among other things, and from what I read, this Indy is now very very stable. Indy is maybe the better choice?
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: Warfley on March 03, 2021, 02:20:45 pm
Well, the chatClient example should contain all the information you need, but let me break it down for you.

First you need to open up the websocket stream, represented by the TWebsocketCommunincator class:
Code: Pascal  [Select][+][-]
  1. client := TWebsocketClient.Create(address, port, path); // e.g. for wss://example.com:8080/foo?bar=foobar address would be example.com port would be 8080 (default 80) and path would be /foo?bar=foobar (default /)
  2.   try
  3.     wsstream := client.Connect(TSocketHandler.Create); // Use an SSL socket handler for SSL, there are examples in this forum how to do this
  4.     try
  5.       ...
  6.     finally
  7.       wsstream.Free;
  8.     end;
  9.   finally
  10.     client.Free;
  11.   end;
Next step is to send a json string:
Code: Pascal  [Select][+][-]
  1. wsstream.WriteStringMessage(jsonString);
For recieving messages just call:
Code: Pascal  [Select][+][-]
  1. wsstream.RecieveMessage;
This function will block until a message was read. Messages read by it will be stored in the thread safe wsstream.Messages datastructure. To read all recieved messages call "GetUnprocessedMessages".

So if you expect exactly one response message and you want to wait for it blocking:
Code: Pascal  [Select][+][-]
  1. function RecieveResponse(wsstream: TWebsocketCommunicator): String;
  2. var
  3.   msgList: TWebsocketMessageOwnerList;
  4.   msg: TWebsocketMessage;
  5. begin
  6.   Result := nil;
  7.   msgList := TWebsocketMessageOwnerList.Create;
  8.   try
  9.     while wsstream.Open and not Assigned(Result) do
  10.     begin
  11.       wsstream.Recievemessage;
  12.       msgList.Clear;
  13.       if wsstream.GetUnprocessedMessages(msgList) > 0 then
  14.         for msg in msgList do
  15.           if msg is TWebsocketStringMessage then
  16.             Exit(TWebsocketStringMessage(msg).Data);
  17.     end;
  18.   finally
  19.     msgList.Free;
  20.   end;
  21. end;
This function will wait for and recieve the very first response that is a string and discards all other messages (binary messages or pong messages)

If you want to recieve messages asynchonously, i.e. you don't want to explicitly wait for messages, you can do so by setting the OnRecieveMessage event handler and execute Recievemessage in a seperate thread:
Code: Pascal  [Select][+][-]
  1.   TRecieverThread = class(TThread)
  2.   private
  3.     FCommunicator: TWebsocketCommunincator;
  4.   protected
  5.     procedure Execute; override;
  6.   public
  7.     constructor Create(ACommunicator: TWebsocketCommunincator);
  8.   end;
  9.  
  10. ...
  11.  
  12. procedure TRecieverThread.Execute;
  13. begin
  14.   while not Terminated and FCommunicator.Open do
  15.   begin
  16.     FCommunicator.RecieveMessage;
  17.     Sleep(100);
  18.   end;
  19. end;
  20.  
  21. constructor TRecieverThread.Create(ACommunicator: TWebsocketCommunincator);
  22. begin
  23.   FCommunicator := ACommunicator;
  24.   inherited Create(True);
  25.   FreeOnTerminate := True;
  26. end;
  27.  
  28. ...
  29.   wsstream.OnRecieveMessage := @MessageHandler;
  30.   TRecieverThread.Create(wsstream).Start;
Now the thread will automatically check for messages in the background and when one is recieved, MessageHandler will be called. BUT: MessageHandler will be called from within the context of the Reciever thread. So if you want to, for example, display the message in your GUI, you need to use synchronize or TThread.Queue:
Code: Pascal  [Select][+][-]
  1. TMyClass = class
  2.   LatestMessage: String;
  3.   procedure UpdateUI;
  4.   procedure MessageHandler(Sender: TObject);
  5.  
  6. ...
  7.  
  8. procedure TMyClass.UpdateUI;
  9. begin
  10.   Label1.Caption := LatestMessage;
  11. end;
  12.  
  13. procedure TMyClass.MessageHandler(Sender: TObject);
  14. begin
  15. var
  16.   MsgList: TWebsocketMessageOwnerList;
  17.   m: TWebsocketMessage;
  18. begin
  19.   LatestMessage := nil;
  20.   MsgList := TWebsocketMessageOwnerList.Create(True);
  21.   try
  22.     TWebsocketCommunicator(Sender).GetUnprocessedMessages(MsgList);
  23.     for m in MsgList do
  24.       if m is TWebsocketStringMessage then
  25.         LatestMessage := TWebsocketStringMessage(m).Data;
  26.   finally
  27.     MsgList.Free;
  28.   end;
  29.   if Assigned(LatestMessage) then
  30.     Synchronize(@UpdateUI);
  31. end;
TMyClass can for example be your TForm derivate you are working in, if you are working in a GUI application.
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on March 03, 2021, 03:31:21 pm
Well, the chatClient example should contain all the information you need, but let me break it down for you.

Thank you so much for the "breakdown"  :D
I think I can handle this pretty well.

It is too late for me tonight, but if I have time, I will try to do something tomorrow.
Ideally, what I would like to do is to make my form wait for input from mouse and keyboard at all times.
a timer will be set to check for messages maybe 10 times per second (just an example.)
Then if the timed "interrupt" find received message(s), they will be dealt with before letting go.

Your "breakdown" of this is great. If I encounter problems, I will try to ask for more details.
Thank you!
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: Warfley on March 03, 2021, 04:10:50 pm
After writing this post I noticed that this is a little bit overcomplicated, so I just added some new functions to the library that makes most of what I wrote previously invalid.

You now don't need to call RecieveMessage anymore. If you are waiting for a message and want to block until you have it, you can now use WaitForMessage or if you are awaiting a special kind of message WaitForStringMessage, WaitForBinaryMessage and WaitForPongMessage. In your case, awaiting the response will therefore look something like this:
Code: Pascal  [Select][+][-]
  1. msg := wsstream.WaitForStringMessage;
  2. try
  3.   if not Assigned(msg) then // stream closed (disconnect, server closed stream, etc.) before message was recieved
  4.     // Error handling
  5.   jsonString := msg.Data;
  6. finally
  7.   msg.Free;
  8. end;
This is very useful if you are expecting a response, e.g. when you do your subscription and need to check if it was successful.

You also now don't need to create your own TThread instance for recieving messages, simply call wsstream.StartRecieveMessageThread and a new thread will be started recieving messages (see the Readme for further information about threading).

Most importantly, WaitForXXXMessage works fine in both situations, if you have a reciever thread and if you don't.

Quote
Then if the timed "interrupt" find received message(s), they will be dealt with before letting go.
This is also quite simple, first start the reciever thread, then do the following in your timer:
Code: Pascal  [Select][+][-]
  1. procedure TForm1.Timer1Timer(Sender: TObject);
  2. var
  3.   Messages: TWebsocketMessageOwnerList;
  4.   Message: TWebsocketMessage;
  5. begin
  6.   Messages := TWebsocketMessageOwnerList.Create;
  7.   try
  8.     wsstream.GetUnprocessedMessages(Messages);
  9.     for Message in Messages do
  10.       // Update your UI with the contents of Message
  11.   finally
  12.     Messages.Free;
  13.   end;
  14. end;

This is also completely thread safe, so you don't need to worry about threading issues (as would be the case when using the event OnRecieveMessage)
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: bytebites on March 04, 2021, 12:26:54 pm
Recieve should be written as receive.
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on March 05, 2021, 03:31:34 pm
First you need to open up the websocket stream, represented by the TWebsocketCommunincator class:
Code: Pascal  [Select][+][-]
  1. client := TWebsocketClient.Create(address, port, path); // e.g. for wss://example.com:8080/foo?bar=foobar address would be example.com port would be 8080 (default 80) and path would be /foo?bar=foobar (default /)
  2.   try
  3.     wsstream := client.Connect(TSocketHandler.Create); // Use an SSL socket handler for SSL, there are examples in this forum how to do this
  4.     try
  5.       ...
  6.     finally
  7.       wsstream.Free;
  8.     end;
  9.   finally
  10.     client.Free;
  11.   end;
I have to admit. SSL Socket handler is also new to me. I have no clue.
Basically what I end up with when I try to compile the line: "wsstream := client.Connect(..."
is an errormessage: Fatal: Syntax error, "." expected but ":=" found.

I started with a button to start subscription
Code: Pascal  [Select][+][-]
  1. client := TWebsocketClient.Create('wss://stream.binance.com', 9443, '/');
and should send a JSON-string to request subscription data when connection has been confirmed.
{
    "type": "subscribe",
    "exchange": "binance",
    "pair": "eth-btc",
    "channel": "trade"
}

My simple idea here was to start a timer that constantly checks if a message is received, and then displays whatever is received in an edit-box or something.

Then a button to stop subscription


Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: Warfley on March 05, 2021, 06:05:25 pm
I have to admit. SSL Socket handler is also new to me. I have no clue.
I think you can simply use the class method "TSSLSocketHandler.GetDefaultHandler" from the unit sslsockets to use ssl
Basically what I end up with when I try to compile the line: "wsstream := client.Connect(..."
is an errormessage: Fatal: Syntax error, "." expected but ":=" found.
Sorry, I am stupid, wsstream is the name of one of my units, so you need to name the variable differently.

I started with a button to start subscription
and should send a JSON-string to request subscription data when connection has been confirmed.
{
    "type": "subscribe",
    "exchange": "binance",
    "pair": "eth-btc",
    "channel": "trade"
}

My simple idea here was to start a timer that constantly checks if a message is received, and then displays whatever is received in an edit-box or something.

Then a button to stop subscription

I recommend the following, put the client and stream variable into you Form (e.g. as private variables). In FormCreate create both variables and set up events:
Code: Pascal  [Select][+][-]
  1. Self.Client := TWebsocketClient.Create('stream.binance.com', 9443, '/');
  2. Self.Stream := Self.Client.Connect(TSSLSocketHandler.GetDefaultHandler);
  3. Self.Stream.OnClose := @HandleClose; // Handle close is a procedure HandleClose(sender: TObject); in your class
  4. Self.Stream.StartReceiveMessageThread; // handle all incoming messages
And free them in the FormDestroy event:
Code: Pascal  [Select][+][-]
  1. Self.Stream.Free;
  2. Self.Client.Free;

Then in your subscribe button you start the subscription by sending your json request and awaiting the correct answer:
Code: Pascal  [Select][+][-]
  1. Self.Stream.WriteStringMessage('JSON subscribe request');
  2. response := Self.Stream.WaitForStringMessage;
  3. try
  4.   // Check if response.Data contains the expected response
  5. finally
  6.   response.Free;
  7. end;
In your unsubscribe button you do the same with the unsubscribe request

In your timer you then put the loop to check for received messages
Code: Pascal  [Select][+][-]
  1.     procedure TForm1.Timer1Timer(Sender: TObject);
  2.     var
  3.       Messages: TWebsocketMessageOwnerList;
  4.       Message: TWebsocketMessage;
  5.     begin
  6.       Messages := TWebsocketMessageOwnerList.Create;
  7.       try
  8.         wsstream.GetUnprocessedMessages(Messages);
  9.         for Message in Messages do
  10.           // Update your UI with the contents of Message
  11.       finally
  12.         Messages.Free;
  13.       end;
  14.     end;

Also note, that I had updated the code today, so I highly recommend you downloading the newest version. Best to load it via git so you can easiely check for updates using git pull
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on March 07, 2021, 03:05:12 pm
I have now been back and forth a bit with the code, but there are issues I cannot easily solve here.
First thing first, a tip: "TWebSocketCommunincator", should this not be "TWebSocketCommunicator"? (Just a typo, maybe intended, I don't know.)
-
Then my big problem.
First my code:
Code: Pascal  [Select][+][-]
  1. uses
  2.   Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls, ExtCtrls,
  3.   wsutils,
  4.   wsmessages,
  5.   wsstream,
  6.   ssockets,
  7.   WebsocketsClient;
  8.  
  9. type
  10.  
  11.   { TForm1 }
  12.  
  13.   TForm1 = class(TForm)
  14.     btnSubscribe: TButton;
  15.     btnUnsubscribe: TButton;
  16.     edtPort: TEdit;
  17.     edtSubscribeAddress: TEdit;
  18.     mmoResult: TMemo;
  19.     mmoSubscriptionInfo: TMemo;
  20.     MessageTimer: TTimer;
  21.     procedure btnSubscribeClick(Sender: TObject);
  22.     procedure FormCreate(Sender: TObject);
  23.     procedure FormDestroy(Sender: TObject);
  24.     procedure MessageTimerTimer(Sender: TObject);
  25.   private
  26.     MyClient   : TWebsocketClient;
  27.     MyStream   : TWebSocketCommunincator;//TStream;
  28.     procedure HandleClose(sender: TObject);
  29.   public
  30.   end;
  31.  
  32. var
  33.   Form1: TForm1;
  34.   MyStreamVar : TLockedSocketStream;
  35. implementation
  36.  
  37. {$R *.lfm}
  38.  
  39. { TForm1 }
  40.  
  41. procedure TForm1.HandleClose(Sender: TObject);
  42.   begin
  43.   end;
  44.  
  45. procedure TForm1.FormCreate(Sender: TObject);
  46.   begin
  47.     MyStream.Create(MyStreamVar,False,False);
  48.     MyClient := TWebsocketClient.Create('stream.binance.com', 9443, '/');
  49.     MyStream := MyClient.Connect(TSocketHandler.Create);// GetDefaultHandler
  50.     MyStream.OnClose := @HandleClose; // Handle close is a procedure HandleClose(sender: TObject); in your class
  51.     MyStream.StartReceiveMessageThread; // handle all incoming messages
  52.   end;
  53.  
  54. procedure TForm1.btnSubscribeClick(Sender: TObject);
  55.   VAR
  56.     MyResponse : TWebSocketStringMessage;
  57.     MyTextData : UTF8String;
  58.   begin
  59.     {
  60.     MyTextData:='';
  61.     MyResponse.Create(MyTextData);
  62.     MyStream.WriteStringMessage(mmoSubscriptionInfo.Lines.Text);// 'JSON subscribe request'
  63.     MyResponse := MyStream.WaitForStringMessage;
  64.     try
  65.       // Check if response.Data contains the expected response
  66.     finally
  67.       MyResponse.Free;
  68.     end;
  69.     }
  70.   end;
  71.  
  72. procedure TForm1.FormDestroy(Sender: TObject);
  73.   begin
  74.     MyStream.Free;
  75.     MyClient.Free;
  76.   end;
  77.  
  78.  
  79. procedure TForm1.MessageTimerTimer(Sender: TObject);
  80.   var
  81.     Messages: TWebsocketMessageOwnerList;
  82.     Message: TWebsocketMessage;
  83.   begin
  84.     {
  85.     Messages := TWebsocketMessageOwnerList.Create;
  86.     try
  87.       MyStream.GetUnprocessedMessages(Messages);
  88.       for Message in Messages do
  89.         // Update your UI with the contents of Message
  90.       mmoResult.Append(Message.ToString);
  91.     finally
  92.       Messages.Free;
  93.     end;
  94.     }
  95.   end;
  96. end.
So far, I get no error on compile.
When running, I get SIGSEGV on: MyStream.OnClose := @HandleClose;
When I // don't do this line, I get SIGSEGV on the next line, MyStream.StartReceiveMessageThread;, which then point to line 645 in the wsstream-unit:   if not Assigned(FReceiveMessageThread) then...

There just seems to be an endless list of things popping up here.
Maybe because I don't have the right units, maybe because I don't know how to initialize the variables properly.
I realize that this wasn't a walk in the park :) But I don't want to give up  >:D 8-)

I don't even know if I need to install something for SSL-functionality.
Your example gives: TSSLSocketHandler
But I cannot seem to find this here.


Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on March 08, 2021, 06:43:08 am
I finally found out about the TSSLSocketHandler.GetDefaultHandler. Hoorayyy. One step further!
Had to add sslsockets and OpenSSLsockets to the Uses  :D
But still. The problem persist when I start the handling of incoming messages: X.StartReceiveMessageThread.
It goes to line 645 in the wsstream-unit and ends with a SIGSEGV
The exact error: Project xxx raised exception class 'External: SIGSEGV'. In file wsstream.pas at line 645: if not Assigned(FReceiveMessageThread) then
-
I hope you can point me in the right direction here.
Thanks in advance.
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: Warfley on March 08, 2021, 04:20:02 pm
This problem is because the server declined the handshake, in this case Client.Connect will return nil, and when trying to work on that, you get an access violation.

To be precise, the server returns http code "400: Bad request". Probably something I should throw an exception over, is on my todo list. But for now, it seems like you are doing a faulty request. Probably because you try to access the path '/', while the github doc states:
Quote
Raw streams are accessed at /ws/<streamName>
Combined streams are accessed at /stream?streams=<streamName1>/<streamName2>/<streamName3>
so your path is probably something like '/ws/streamname' or '/stream?streams=...'

PS: the line
Code: Pascal  [Select][+][-]
  1. MyStream.Create(MyStreamVar,False,False);
Is completely out of place and should be removed
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on March 09, 2021, 07:45:44 am
There are a few issues, yes.
I have tried and tried various approaches to achieve a connection and not get the SIGSEGV-error.
In the end, I installed a Chrome Addon with Direct Web Sockets Functionality:
  https://chrome.google.com/webstore/detail/smart-websocket-client/omalebghpgejjiaoknljcfmglgbpocdp/related
This to test the address that I try to use via Lazarus.
-
First things first.
The Chrome Addon, with the address:
  wss://stream.binance.com:9443/ws/btcusdt@ticker
connects very well and instantly starts receiving  the tickers in JSON-format
{
  "e": "24hrTicker",
  "E": 1615268894603,
  "s": "BTCUSDT",
  "p": "3394.80000000",
  "P": "6.697",
  "w": "51548.22027742",
  "x": "50688.12000000",       ... and so on ...

However.
When I try to do the same with your WebSocket-implementation, I cannot get it to work.
Code: Pascal  [Select][+][-]
  1.     MyClient := TWebsocketClient.Create('wss://stream.binance.com',9443,'/ws/btcusdt@ticker');
  2.  
does not produce an error, but the next step:
Code: Pascal  [Select][+][-]
  1. MyCommunicator := MyClient.Connect(TSSLSocketHandler.GetDefaultHandler);
gives a Debugger Exception Notification: Project xxx raised exception class 'ESocketError' with message: Host name resolution for "wss://stream.binance.com" failed.
It basically failes at line 74: ASocket.Connect; in the WebsocketsClient-file
-
I have performed a nslookup from a command window:
nslookup wss://stream.binance.com
and the answer is 52.198.117.1 and 54.65.212.75
-This is the exact same address as ASocket.Connect cannot resolve.

Unfortunately, from there and on, I am in deep waters,

More things I have tried.
Deactivated my AntiVirus.
Deactivated Windows Firewall.
Neither of which changes the outcome.





Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: Warfley on March 09, 2021, 10:01:24 am
Have you tried to not write "wss://" in front of the hostname?
i.e. TWebsocketClient.Create('stream.binance.com',9443,'/ws/btcusdt@ticker')

The reason for this is simple, the wss:// is not part of the host, but is part of the URI. But, this library doe not work with URIs
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on March 09, 2021, 10:16:49 am
I have tried just about everything, also leaving out the "wss://"
When leaving out "wss://" it ends up with external SIGSEGV in wsstream.pas at line 645 as earlier mentioned.
-
Sad. I need to work with URIs.
Do you have any "immediate" plans to include this in your library?
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: Warfley on March 09, 2021, 11:07:40 am
I've found the problem. The biance server used a lowercase value for the connection header, which is not explicitly allowed by the HTTP standard (but is by the websockets standard), and therefore was not implemented by me that way.
I updated the code to allow for this. Now the following example works completely fine:
Code: Pascal  [Select][+][-]
  1. uses
  2.   Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls, ExtCtrls,
  3.   opensslsockets, sslsockets, WebsocketsClient, wsstream, wsmessages;
  4.  
  5. type
  6.  
  7.   { TForm1 }
  8.  
  9.   TForm1 = class(TForm)
  10.     Memo1: TMemo;
  11.     Timer1: TTimer;
  12.     procedure FormCreate(Sender: TObject);
  13.     procedure FormDestroy(Sender: TObject);
  14.     procedure Timer1Timer(Sender: TObject);
  15.   private
  16.     FClient: TWebsocketClient;
  17.     FStream: TWebsocketCommunincator;
  18.   public
  19.  
  20.   end;
  21.  
  22. var
  23.   Form1: TForm1;
  24.  
  25. implementation
  26.  
  27. {$R *.lfm}
  28.  
  29. { TForm1 }
  30.  
  31. procedure TForm1.FormCreate(Sender: TObject);
  32. begin
  33.   FClient := TWebsocketClient.Create('stream.binance.com',9443,'/ws/btcusdt@ticker');
  34.   FStream := FClient.Connect(TSSLSocketHandler.GetDefaultHandler);
  35.   FStream.StartReceiveMessageThread;
  36. end;
  37.  
  38. procedure TForm1.FormDestroy(Sender: TObject);
  39. begin
  40.   FStream.Free;
  41.   FClient.Free;
  42. end;
  43.  
  44. procedure TForm1.Timer1Timer(Sender: TObject);
  45. var
  46.   MsgList: TWebsocketMessageOwnerList;
  47.   Message: TWebsocketMessage;
  48. begin
  49.   MsgList := TWebsocketMessageOwnerList.Create;
  50.   try
  51.     FStream.GetUnprocessedMessages(MsgList);
  52.     for Message in MsgList do
  53.     begin
  54.       if not (Message is TWebsocketStringMessage) then
  55.         Continue;
  56.       Memo1.Lines.Add(TWebsocketStringMessage(Message).Data);
  57.     end;
  58.   finally
  59.     MsgList.Free;
  60.   end;
  61. end;

PS: HTTP does not know about URIs so from a technical standpoint including wss:// does not make any sense. This is just so general purpose applications like JS can distinquish between websocket and HTTP URIs, but if you start instanciating a websocket client, it should already be pretty obvious that this is not HTTP. Also the second s in wss means using SSL (i.e. a TSSLSocketHandler), so if you have a ws:// uri you don't need ssl, if you have wss:// you need ssl. But this is all handled seperately in the code, so there is no need for URIs
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on March 09, 2021, 05:41:34 pm
I updated the code to allow for this. Now the following example works completely fine:
Code: Pascal  [Select][+][-]
  1. uses
  2. procedure TForm1.FormCreate(Sender: TObject);
  3. begin
  4.   FClient := TWebsocketClient.Create('stream.binance.com',9443,'/ws/btcusdt@ticker');
  5.   FStream := FClient.Connect(TSSLSocketHandler.GetDefaultHandler);
  6.   FStream.StartReceiveMessageThread;
  7. end;
  8.  

Thank you for your quick reply and update... but alas. No success on my end.
There is still a problem when the third line:   FStream.StartReceiveMessageThread;   is executed.
Then I get the errormessage: Project xxx raised exception class 'EWebsocketReadError' with message: error reading from stream... in file wsstream.pas at line 466: Stream.LastError);

However.
If I do a step by step approach, and step into the lines, then the connection is Ok. The stream gets a few seconds to "fill up", and when the timer kicks in, the Memo is filled with Binance Ticker-data.
And then... when the stream is empty, it crashes: At least that is what I had here a couple of times, but I can't even reproduce this now.


And again. I had an Idea.
Code: Pascal  [Select][+][-]
  1. function TWebsocketCommunincator.ReceiveMessage: TWebsocketMessage;
  2.  
  3.   procedure ReadData(var buffer; const len: int64);
  4.   var
  5.     ToRead: longint;
  6.     Read: longint;
  7.     LeftToRead: int64;
  8.     TotalRead: int64;
  9.     oldTO: integer;
  10.     Stream: TSocketStream;
  11.   const
  12.     IOTimeoutError = {$IFDEF UNIX}11{$ELSE}10060{$EndIf};
  13.     WaitingTime = 10;
  14.   begin
  15.     Sleep(1000);
  16.     TotalRead := 0;
  17.     repeat
  18.       // how much we are trying to read at a time
  19.       LeftToRead := len - TotalRead;
  20.       if LeftToRead > ToRead.MaxValue then
  21.         ToRead := ToRead.MaxValue
  22.       else
  23.         ToRead := LeftToRead;
  24.       // Reading
  25.  
  26.       Stream := FStream.LockRead;
  27.       try
  28.         if not Assigned(Stream) then
  29.         begin
  30.           raise EWebsocketReadError.Create('Socket already closed', 0);
  31.         end;
  32.         oldTO := Stream.IOTimeout;
  33.         Stream.IOTimeout := 1;
  34.         try
  35.           Read := Stream.Read(PByte(@buffer)[TotalRead], ToRead);
  36.           if Read < 0 then
  37.           begin
  38.             // on Error
  39.             if Stream.LastError <> IOTimeoutError then
  40.               raise EWebsocketReadError.Create('error reading from stream',
  41.                 Stream.LastError);
  42.           end
  43.           else
  44.           begin
  45.             // Increase the amount to read
  46.             TotalRead += Read;
  47.           end;
  48.         finally
  49.           Stream.IOTimeout := oldTO;
  50.         end;
  51.       finally
  52.         FStream.UnlockRead;
  53.       end;
  54.       if (TotalRead < len) and (Read <> ToRead) then // not finished, wait for some data
  55.         Sleep(WaitingTime);
  56.     until TotalRead >= len;
  57.   end;
As you notice, I added Sleep(1000) in your ReceiveMessage-function.
As a less than elegant solution, this solves the problem, and I can now listen to the stream. No more errors...
I really would like more control over the error-situation.
How about replacing the Raise EWebsocketReadError with simply a message? like: "!No data yet!", and that way, when I look for the JSON-strings, I can just avoid the "!No data yet!"-strings, or even count them, and if too many occur on a row, then I can close and restart the connection automatically.
I also see that you have a Stream.IOTimeout=1 in there. How about changing this so that this is a property that can be changed, but that it defaults to 1.
Not sure what 1 is, maybe 1 millisecond.

Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: Warfley on March 09, 2021, 10:22:21 pm
The thing with IOTimeout is that this is intendet to be cought:
Code: Pascal  [Select][+][-]
  1.             if Stream.LastError <> IOTimeoutError then
  2.               raise EWebsocketReadError.Create('error reading from stream',
  3.                 Stream.LastError);
The idea is, as this function acquires some locks, to only read what data is there and if there was a timeout to simply wait a little bit and try again, to not block all the locks until the whole data has been read. So if no data is there yet, it should behave exactly as you want it to, it simply waits a bit and tries again. 1 is the minimum timeout (1 ms) so it basically just reads  what the OS has buffered and does not wait for anything

Could you check out what the actual value of Stream.LastError is and what OS you are using (these error codes are OS specific, e.g. 10060 on windows is the timeout error code).
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on March 10, 2021, 03:32:53 am
Not being able to directly see the error-code from the Stream.LastError, I changed the code slightly:
I added an extra variable.
Code: Pascal  [Select][+][-]
  1. var
  2.   ErrorNumber : LongInt;

and then:
Code: Pascal  [Select][+][-]
  1.             if Stream.LastError <> IOTimeoutError then BEGIN
  2.               ErrorNumber := Stream.LastError;
  3.               raise EWebsocketReadError.Create('error reading from stream',Stream.LastError);
  4.             END;
  5.  
After the break or raised error:
When I then mouse-over IOTimeoutError, I get the value 10060, just to show that the variable values are showing when debugging.
The same mouse-over on ErrorNumber, gives 0 or Zero.

My Os is Windows 10
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: Jurassic Pork on March 10, 2021, 07:27:23 am
hello,
with this code in the function TWebsocketCommunincator.ReceiveMessage of the file wsstream.pas, it seems to be OK :
Code: Pascal  [Select][+][-]
  1.        try
  2.           Read := Stream.Read(PByte(@buffer)[TotalRead], ToRead);
  3.           if Read < 0 then
  4.           begin
  5.             // on Error
  6.              if (Stream.LastError <> IOTimeoutError) and
  7.                 (Stream.LastError <> 0) then
  8.                raise EWebsocketReadError.Create('error reading from stream',
  9.                      Stream.LastError);
  10.           end
  11.           else
  12.           begin
  13.             // Increase the amount to read
  14.             TotalRead += Read;
  15.           end;
  16.         finally
  17.           Stream.IOTimeout := oldTO;
  18.         end;

Windows 10 Lazarus 2.0.12 fpc 3.2.0

friendly, J.P
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: Warfley on March 10, 2021, 04:23:08 pm
Well, this is really weird, because error number 0 means there was no error, maybe some weird quirk of the windows api, but alas, I added the (error <> 0) condition as it seems to not break anything
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on March 10, 2021, 06:02:18 pm
Great. I was going to add that myself here, then I saw The friendly "Pork" here suggest it too (Thank you Pork, much appreciated), but I wanted to wait for your judgement on this too O:-)

I have not had time to check yet how it works here, but if it works, then I believe I can celebrate a little.

Thank you very much.

My next step is to make REST-calls using GET and/or PUT, but I suppose that should be in a different thread altogether.
Anyway.
Thank you very much again!
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on March 13, 2021, 07:28:59 am
 :D Everything here works very well now.
All tested at my side now and I see no problems with the reception of streams anymore.

Thank you again for your great help 8-) O:-)
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on April 13, 2021, 05:29:20 pm
After testing a lot with the WebSocket-solution, I was trying to ramp up the size of receiving text.
And this... not functional in a meaningful manner.
I cannot say exactly where it goes wrong (sizewise), but text in the order of half a megabyte seems to be not possible.
The file/unit "wsstream" constantly reports RunError 201 at line 563:
Code: Pascal  [Select][+][-]
  1.         if Header.PayloadLen < 126 then
  2.           len := Header.PayloadLen
  3.         else if Header.PayloadLen = 126 then
  4.         begin
  5.           ReadData(len16, SizeOf(len16));
  6.           len := NToHs(len16);
  7.         end
  8.         else
  9.         begin
  10.           ReadData(len64, SizeOf(len64));
  11.           len := ntohll(len64);   // Recurring errors here when the size is "substantial".
  12.           // Always RunError 201
  13.         end;
  14.         if Header.Mask then
  15.         begin
  16.           ReadData(MaskRec.Key, SizeOf(MaskRec.Key));
  17.         end
When I break and check the value of "len", some times it is "0" = Zero
Other times it is like "8442272860644245284" (Actual example copied out from the "Watches"-window.)
Sometimes my program runs for 1 second after I enable the websockets, other times it just crashes right away.
When it runs for 1 second, I do receive ONE String.
During debug I have found the size to be around 303642–306021 bytes, but most often it just crashes right away.

The line
Code: Pascal  [Select][+][-]
  1. ReadData(len64, SizeOf(len64));
During debug I can find that the value of len64 is -3392894971223736320

I took my chances, and thought that Maybe, the len64 was declared wrongly in the VAR-section.
Code: Pascal  [Select][+][-]
  1. var
  2.   Header: TWebsocketFrameHeader;
  3. //  len64: int64;
  4.   len64: QWord; // Mortens attempt 13.4.2021 to fix error below at line 563
  5.   len16: word;
  6.   len: int64;
  7.   MaskRec: TMaskRec;
  8.   buffer: TBytes;

I changed the int64 to QWord, and Voila.
Now I can receive the huge chunks of texts, seemingly without errors.
I have NO IDEA if my change might cause different problems, but this solved my immediate problem. :D :D :D
Maybe Warfley has a comment on my fix?
Is it an Ok fix? or should the problem be fixed elsewhere?
Thanks in advance  8) 8) 8)
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: Warfley on April 14, 2021, 10:42:17 am
The reason the len64 is negative is because it is in network byte order, not host byte order. On an x86(_64) system this means the bytes need to be swapped, this is what ntohll (network to host long long) is for.

But the runtime error is really weird, are rangechecks enabled? Thats probably the reason, but in any case, using QWord is probably the best solution here, as the sign is not required anyway prior to conversion.

It would be nice if you could put this change into a pull request for the repository, or at least open up an issue so I can fix this in the repository myself later (just so I don't forgett).
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on April 14, 2021, 01:03:45 pm
So many things I don't know about network standards and communication things there :(
Great that QWord is an acceptable solution.

As for the runtime error, yes. RangeChecks are enabled in my setup for my project.
I am doing a bit of experimenting with types, pointers and variables, so safer for me that way.
Working on data in many dimensions, and one error and everything goes wrong :p

Fortunately, your websocket-solution seems to be handling everything I need now.
What I need next is

For my need, I can also send Pong without it being requested. I just need to learn how to do it.
This to prevent the broadcaster from closing the connection, thinking that it is not in use anymore.

Also. One last question.
Regarding speed.
How much data/text should I be able to receive per second through this solution? Is it possible to make an estimate here, or is this completely dependent on the Internet-connection and computer hardware?
Thanks a lot!
Morten.
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on April 14, 2021, 01:34:28 pm
Not sure how a pull-request works in the repository, so I made an issue there. Such a minor one :D
Thanks again :D
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: Warfley on April 14, 2021, 04:12:16 pm
For my need, I can also send Pong without it being requested. I just need to learn how to do it.
This to prevent the broadcaster from closing the connection, thinking that it is not in use anymore.
Unsolicited pong messages are ignored according to the websocket standard. You don't need to handle them yourself, when the RecieveMessage function recieves a ping, it will automatically answer with a pong. So as long as you keep receiving, it is taken care of.
Also. One last question.
Regarding speed.
How much data/text should I be able to receive per second through this solution? Is it possible to make an estimate here, or is this completely dependent on the Internet-connection and computer hardware?
Can depend on both, but usually your network connection is the limiting factor. But of course, if you have a high CPU usage, a very old motherboard or a very fast network connection, it could be that your network delives the data faster then the thread can read it. But generally speaking it should be as fast as your usual download speed. (Unless the server has a limit in, or is under heavy load, you can of course not receive faster than the server is sending)
Title: Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
Post by: MortenB on April 14, 2021, 06:22:20 pm
Great.
Thank you for your answers.
In regards to the Pong, I don't know much about the websocket standards, but I have seen that Binance recommends one being sent every half hour or so in order to ensure connection not being broken.
They also say that unsolicited are allowed.
Well. Since you say it is automatically taken care of, I will not give that another thought.
In fact, sometimes the connection runs for 1 hour, sometimes it runs for 18 hours. My only problem so far is to detect when it breaks so that I can restart it, and this must be done before I start decoding the received string of data.
At the moment, my decoder just crashes when the connection is lost, something I expect is because the received string begins to contain garbage at the end. :p
-
Is there a simple call I can do to check if the connection is working properly, or should I make a check on the received string to make sure it is Ok? Maybe just check the length, and if the length is kind of Ok, then check the end of the string.  If it ends like a JSON-text, then OK, and discard string and restart connection if the end of the text is weird.
TinyPortal © 2005-2018