Recent

Author Topic: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32  (Read 6694 times)

MortenB

  • Jr. Member
  • **
  • Posts: 52
Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« on: March 02, 2021, 04:53:28 am »
I would like to subscribe to certain feeds of data, and then display this data on my computer,
What I really would like now is to make an app that listens to the Binance Price-ticker, and simply displays the numbers, or everything which is received.

Does anybody have information regarding which WebSocket addon I would need, and an example of how to set this up?

I need to start a subscription.
Then I need to be able to read the returned stream.
And then close the subscription.
This just to get started. I learn best from relevant examples.

Thank you in advance.

Jurassic Pork

  • Hero Member
  • *****
  • Posts: 962
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #1 on: March 02, 2021, 06:57:48 am »
hello,
you can try lazWebsockets !

Friendly, J.P
Jurassic computer : Sinclair ZX81 - Zilog Z80A à 3,25 MHz - RAM 1 Ko - ROM 8 Ko

MortenB

  • Jr. Member
  • **
  • Posts: 52
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #2 on: March 02, 2021, 08:36:32 am »
I have installed this Lazarus WebSocket-solution.
The problem is a total lack of guides as in how to use it against various providers of data one can subscribe to.
I find no real examples of anything for this, except the provided example of a chat server and client.

Warfley

  • Sr. Member
  • ****
  • Posts: 413
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #3 on: March 02, 2021, 10:33:16 pm »
The library does only implement the base websocket protocol. Your subscription system must use some form of sub protocol on top of websockets, this is up to you to implement. The library just gives you the networking capabilities to establish streams and send and recieve messages.

The service you are subscribing to needs to have some form of documentation. If you find that, maybe I can help you with specific questions.

PierceNg

  • Jr. Member
  • **
  • Posts: 88
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #4 on: March 03, 2021, 01:21:06 am »
The library does only implement the base websocket protocol. Your subscription system must use some form of sub protocol on top of websockets, this is up to you to implement. The library just gives you the networking capabilities to establish streams and send and recieve messages.

The service you are subscribing to needs to have some form of documentation. If you find that, maybe I can help you with specific questions.

OP mentioned Binance, so possibly this one: https://github.com/binance/binance-spot-api-docs/blob/master/web-socket-streams.md

MortenB

  • Jr. Member
  • **
  • Posts: 52
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #5 on: March 03, 2021, 08:36:30 am »
Thank you for your time and effort in this matter.
And yes.
I did mention Binance, and of course there is a lot of things I read from the Binance API.
Everything there seems to be in Joson-format.
And... making the Joson-queries and reading the Joson-answers is a piece of cake. I have no problems there, and I believe the API is well enough documented for me to work with that.
-
However.
What I have never done before is established a subscription.
Or sent any messages at all via the Internet.
What I really would like is:
1. An example of how to send messages/streams/strings to the service provider
2. An example of how to know if I received a reply, errormessage or anything that I can interpret.
3. An example of how to read the received stream.
-
In my simple mind, I consider the workflow to be like:
1. Subscribe to DataStream
2. Check for errormessage.
3. If no errormessage, then check if new data is available on/in the stream 3–4 times every second.
4. If data is available, read and display information.
and then just cycle 3 and 4.
-
-
It doesn't have to be using the Lazarus WebSockets.
I also see that Indy has all the functionality for WebSockets, among other things, and from what I read, this Indy is now very very stable. Indy is maybe the better choice?

Warfley

  • Sr. Member
  • ****
  • Posts: 413
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #6 on: March 03, 2021, 02:20:45 pm »
Well, the chatClient example should contain all the information you need, but let me break it down for you.

First you need to open up the websocket stream, represented by the TWebsocketCommunincator class:
Code: Pascal  [Select][+][-]
  1. client := TWebsocketClient.Create(address, port, path); // e.g. for wss://example.com:8080/foo?bar=foobar address would be example.com port would be 8080 (default 80) and path would be /foo?bar=foobar (default /)
  2.   try
  3.     wsstream := client.Connect(TSocketHandler.Create); // Use an SSL socket handler for SSL, there are examples in this forum how to do this
  4.     try
  5.       ...
  6.     finally
  7.       wsstream.Free;
  8.     end;
  9.   finally
  10.     client.Free;
  11.   end;
Next step is to send a json string:
Code: Pascal  [Select][+][-]
  1. wsstream.WriteStringMessage(jsonString);
For recieving messages just call:
Code: Pascal  [Select][+][-]
  1. wsstream.RecieveMessage;
This function will block until a message was read. Messages read by it will be stored in the thread safe wsstream.Messages datastructure. To read all recieved messages call "GetUnprocessedMessages".

So if you expect exactly one response message and you want to wait for it blocking:
Code: Pascal  [Select][+][-]
  1. function RecieveResponse(wsstream: TWebsocketCommunicator): String;
  2. var
  3.   msgList: TWebsocketMessageOwnerList;
  4.   msg: TWebsocketMessage;
  5. begin
  6.   Result := nil;
  7.   msgList := TWebsocketMessageOwnerList.Create;
  8.   try
  9.     while wsstream.Open and not Assigned(Result) do
  10.     begin
  11.       wsstream.Recievemessage;
  12.       msgList.Clear;
  13.       if wsstream.GetUnprocessedMessages(msgList) > 0 then
  14.         for msg in msgList do
  15.           if msg is TWebsocketStringMessage then
  16.             Exit(TWebsocketStringMessage(msg).Data);
  17.     end;
  18.   finally
  19.     msgList.Free;
  20.   end;
  21. end;
This function will wait for and recieve the very first response that is a string and discards all other messages (binary messages or pong messages)

If you want to recieve messages asynchonously, i.e. you don't want to explicitly wait for messages, you can do so by setting the OnRecieveMessage event handler and execute Recievemessage in a seperate thread:
Code: Pascal  [Select][+][-]
  1.   TRecieverThread = class(TThread)
  2.   private
  3.     FCommunicator: TWebsocketCommunincator;
  4.   protected
  5.     procedure Execute; override;
  6.   public
  7.     constructor Create(ACommunicator: TWebsocketCommunincator);
  8.   end;
  9.  
  10. ...
  11.  
  12. procedure TRecieverThread.Execute;
  13. begin
  14.   while not Terminated and FCommunicator.Open do
  15.   begin
  16.     FCommunicator.RecieveMessage;
  17.     Sleep(100);
  18.   end;
  19. end;
  20.  
  21. constructor TRecieverThread.Create(ACommunicator: TWebsocketCommunincator);
  22. begin
  23.   FCommunicator := ACommunicator;
  24.   inherited Create(True);
  25.   FreeOnTerminate := True;
  26. end;
  27.  
  28. ...
  29.   wsstream.OnRecieveMessage := @MessageHandler;
  30.   TRecieverThread.Create(wsstream).Start;
Now the thread will automatically check for messages in the background and when one is recieved, MessageHandler will be called. BUT: MessageHandler will be called from within the context of the Reciever thread. So if you want to, for example, display the message in your GUI, you need to use synchronize or TThread.Queue:
Code: Pascal  [Select][+][-]
  1. TMyClass = class
  2.   LatestMessage: String;
  3.   procedure UpdateUI;
  4.   procedure MessageHandler(Sender: TObject);
  5.  
  6. ...
  7.  
  8. procedure TMyClass.UpdateUI;
  9. begin
  10.   Label1.Caption := LatestMessage;
  11. end;
  12.  
  13. procedure TMyClass.MessageHandler(Sender: TObject);
  14. begin
  15. var
  16.   MsgList: TWebsocketMessageOwnerList;
  17.   m: TWebsocketMessage;
  18. begin
  19.   LatestMessage := nil;
  20.   MsgList := TWebsocketMessageOwnerList.Create(True);
  21.   try
  22.     TWebsocketCommunicator(Sender).GetUnprocessedMessages(MsgList);
  23.     for m in MsgList do
  24.       if m is TWebsocketStringMessage then
  25.         LatestMessage := TWebsocketStringMessage(m).Data;
  26.   finally
  27.     MsgList.Free;
  28.   end;
  29.   if Assigned(LatestMessage) then
  30.     Synchronize(@UpdateUI);
  31. end;
TMyClass can for example be your TForm derivate you are working in, if you are working in a GUI application.

MortenB

  • Jr. Member
  • **
  • Posts: 52
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #7 on: March 03, 2021, 03:31:21 pm »
Well, the chatClient example should contain all the information you need, but let me break it down for you.

Thank you so much for the "breakdown"  :D
I think I can handle this pretty well.

It is too late for me tonight, but if I have time, I will try to do something tomorrow.
Ideally, what I would like to do is to make my form wait for input from mouse and keyboard at all times.
a timer will be set to check for messages maybe 10 times per second (just an example.)
Then if the timed "interrupt" find received message(s), they will be dealt with before letting go.

Your "breakdown" of this is great. If I encounter problems, I will try to ask for more details.
Thank you!

Warfley

  • Sr. Member
  • ****
  • Posts: 413
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #8 on: March 03, 2021, 04:10:50 pm »
After writing this post I noticed that this is a little bit overcomplicated, so I just added some new functions to the library that makes most of what I wrote previously invalid.

You now don't need to call RecieveMessage anymore. If you are waiting for a message and want to block until you have it, you can now use WaitForMessage or if you are awaiting a special kind of message WaitForStringMessage, WaitForBinaryMessage and WaitForPongMessage. In your case, awaiting the response will therefore look something like this:
Code: Pascal  [Select][+][-]
  1. msg := wsstream.WaitForStringMessage;
  2. try
  3.   if not Assigned(msg) then // stream closed (disconnect, server closed stream, etc.) before message was recieved
  4.     // Error handling
  5.   jsonString := msg.Data;
  6. finally
  7.   msg.Free;
  8. end;
This is very useful if you are expecting a response, e.g. when you do your subscription and need to check if it was successful.

You also now don't need to create your own TThread instance for recieving messages, simply call wsstream.StartRecieveMessageThread and a new thread will be started recieving messages (see the Readme for further information about threading).

Most importantly, WaitForXXXMessage works fine in both situations, if you have a reciever thread and if you don't.

Quote
Then if the timed "interrupt" find received message(s), they will be dealt with before letting go.
This is also quite simple, first start the reciever thread, then do the following in your timer:
Code: Pascal  [Select][+][-]
  1. procedure TForm1.Timer1Timer(Sender: TObject);
  2. var
  3.   Messages: TWebsocketMessageOwnerList;
  4.   Message: TWebsocketMessage;
  5. begin
  6.   Messages := TWebsocketMessageOwnerList.Create;
  7.   try
  8.     wsstream.GetUnprocessedMessages(Messages);
  9.     for Message in Messages do
  10.       // Update your UI with the contents of Message
  11.   finally
  12.     Messages.Free;
  13.   end;
  14. end;

This is also completely thread safe, so you don't need to worry about threading issues (as would be the case when using the event OnRecieveMessage)

bytebites

  • Sr. Member
  • ****
  • Posts: 393
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #9 on: March 04, 2021, 12:26:54 pm »
Recieve should be written as receive.

MortenB

  • Jr. Member
  • **
  • Posts: 52
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #10 on: March 05, 2021, 03:31:34 pm »
First you need to open up the websocket stream, represented by the TWebsocketCommunincator class:
Code: Pascal  [Select][+][-]
  1. client := TWebsocketClient.Create(address, port, path); // e.g. for wss://example.com:8080/foo?bar=foobar address would be example.com port would be 8080 (default 80) and path would be /foo?bar=foobar (default /)
  2.   try
  3.     wsstream := client.Connect(TSocketHandler.Create); // Use an SSL socket handler for SSL, there are examples in this forum how to do this
  4.     try
  5.       ...
  6.     finally
  7.       wsstream.Free;
  8.     end;
  9.   finally
  10.     client.Free;
  11.   end;
I have to admit. SSL Socket handler is also new to me. I have no clue.
Basically what I end up with when I try to compile the line: "wsstream := client.Connect(..."
is an errormessage: Fatal: Syntax error, "." expected but ":=" found.

I started with a button to start subscription
Code: Pascal  [Select][+][-]
  1. client := TWebsocketClient.Create('wss://stream.binance.com', 9443, '/');
and should send a JSON-string to request subscription data when connection has been confirmed.
{
    "type": "subscribe",
    "exchange": "binance",
    "pair": "eth-btc",
    "channel": "trade"
}

My simple idea here was to start a timer that constantly checks if a message is received, and then displays whatever is received in an edit-box or something.

Then a button to stop subscription



Warfley

  • Sr. Member
  • ****
  • Posts: 413
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #11 on: March 05, 2021, 06:05:25 pm »
I have to admit. SSL Socket handler is also new to me. I have no clue.
I think you can simply use the class method "TSSLSocketHandler.GetDefaultHandler" from the unit sslsockets to use ssl
Basically what I end up with when I try to compile the line: "wsstream := client.Connect(..."
is an errormessage: Fatal: Syntax error, "." expected but ":=" found.
Sorry, I am stupid, wsstream is the name of one of my units, so you need to name the variable differently.

I started with a button to start subscription
and should send a JSON-string to request subscription data when connection has been confirmed.
{
    "type": "subscribe",
    "exchange": "binance",
    "pair": "eth-btc",
    "channel": "trade"
}

My simple idea here was to start a timer that constantly checks if a message is received, and then displays whatever is received in an edit-box or something.

Then a button to stop subscription

I recommend the following, put the client and stream variable into you Form (e.g. as private variables). In FormCreate create both variables and set up events:
Code: Pascal  [Select][+][-]
  1. Self.Client := TWebsocketClient.Create('stream.binance.com', 9443, '/');
  2. Self.Stream := Self.Client.Connect(TSSLSocketHandler.GetDefaultHandler);
  3. Self.Stream.OnClose := @HandleClose; // Handle close is a procedure HandleClose(sender: TObject); in your class
  4. Self.Stream.StartReceiveMessageThread; // handle all incoming messages
And free them in the FormDestroy event:
Code: Pascal  [Select][+][-]
  1. Self.Stream.Free;
  2. Self.Client.Free;

Then in your subscribe button you start the subscription by sending your json request and awaiting the correct answer:
Code: Pascal  [Select][+][-]
  1. Self.Stream.WriteStringMessage('JSON subscribe request');
  2. response := Self.Stream.WaitForStringMessage;
  3. try
  4.   // Check if response.Data contains the expected response
  5. finally
  6.   response.Free;
  7. end;
In your unsubscribe button you do the same with the unsubscribe request

In your timer you then put the loop to check for received messages
Code: Pascal  [Select][+][-]
  1.     procedure TForm1.Timer1Timer(Sender: TObject);
  2.     var
  3.       Messages: TWebsocketMessageOwnerList;
  4.       Message: TWebsocketMessage;
  5.     begin
  6.       Messages := TWebsocketMessageOwnerList.Create;
  7.       try
  8.         wsstream.GetUnprocessedMessages(Messages);
  9.         for Message in Messages do
  10.           // Update your UI with the contents of Message
  11.       finally
  12.         Messages.Free;
  13.       end;
  14.     end;

Also note, that I had updated the code today, so I highly recommend you downloading the newest version. Best to load it via git so you can easiely check for updates using git pull

MortenB

  • Jr. Member
  • **
  • Posts: 52
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #12 on: March 07, 2021, 03:05:12 pm »
I have now been back and forth a bit with the code, but there are issues I cannot easily solve here.
First thing first, a tip: "TWebSocketCommunincator", should this not be "TWebSocketCommunicator"? (Just a typo, maybe intended, I don't know.)
-
Then my big problem.
First my code:
Code: Pascal  [Select][+][-]
  1. uses
  2.   Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls, ExtCtrls,
  3.   wsutils,
  4.   wsmessages,
  5.   wsstream,
  6.   ssockets,
  7.   WebsocketsClient;
  8.  
  9. type
  10.  
  11.   { TForm1 }
  12.  
  13.   TForm1 = class(TForm)
  14.     btnSubscribe: TButton;
  15.     btnUnsubscribe: TButton;
  16.     edtPort: TEdit;
  17.     edtSubscribeAddress: TEdit;
  18.     mmoResult: TMemo;
  19.     mmoSubscriptionInfo: TMemo;
  20.     MessageTimer: TTimer;
  21.     procedure btnSubscribeClick(Sender: TObject);
  22.     procedure FormCreate(Sender: TObject);
  23.     procedure FormDestroy(Sender: TObject);
  24.     procedure MessageTimerTimer(Sender: TObject);
  25.   private
  26.     MyClient   : TWebsocketClient;
  27.     MyStream   : TWebSocketCommunincator;//TStream;
  28.     procedure HandleClose(sender: TObject);
  29.   public
  30.   end;
  31.  
  32. var
  33.   Form1: TForm1;
  34.   MyStreamVar : TLockedSocketStream;
  35. implementation
  36.  
  37. {$R *.lfm}
  38.  
  39. { TForm1 }
  40.  
  41. procedure TForm1.HandleClose(Sender: TObject);
  42.   begin
  43.   end;
  44.  
  45. procedure TForm1.FormCreate(Sender: TObject);
  46.   begin
  47.     MyStream.Create(MyStreamVar,False,False);
  48.     MyClient := TWebsocketClient.Create('stream.binance.com', 9443, '/');
  49.     MyStream := MyClient.Connect(TSocketHandler.Create);// GetDefaultHandler
  50.     MyStream.OnClose := @HandleClose; // Handle close is a procedure HandleClose(sender: TObject); in your class
  51.     MyStream.StartReceiveMessageThread; // handle all incoming messages
  52.   end;
  53.  
  54. procedure TForm1.btnSubscribeClick(Sender: TObject);
  55.   VAR
  56.     MyResponse : TWebSocketStringMessage;
  57.     MyTextData : UTF8String;
  58.   begin
  59.     {
  60.     MyTextData:='';
  61.     MyResponse.Create(MyTextData);
  62.     MyStream.WriteStringMessage(mmoSubscriptionInfo.Lines.Text);// 'JSON subscribe request'
  63.     MyResponse := MyStream.WaitForStringMessage;
  64.     try
  65.       // Check if response.Data contains the expected response
  66.     finally
  67.       MyResponse.Free;
  68.     end;
  69.     }
  70.   end;
  71.  
  72. procedure TForm1.FormDestroy(Sender: TObject);
  73.   begin
  74.     MyStream.Free;
  75.     MyClient.Free;
  76.   end;
  77.  
  78.  
  79. procedure TForm1.MessageTimerTimer(Sender: TObject);
  80.   var
  81.     Messages: TWebsocketMessageOwnerList;
  82.     Message: TWebsocketMessage;
  83.   begin
  84.     {
  85.     Messages := TWebsocketMessageOwnerList.Create;
  86.     try
  87.       MyStream.GetUnprocessedMessages(Messages);
  88.       for Message in Messages do
  89.         // Update your UI with the contents of Message
  90.       mmoResult.Append(Message.ToString);
  91.     finally
  92.       Messages.Free;
  93.     end;
  94.     }
  95.   end;
  96. end.
So far, I get no error on compile.
When running, I get SIGSEGV on: MyStream.OnClose := @HandleClose;
When I // don't do this line, I get SIGSEGV on the next line, MyStream.StartReceiveMessageThread;, which then point to line 645 in the wsstream-unit:   if not Assigned(FReceiveMessageThread) then...

There just seems to be an endless list of things popping up here.
Maybe because I don't have the right units, maybe because I don't know how to initialize the variables properly.
I realize that this wasn't a walk in the park :) But I don't want to give up  >:D 8-)

I don't even know if I need to install something for SSL-functionality.
Your example gives: TSSLSocketHandler
But I cannot seem to find this here.



MortenB

  • Jr. Member
  • **
  • Posts: 52
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #13 on: March 08, 2021, 06:43:08 am »
I finally found out about the TSSLSocketHandler.GetDefaultHandler. Hoorayyy. One step further!
Had to add sslsockets and OpenSSLsockets to the Uses  :D
But still. The problem persist when I start the handling of incoming messages: X.StartReceiveMessageThread.
It goes to line 645 in the wsstream-unit and ends with a SIGSEGV
The exact error: Project xxx raised exception class 'External: SIGSEGV'. In file wsstream.pas at line 645: if not Assigned(FReceiveMessageThread) then
-
I hope you can point me in the right direction here.
Thanks in advance.

Warfley

  • Sr. Member
  • ****
  • Posts: 413
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #14 on: March 08, 2021, 04:20:02 pm »
This problem is because the server declined the handshake, in this case Client.Connect will return nil, and when trying to work on that, you get an access violation.

To be precise, the server returns http code "400: Bad request". Probably something I should throw an exception over, is on my todo list. But for now, it seems like you are doing a faulty request. Probably because you try to access the path '/', while the github doc states:
Quote
Raw streams are accessed at /ws/<streamName>
Combined streams are accessed at /stream?streams=<streamName1>/<streamName2>/<streamName3>
so your path is probably something like '/ws/streamname' or '/stream?streams=...'

PS: the line
Code: Pascal  [Select][+][-]
  1. MyStream.Create(MyStreamVar,False,False);
Is completely out of place and should be removed
« Last Edit: March 08, 2021, 04:21:35 pm by Warfley »

 

TinyPortal © 2005-2018