Recent

Author Topic: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32  (Read 13619 times)

MortenB

  • Jr. Member
  • **
  • Posts: 59
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #15 on: March 09, 2021, 07:45:44 am »
There are a few issues, yes.
I have tried and tried various approaches to achieve a connection and not get the SIGSEGV-error.
In the end, I installed a Chrome Addon with Direct Web Sockets Functionality:
  https://chrome.google.com/webstore/detail/smart-websocket-client/omalebghpgejjiaoknljcfmglgbpocdp/related
This to test the address that I try to use via Lazarus.
-
First things first.
The Chrome Addon, with the address:
  wss://stream.binance.com:9443/ws/btcusdt@ticker
connects very well and instantly starts receiving  the tickers in JSON-format
{
  "e": "24hrTicker",
  "E": 1615268894603,
  "s": "BTCUSDT",
  "p": "3394.80000000",
  "P": "6.697",
  "w": "51548.22027742",
  "x": "50688.12000000",       ... and so on ...

However.
When I try to do the same with your WebSocket-implementation, I cannot get it to work.
Code: Pascal  [Select][+][-]
  1.     MyClient := TWebsocketClient.Create('wss://stream.binance.com',9443,'/ws/btcusdt@ticker');
  2.  
does not produce an error, but the next step:
Code: Pascal  [Select][+][-]
  1. MyCommunicator := MyClient.Connect(TSSLSocketHandler.GetDefaultHandler);
gives a Debugger Exception Notification: Project xxx raised exception class 'ESocketError' with message: Host name resolution for "wss://stream.binance.com" failed.
It basically failes at line 74: ASocket.Connect; in the WebsocketsClient-file
-
I have performed a nslookup from a command window:
nslookup wss://stream.binance.com
and the answer is 52.198.117.1 and 54.65.212.75
-This is the exact same address as ASocket.Connect cannot resolve.

Unfortunately, from there and on, I am in deep waters,

More things I have tried.
Deactivated my AntiVirus.
Deactivated Windows Firewall.
Neither of which changes the outcome.






Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #16 on: March 09, 2021, 10:01:24 am »
Have you tried to not write "wss://" in front of the hostname?
i.e. TWebsocketClient.Create('stream.binance.com',9443,'/ws/btcusdt@ticker')

The reason for this is simple, the wss:// is not part of the host, but is part of the URI. But, this library doe not work with URIs

MortenB

  • Jr. Member
  • **
  • Posts: 59
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #17 on: March 09, 2021, 10:16:49 am »
I have tried just about everything, also leaving out the "wss://"
When leaving out "wss://" it ends up with external SIGSEGV in wsstream.pas at line 645 as earlier mentioned.
-
Sad. I need to work with URIs.
Do you have any "immediate" plans to include this in your library?

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #18 on: March 09, 2021, 11:07:40 am »
I've found the problem. The biance server used a lowercase value for the connection header, which is not explicitly allowed by the HTTP standard (but is by the websockets standard), and therefore was not implemented by me that way.
I updated the code to allow for this. Now the following example works completely fine:
Code: Pascal  [Select][+][-]
  1. uses
  2.   Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls, ExtCtrls,
  3.   opensslsockets, sslsockets, WebsocketsClient, wsstream, wsmessages;
  4.  
  5. type
  6.  
  7.   { TForm1 }
  8.  
  9.   TForm1 = class(TForm)
  10.     Memo1: TMemo;
  11.     Timer1: TTimer;
  12.     procedure FormCreate(Sender: TObject);
  13.     procedure FormDestroy(Sender: TObject);
  14.     procedure Timer1Timer(Sender: TObject);
  15.   private
  16.     FClient: TWebsocketClient;
  17.     FStream: TWebsocketCommunincator;
  18.   public
  19.  
  20.   end;
  21.  
  22. var
  23.   Form1: TForm1;
  24.  
  25. implementation
  26.  
  27. {$R *.lfm}
  28.  
  29. { TForm1 }
  30.  
  31. procedure TForm1.FormCreate(Sender: TObject);
  32. begin
  33.   FClient := TWebsocketClient.Create('stream.binance.com',9443,'/ws/btcusdt@ticker');
  34.   FStream := FClient.Connect(TSSLSocketHandler.GetDefaultHandler);
  35.   FStream.StartReceiveMessageThread;
  36. end;
  37.  
  38. procedure TForm1.FormDestroy(Sender: TObject);
  39. begin
  40.   FStream.Free;
  41.   FClient.Free;
  42. end;
  43.  
  44. procedure TForm1.Timer1Timer(Sender: TObject);
  45. var
  46.   MsgList: TWebsocketMessageOwnerList;
  47.   Message: TWebsocketMessage;
  48. begin
  49.   MsgList := TWebsocketMessageOwnerList.Create;
  50.   try
  51.     FStream.GetUnprocessedMessages(MsgList);
  52.     for Message in MsgList do
  53.     begin
  54.       if not (Message is TWebsocketStringMessage) then
  55.         Continue;
  56.       Memo1.Lines.Add(TWebsocketStringMessage(Message).Data);
  57.     end;
  58.   finally
  59.     MsgList.Free;
  60.   end;
  61. end;

PS: HTTP does not know about URIs so from a technical standpoint including wss:// does not make any sense. This is just so general purpose applications like JS can distinquish between websocket and HTTP URIs, but if you start instanciating a websocket client, it should already be pretty obvious that this is not HTTP. Also the second s in wss means using SSL (i.e. a TSSLSocketHandler), so if you have a ws:// uri you don't need ssl, if you have wss:// you need ssl. But this is all handled seperately in the code, so there is no need for URIs
« Last Edit: March 09, 2021, 11:13:06 am by Warfley »

MortenB

  • Jr. Member
  • **
  • Posts: 59
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #19 on: March 09, 2021, 05:41:34 pm »
I updated the code to allow for this. Now the following example works completely fine:
Code: Pascal  [Select][+][-]
  1. uses
  2. procedure TForm1.FormCreate(Sender: TObject);
  3. begin
  4.   FClient := TWebsocketClient.Create('stream.binance.com',9443,'/ws/btcusdt@ticker');
  5.   FStream := FClient.Connect(TSSLSocketHandler.GetDefaultHandler);
  6.   FStream.StartReceiveMessageThread;
  7. end;
  8.  

Thank you for your quick reply and update... but alas. No success on my end.
There is still a problem when the third line:   FStream.StartReceiveMessageThread;   is executed.
Then I get the errormessage: Project xxx raised exception class 'EWebsocketReadError' with message: error reading from stream... in file wsstream.pas at line 466: Stream.LastError);

However.
If I do a step by step approach, and step into the lines, then the connection is Ok. The stream gets a few seconds to "fill up", and when the timer kicks in, the Memo is filled with Binance Ticker-data.
And then... when the stream is empty, it crashes: At least that is what I had here a couple of times, but I can't even reproduce this now.


And again. I had an Idea.
Code: Pascal  [Select][+][-]
  1. function TWebsocketCommunincator.ReceiveMessage: TWebsocketMessage;
  2.  
  3.   procedure ReadData(var buffer; const len: int64);
  4.   var
  5.     ToRead: longint;
  6.     Read: longint;
  7.     LeftToRead: int64;
  8.     TotalRead: int64;
  9.     oldTO: integer;
  10.     Stream: TSocketStream;
  11.   const
  12.     IOTimeoutError = {$IFDEF UNIX}11{$ELSE}10060{$EndIf};
  13.     WaitingTime = 10;
  14.   begin
  15.     Sleep(1000);
  16.     TotalRead := 0;
  17.     repeat
  18.       // how much we are trying to read at a time
  19.       LeftToRead := len - TotalRead;
  20.       if LeftToRead > ToRead.MaxValue then
  21.         ToRead := ToRead.MaxValue
  22.       else
  23.         ToRead := LeftToRead;
  24.       // Reading
  25.  
  26.       Stream := FStream.LockRead;
  27.       try
  28.         if not Assigned(Stream) then
  29.         begin
  30.           raise EWebsocketReadError.Create('Socket already closed', 0);
  31.         end;
  32.         oldTO := Stream.IOTimeout;
  33.         Stream.IOTimeout := 1;
  34.         try
  35.           Read := Stream.Read(PByte(@buffer)[TotalRead], ToRead);
  36.           if Read < 0 then
  37.           begin
  38.             // on Error
  39.             if Stream.LastError <> IOTimeoutError then
  40.               raise EWebsocketReadError.Create('error reading from stream',
  41.                 Stream.LastError);
  42.           end
  43.           else
  44.           begin
  45.             // Increase the amount to read
  46.             TotalRead += Read;
  47.           end;
  48.         finally
  49.           Stream.IOTimeout := oldTO;
  50.         end;
  51.       finally
  52.         FStream.UnlockRead;
  53.       end;
  54.       if (TotalRead < len) and (Read <> ToRead) then // not finished, wait for some data
  55.         Sleep(WaitingTime);
  56.     until TotalRead >= len;
  57.   end;
As you notice, I added Sleep(1000) in your ReceiveMessage-function.
As a less than elegant solution, this solves the problem, and I can now listen to the stream. No more errors...
I really would like more control over the error-situation.
How about replacing the Raise EWebsocketReadError with simply a message? like: "!No data yet!", and that way, when I look for the JSON-strings, I can just avoid the "!No data yet!"-strings, or even count them, and if too many occur on a row, then I can close and restart the connection automatically.
I also see that you have a Stream.IOTimeout=1 in there. How about changing this so that this is a property that can be changed, but that it defaults to 1.
Not sure what 1 is, maybe 1 millisecond.


Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #20 on: March 09, 2021, 10:22:21 pm »
The thing with IOTimeout is that this is intendet to be cought:
Code: Pascal  [Select][+][-]
  1.             if Stream.LastError <> IOTimeoutError then
  2.               raise EWebsocketReadError.Create('error reading from stream',
  3.                 Stream.LastError);
The idea is, as this function acquires some locks, to only read what data is there and if there was a timeout to simply wait a little bit and try again, to not block all the locks until the whole data has been read. So if no data is there yet, it should behave exactly as you want it to, it simply waits a bit and tries again. 1 is the minimum timeout (1 ms) so it basically just reads  what the OS has buffered and does not wait for anything

Could you check out what the actual value of Stream.LastError is and what OS you are using (these error codes are OS specific, e.g. 10060 on windows is the timeout error code).
« Last Edit: March 09, 2021, 10:23:54 pm by Warfley »

MortenB

  • Jr. Member
  • **
  • Posts: 59
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #21 on: March 10, 2021, 03:32:53 am »
Not being able to directly see the error-code from the Stream.LastError, I changed the code slightly:
I added an extra variable.
Code: Pascal  [Select][+][-]
  1. var
  2.   ErrorNumber : LongInt;

and then:
Code: Pascal  [Select][+][-]
  1.             if Stream.LastError <> IOTimeoutError then BEGIN
  2.               ErrorNumber := Stream.LastError;
  3.               raise EWebsocketReadError.Create('error reading from stream',Stream.LastError);
  4.             END;
  5.  
After the break or raised error:
When I then mouse-over IOTimeoutError, I get the value 10060, just to show that the variable values are showing when debugging.
The same mouse-over on ErrorNumber, gives 0 or Zero.

My Os is Windows 10

Jurassic Pork

  • Hero Member
  • *****
  • Posts: 1228
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #22 on: March 10, 2021, 07:27:23 am »
hello,
with this code in the function TWebsocketCommunincator.ReceiveMessage of the file wsstream.pas, it seems to be OK :
Code: Pascal  [Select][+][-]
  1.        try
  2.           Read := Stream.Read(PByte(@buffer)[TotalRead], ToRead);
  3.           if Read < 0 then
  4.           begin
  5.             // on Error
  6.              if (Stream.LastError <> IOTimeoutError) and
  7.                 (Stream.LastError <> 0) then
  8.                raise EWebsocketReadError.Create('error reading from stream',
  9.                      Stream.LastError);
  10.           end
  11.           else
  12.           begin
  13.             // Increase the amount to read
  14.             TotalRead += Read;
  15.           end;
  16.         finally
  17.           Stream.IOTimeout := oldTO;
  18.         end;

Windows 10 Lazarus 2.0.12 fpc 3.2.0

friendly, J.P
« Last Edit: March 10, 2021, 08:33:17 am by Jurassic Pork »
Jurassic computer : Sinclair ZX81 - Zilog Z80A à 3,25 MHz - RAM 1 Ko - ROM 8 Ko

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #23 on: March 10, 2021, 04:23:08 pm »
Well, this is really weird, because error number 0 means there was no error, maybe some weird quirk of the windows api, but alas, I added the (error <> 0) condition as it seems to not break anything

MortenB

  • Jr. Member
  • **
  • Posts: 59
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #24 on: March 10, 2021, 06:02:18 pm »
Great. I was going to add that myself here, then I saw The friendly "Pork" here suggest it too (Thank you Pork, much appreciated), but I wanted to wait for your judgement on this too O:-)

I have not had time to check yet how it works here, but if it works, then I believe I can celebrate a little.

Thank you very much.

My next step is to make REST-calls using GET and/or PUT, but I suppose that should be in a different thread altogether.
Anyway.
Thank you very much again!

MortenB

  • Jr. Member
  • **
  • Posts: 59
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #25 on: March 13, 2021, 07:28:59 am »
 :D Everything here works very well now.
All tested at my side now and I see no problems with the reception of streams anymore.

Thank you again for your great help 8-) O:-)

MortenB

  • Jr. Member
  • **
  • Posts: 59
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #26 on: April 13, 2021, 05:29:20 pm »
After testing a lot with the WebSocket-solution, I was trying to ramp up the size of receiving text.
And this... not functional in a meaningful manner.
I cannot say exactly where it goes wrong (sizewise), but text in the order of half a megabyte seems to be not possible.
The file/unit "wsstream" constantly reports RunError 201 at line 563:
Code: Pascal  [Select][+][-]
  1.         if Header.PayloadLen < 126 then
  2.           len := Header.PayloadLen
  3.         else if Header.PayloadLen = 126 then
  4.         begin
  5.           ReadData(len16, SizeOf(len16));
  6.           len := NToHs(len16);
  7.         end
  8.         else
  9.         begin
  10.           ReadData(len64, SizeOf(len64));
  11.           len := ntohll(len64);   // Recurring errors here when the size is "substantial".
  12.           // Always RunError 201
  13.         end;
  14.         if Header.Mask then
  15.         begin
  16.           ReadData(MaskRec.Key, SizeOf(MaskRec.Key));
  17.         end
When I break and check the value of "len", some times it is "0" = Zero
Other times it is like "8442272860644245284" (Actual example copied out from the "Watches"-window.)
Sometimes my program runs for 1 second after I enable the websockets, other times it just crashes right away.
When it runs for 1 second, I do receive ONE String.
During debug I have found the size to be around 303642–306021 bytes, but most often it just crashes right away.

The line
Code: Pascal  [Select][+][-]
  1. ReadData(len64, SizeOf(len64));
During debug I can find that the value of len64 is -3392894971223736320

I took my chances, and thought that Maybe, the len64 was declared wrongly in the VAR-section.
Code: Pascal  [Select][+][-]
  1. var
  2.   Header: TWebsocketFrameHeader;
  3. //  len64: int64;
  4.   len64: QWord; // Mortens attempt 13.4.2021 to fix error below at line 563
  5.   len16: word;
  6.   len: int64;
  7.   MaskRec: TMaskRec;
  8.   buffer: TBytes;

I changed the int64 to QWord, and Voila.
Now I can receive the huge chunks of texts, seemingly without errors.
I have NO IDEA if my change might cause different problems, but this solved my immediate problem. :D :D :D
Maybe Warfley has a comment on my fix?
Is it an Ok fix? or should the problem be fixed elsewhere?
Thanks in advance  8) 8) 8)

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #27 on: April 14, 2021, 10:42:17 am »
The reason the len64 is negative is because it is in network byte order, not host byte order. On an x86(_64) system this means the bytes need to be swapped, this is what ntohll (network to host long long) is for.

But the runtime error is really weird, are rangechecks enabled? Thats probably the reason, but in any case, using QWord is probably the best solution here, as the sign is not required anyway prior to conversion.

It would be nice if you could put this change into a pull request for the repository, or at least open up an issue so I can fix this in the repository myself later (just so I don't forgett).

MortenB

  • Jr. Member
  • **
  • Posts: 59
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #28 on: April 14, 2021, 01:03:45 pm »
So many things I don't know about network standards and communication things there :(
Great that QWord is an acceptable solution.

As for the runtime error, yes. RangeChecks are enabled in my setup for my project.
I am doing a bit of experimenting with types, pointers and variables, so safer for me that way.
Working on data in many dimensions, and one error and everything goes wrong :p

Fortunately, your websocket-solution seems to be handling everything I need now.
What I need next is
  • To identify when a connection is closed from the other end.
  • To identify a Ping
  • To answer a Ping with a Pong

For my need, I can also send Pong without it being requested. I just need to learn how to do it.
This to prevent the broadcaster from closing the connection, thinking that it is not in use anymore.

Also. One last question.
Regarding speed.
How much data/text should I be able to receive per second through this solution? Is it possible to make an estimate here, or is this completely dependent on the Internet-connection and computer hardware?
Thanks a lot!
Morten.

MortenB

  • Jr. Member
  • **
  • Posts: 59
Re: Using WebSockets with Lazarus-2.0.12-fpc-3.2.0-win32
« Reply #29 on: April 14, 2021, 01:34:28 pm »
Not sure how a pull-request works in the repository, so I made an issue there. Such a minor one :D
Thanks again :D

 

TinyPortal © 2005-2018