Recent

Author Topic: Using TFPHttpClient on multiple threads  (Read 1428 times)

Frogfather

  • New Member
  • *
  • Posts: 26
Using TFPHttpClient on multiple threads
« on: September 17, 2021, 04:13:27 pm »
Hello,

I'm having a play around with TFPHttpClient and TThread in order to get a better understanding of how these components work together. I used https://forum.lazarus.freepascal.org/index.php?topic=40773.0 as a starting point although that post is a few years old and I'm aware that we should use opensslsockets instead.

So, following the post above I have a class TMyHttpRequest:

Code: Pascal  [Select][+][-]
  1. type
  2.  
  3. uses
  4.   Classes, SysUtils, fphttpclient, opensslsockets;        
  5.  
  6.   { TMyHTTPRequest }
  7.  
  8.   TMyHTTPRequest = class(TThread)
  9.   private
  10.     fApi: String;
  11.     fRes: String;
  12.   public
  13.     procedure Execute; override;
  14.   property api: string read fApi write fApi;
  15.   property res: string read fRes write fRes;
  16.   end;
  17.  
  18. implementation
  19.  
  20. { TMyHTTPRequest }
  21.  
  22. procedure TMyHTTPRequest.Execute;
  23. var
  24.   Client: TFPHTTPClient;
  25. begin
  26.   Client := TFPHTTPClient.Create(nil);
  27.   writeln(api);
  28.   res := Client.Get(api);
  29.   writeln(res);
  30.   FreeAndNil(Client);
  31. end;                          
  32.  

and the idea is that the main program can create multiple instances of these and they'll go off and do their thing in separate threads.

Inevitably the calling method is a button click on a form:

Code: Pascal  [Select][+][-]
  1. procedure TForm1.Button1Click(Sender: TObject);
  2. var
  3.   MYThread: array[0..2] of TMyHTTPRequest;
  4.   i: integer;
  5. begin
  6.    for i:=1 to 3 do
  7.   begin
  8.     MYThread[i] := TMyHTTPRequest.Create(True);
  9.     MyThread[i].api:='https://jsonplaceholder.typicode.com/posts/'+inttostr(i);
  10.     MYThread[i].FreeOnTerminate := True;
  11.     MYThread[i].Start;
  12.   end;
  13. end;    
  14.  

So, the first thread works as expected - the call is made and data is returned correctly, but the second and third result in an EInOutError 'could not initialize openssl library'.

If I put sleep(1000) in the loop, giving each thread time to complete before starting the next it works fine. So that suggests that it isn't possible to initialize openssl when it's already in use.

So I'm wondering what is the correct way to make multiple https calls on separate threads?

Thanks in anticipation.
John C

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 757
  • Professional amateur ;-P
Re: Using TFPHttpClient on multiple threads
« Reply #1 on: September 17, 2021, 04:25:46 pm »
Hey John C,

I haven't tried your code, but if you pay attention to the thread you mention, the OP mentions that he solved the issue with:
Code: Pascal  [Select][+][-]
  1. InitSSLInterface;

Before he instantiates any of the threads.

I don't see that in your code.
Could that be the solution?

Again, I'm sorry if I misspoke, since I haven't tested you code, and I'm basing this reply on the solution that the OP on that post got to.

Hope it helps in any way!!

Cheers,
Gus
Lazarus 2.3.0(trunk) FPC 3.3.1(trunk) Ubuntu 21.04 64b Dark Theme
Lazarus 2.0.12(stable) FPC 3.2.2(stable) Ubuntu 21.04 64b Dark Theme
http://github.com/gcarreno

Frogfather

  • New Member
  • *
  • Posts: 26
Re: Using TFPHttpClient on multiple threads
« Reply #2 on: September 17, 2021, 11:39:18 pm »
Hi Gus,
Thanks for the quick reply.

I did try InitSSLInterface without success - same error resulted. Forgot to mention that in my first post.

The OP was using the older method without opensslsockets so maybe it worked using that approach. It does seem to be something to do with only one thread being able to be initialize the openssl library at a time but I don't know enough about its inner workings to work out why that is happening.

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 757
  • Professional amateur ;-P
Re: Using TFPHttpClient on multiple threads
« Reply #3 on: September 17, 2021, 11:48:27 pm »
Hey John,

Thanks for the quick reply.

You're more than welcome!!

I did try InitSSLInterface without success - same error resulted. Forgot to mention that in my first post.

Humm, I thought that would solve the problem, sorry about that!

The OP was using the older method without opensslsockets so maybe it worked using that approach. It does seem to be something to do with only one thread being able to be initialize the openssl library at a time but I don't know enough about its inner workings to work out why that is happening.

Well, this exhaust my knowledge on the matter :(
Sorry I can't be of any further assistance :(

But I do feel that it's a bit odd that the SSL library is not thread safe, or for that matter, is thread aware and messes things up or something.
Quite an interesting conundrum, I'll say!!

Cheers,
Gus
Lazarus 2.3.0(trunk) FPC 3.3.1(trunk) Ubuntu 21.04 64b Dark Theme
Lazarus 2.0.12(stable) FPC 3.2.2(stable) Ubuntu 21.04 64b Dark Theme
http://github.com/gcarreno

Frogfather

  • New Member
  • *
  • Posts: 26
Re: Using TFPHttpClient on multiple threads
« Reply #4 on: September 25, 2021, 12:31:43 am »
Hi Gus,

From the openssl docs:

Quote
OpenSSL can generally be used safely in multi-threaded applications provided that at least two callback functions are set, the locking_function and threadid_func. Note that OpenSSL is not completely thread-safe, and unfortunately not all global resources have the necessary locks.

So it looks like it should be possible - I'll do a bit more investigation and see if I get anywhere.

dbannon

  • Hero Member
  • *****
  • Posts: 1598
    • tomboy-ng, a rewrite of the classic Tomboy
Re: Using TFPHttpClient on multiple threads
« Reply #5 on: September 25, 2021, 02:45:50 am »

Quote
OpenSSL can generally be used safely in multi-threaded applications provided that at least two callback functions are set, the locking_function and threadid_func. Note that OpenSSL is not completely thread-safe, and unfortunately not all global resources have the necessary locks.

Those two sentences appear to be self contradictory.  Its really not enough to have most of something threadsafe. Unless you can determine, reliably, what what calls you can make and be absolutly sure you will not tickle any of the unsafe bits, its not thread safe.

Davo

Lazarus 2, Linux (and reluctantly Win10, OSX)
My Project - https://github.com/tomboy-notes/tomboy-ng

GetMem

  • Hero Member
  • *****
  • Posts: 3492
Re: Using TFPHttpClient on multiple threads
« Reply #6 on: September 25, 2021, 09:47:14 am »
@Frogfather
Writeln is not thread safe, use the Synchronize or Queue methods to update visual controls in the main thread. In your case, since multiple threads write to console at the same time, you should use critical sections to prevent race condition and/or deadlock.

PS: Please do a quick forum search, there are many examples on how to use critical sections.
« Last Edit: September 25, 2021, 10:03:08 am by GetMem »

Frogfather

  • New Member
  • *
  • Posts: 26
Re: Using TFPHttpClient on multiple threads
« Reply #7 on: September 25, 2021, 09:47:59 am »
Hi Davo,

I've just re-read the docs now that I'm slightly more awake and yes, you are of course correct. My knowledge of multi-threaded applications is not great hence my interest in learning.

I didn't need this for any project - I was just curious to see if I could do something similar to a js promise in fpc.

If you know of any solution to this - either out of box or 3rd party then I'd be most grateful.

Frogfather

  • New Member
  • *
  • Posts: 26
Re: Using TFPHttpClient on multiple threads
« Reply #8 on: September 25, 2021, 10:10:57 am »
Hi GetMem,

Thanks for that - I used the article at https://wiki.freepascal.org/Multithreaded_Application_Tutorial as a starting point and it uses writeln hence my use of it.

As mentioned elsewhere this is just out of interest. I used Object Pascal (Delphi/Kylix) for many years before I was a professional developer and it's interesting to revisit it with slightly more experienced eyes!

PascalDragon

  • Hero Member
  • *****
  • Posts: 3403
  • Compiler Developer
Re: Using TFPHttpClient on multiple threads
« Reply #9 on: September 27, 2021, 08:55:35 am »
Writeln is not thread safe, use the Synchronize or Queue methods to update visual controls in the main thread. In your case, since multiple threads write to console at the same time, you should use critical sections to prevent race condition and/or deadlock.

WriteLn to the console (or even a file) is thread safe in the sense that there won't be any deadlock or race condition. What can happen however is garbled output if one thread is interrupted in the middle of flushing its buffer with another thread then writing.

 

TinyPortal © 2005-2018