fphttpclient - ESocketError, time out on localhost ElasticSearch

[LeadGuit]

Thank you for the encouragement ;-) Learning the Syntax and Libraries is always a little journey. especially coming from Python with maybe a little too much convenience to to "you can find a package for everything". Hiding the whole logic from you. On the plus side, Pascal let's me feel more like a Programmer, while the stuff I do in Py let's me feel more like a SysAdmin or Data Scientist xD

So I tried out Synapse in a very quick and hacky way

Code: Pascal  [Select][+][-]

1. procedure Tform1.Button1click(Sender: TObject);
2.   var
3.     httpsender: Thttpsend;
4.     Httpgetresult: boolean;
5.     ss: TStringStream;
6.   begin
7.     ss := TSTringStream.Create('');
8.     httpsender := THTTPSend.Create;
9.     HTTPGetResult := HTTPSender.HTTPMethod('GET', 'http://localhost:9200');
10.     ss.CopyFrom(httpsender.document, 0);
11.     Memo1.Lines := (httpsender.Headers) ;
12.     Memo1.Lines.Add(IntToStr(httpsender.ResultCode));
13.     Memo1.Lines.Add(ss.DataString);
14.   end;  

And yeah - it again can't connect to the instance running in WSL, but can easy get on the one running on Windows.
Interesting is the ResultCode 500 I get back:

Code: Pascal  [Select][+][-]

1. GET / HTTP/1.0
2. Host: 127.0.0.1:9200
3. Keep-Alive: 300
4. Connection: keep-alive
5. User-Agent: Mozilla/4.0 (compatible; Synapse)
6.  
7. 500

vs on the working instance:

Code: Pascal  [Select][+][-]

1. HTTP/1.0 200 OK
2. connection: Keep-Alive
3. content-type: application/json; charset=UTF-8
4. content-length: 501
5.  
6. 200
7. {.....

[LeadGuit]

Ok, a bit further:

Setting ES to "production mode" - or rather development mode with a small - totally insecure and not recommended - hack, I can access it through the "Network IP" the WSL2 machine has assigned. Meaning, I expose the ES instance to the whole network, instead of the normal "development mode" which would only expose it to localhost (aka 0.0.0.0 in production vs localhost in development)

WSL2 should forward to localhost on Windows, which in most instances work. For Pascal though, in fphttpclient and synapse, The forward isn't recognized as it seems, instead, the WSL instance is seen as a "separate computer". Which is technically not wrong since it's a lightweight VM, but still is interesting.
Maybe Windows is doing some behind-the-scenes work with HyperV to get that localhost-forward going, which Pascal doesn't have access to (or rather, doesn't use without explicitly stating so)?
I've tested quickly with Python on Windows using the requests library, and as expected the localhost-forward works as expected.

[Gustavo 'Gus' Carreno]

Hey LeadGuit,

Thank you for the encouragement ;-) Learning the Syntax and Libraries is always a little journey. especially coming from Python with maybe a little too much convenience to to "you can find a package for everything". Hiding the whole logic from you. On the plus side, Pascal let's me feel more like a Programmer, while the stuff I do in Py let's me feel more like a SysAdmin or Data Scientist xD

Oh, I so feel you!!

So I tried out Synapse in a very quick and hacky way

Apart from a try...finally block to make sure ss and httpsender get freed, that's not hacky at all. Actually quite on the nose.

And yeah - it again can't connect to the instance running in WSL, but can easy get on the one running on Windows.

So that confirms that there is something in common with fphhtpclient and synapse that doesn't work in FPC land but does in cURL land. How utterly intriguing!!
But hey, synapse is quite old. Wanna try with Indy? One click away on the OPM... Okay, maybe 2...

Interesting is the ResultCode 500 I get back:

That's very intriguing.
I suspect that synapse is probably eating an Exception and then just spitting out a 500.
I think if you debug and enter the code(F7 enter code, F8 go over) and go into the synapse code you'll find out the ESocketError. But not 100% sure. Talking outta me arse, here...

Cheers,
Gus

[Gustavo 'Gus' Carreno]

Hey LeadGuit,

Ok, a bit further:

Setting ES to "production mode" - or rather development mode with a small - totally insecure and not recommended - hack, I can access it through the "Network IP" the WSL2 machine has assigned. Meaning, I expose the ES instance to the whole network, instead of the normal "development mode" which would only expose it to localhost (aka 0.0.0.0 in production vs localhost in development)

WSL2 should forward to localhost on Windows, which in most instances work. For Pascal though, in fphttpclient and synapse, The forward isn't recognized as it seems, instead, the WSL instance is seen as a "separate computer". Which is technically not wrong since it's a lightweight VM, but still is interesting.
Maybe Windows is doing some behind-the-scenes work with HyperV to get that localhost-forward going, which Pascal doesn't have access to (or rather, doesn't use without explicitly stating so)?
I've tested quickly with Python on Windows using the requests library, and as expected the localhost-forward works as expected.

Hummm, so what this is kinda telling me is that python and cURL are initiating sockets in a manner that enable the WSL redirect(let's call it that), to work.
On the other hand, FPC is initiating sockets in such a way that it pretty much effes with the WSL redirection.

Now is it a binding issue? Does FPC bind and cURL not, or vice-versa?
It's not DNS related because the error is ESocketError Timeout, not DNS resolution error.
Is it the diff between 0.0.0.0 and 127.0.0.1?

AHHHHH, you're killing me!! So many questions!!

Cheers,
Gus

[LeadGuit]

Hey LeadGuit,

Ok, a bit further:

Setting ES to "production mode" - or rather development mode with a small - totally insecure and not recommended - hack, I can access it through the "Network IP" the WSL2 machine has assigned. Meaning, I expose the ES instance to the whole network, instead of the normal "development mode" which would only expose it to localhost (aka 0.0.0.0 in production vs localhost in development)

WSL2 should forward to localhost on Windows, which in most instances work. For Pascal though, in fphttpclient and synapse, The forward isn't recognized as it seems, instead, the WSL instance is seen as a "separate computer". Which is technically not wrong since it's a lightweight VM, but still is interesting.
Maybe Windows is doing some behind-the-scenes work with HyperV to get that localhost-forward going, which Pascal doesn't have access to (or rather, doesn't use without explicitly stating so)?
I've tested quickly with Python on Windows using the requests library, and as expected the localhost-forward works as expected.

Hummm, so what this is kinda telling me is that python and cURL are initiating sockets in a manner that enable the WSL redirect(let's call it that), to work.
On the other hand, FPC is initiating sockets in such a way that it pretty much effes with the WSL redirection.

Now is it a binding issue? Does FPC bind and cURL not, or vice-versa?
It's not DNS related because the error is ESocketError Timeout, not DNS resolution error.
Is it the diff between 0.0.0.0 and 127.0.0.1?

AHHHHH, you're killing me!! So many questions!!

Cheers,
Gus

Haha - so my current hypothesis is that really FPC binds differently - but only on ES, since a simple HTTP Server on WSL2 works...or a uvicorn server as well

Code: Pascal  [Select][+][-]

1. HTTP/1.1 200 OK
2. date: Wed, 31 Mar 2021 20:29:33 GMT
3. server: uvicorn
4. content-type: text/html; charset=utf-8
5. Connection: close
6.  
7. 200

It's only the local ES that goes haywire. Which would mean in the end - ES is screwing with the sockets in some way, which only would affect FPC/Synapse.

I tried to do something with indy, but I couldn't find how - I can't see a http component, and every question seems to skip the "use" part and whatnot. And it's getting too late over here to read question thread through and make code work ;-)

[Gustavo 'Gus' Carreno]

Hey LeadGuit,

Haha - so my current hypothesis is that really FPC binds differently - but only on ES, since a simple HTTP Server on WSL2 works...or a uvicorn server as well

Code: Pascal  [Select][+][-]

1. HTTP/1.1 200 OK
2. date: Wed, 31 Mar 2021 20:29:33 GMT
3. server: uvicorn
4. content-type: text/html; charset=utf-8
5. Connection: close
6.  
7. 200

I'm glad you cleared this issue. I was gonna mention you did this exact test(great minds and all that), but I was also pushing for the Indy test, so I didn't wan to seem bully-ish

It's only the local ES that goes haywire. Which would mean in the end - ES is screwing with the sockets in some way, which only would affect FPC/Synapse.

Indeed. But what still baffles me is exactly that: Why is only FPC tripping on that?!?!?

I tried to do something with indy, but I couldn't find how - I can't see a http component, and every question seems to skip the "use" part and whatnot. And it's getting too late over here to read question thread through and make code work ;-)

Yeah, with fphttpclient and synapse there are no visual components. Indy needs a bit of a paradigm shift since it's ALL visual components.
This means looking for the Indiy Clients component tab and finding the HTTP one, dropping on the... Why am I patronising you? You know this by now: Find lill picture on tabs, click on it, click or drag on form, look at object inspector. Or at least you shouldda have mastered it already

And, yes, please go and have a rest. I also need one. I'm finding myself staring into the void and drooling just a bit...

Cheers,
Gus

[sstvmaster]

I have installed elasticsearch from: https://www.elastic.co/de/downloads/elasticsearch
I use Windows 10 and Lazarus 2.0.12

With this code i got response:

Code: Pascal  [Select][+][-]

1. unit Unit1;
2.  
3. {$mode objfpc}{$H+}
4.  
5.  
6. interface
7.  
8. uses
9.   Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls,
10.  
11.   fpHttpClient;
12.  
13. const
14.   HttpErrorMsg = 'Error %d (%s)';
15.  
16. type
17.  
18.   { TForm1 }
19.  
20.   TForm1 = class(TForm)
21.     Button1: TButton;
22.     Memo1: TMemo;
23.     procedure Button1Click(Sender: TObject);
24.   private
25.  
26.   public
27.  
28.   end;
29.  
30. var
31.   Form1: TForm1;
32.  
33. implementation
34.  
35. {$R *.lfm}
36.  
37. { TForm1 }
38.  
39. procedure TForm1.Button1Click(Sender: TObject);
40. var
41.   http: TFPHTTPClient;
42.   Response: TMemoryStream;
43. begin
44.   http := TFPHTTPClient.Create(nil);
45.   Response := TMemoryStream.Create;
46.   try
47.     try
48.       http.Get('http://127.0.0.1:9200', Response);
49.       Response.Seek(0, soFromBeginning);
50.       Memo1.Lines.LoadFromStream(Response);
51.     except
52.       Memo1.Clear;
53.       Memo1.Lines.Add(Format(HttpErrorMsg, [http.ResponseStatusCode,http.ResponseStatusText]));
54.     end;
55.   finally
56.     http.Free;
57.     Response.Free;
58.   end;
59. end;
60.  
61. end.
62.  

[PierceNg]

Haha - so my current hypothesis is that really FPC binds differently - but only on ES, since a simple HTTP Server on WSL2 works...or a uvicorn server as well

Code: Pascal  [Select][+][-]

1. HTTP/1.1 200 OK
2. date: Wed, 31 Mar 2021 20:29:33 GMT
3. server: uvicorn
4. content-type: text/html; charset=utf-8
5. Connection: close
6.  
7. 200

It's only the local ES that goes haywire. Which would mean in the end - ES is screwing with the sockets in some way, which only would affect FPC/Synapse.

Does this Python server listen on localhost only in WSL though? Or does it listen on 0.0.0.0?

[LeadGuit]

I have installed elasticsearch from: https://www.elastic.co/de/downloads/elasticsearch
I use Windows 10 and Lazarus 2.0.12

With this code i got response:
...

Which Linux are you running on your WSL?
The problem is connecting to an ES instance running on WSL, not Windows (in case you installed ES on Windows instead). Your code results in the same ESocketError for me.

Does this Python server listen on localhost only in WSL though? Or does it listen on 0.0.0.0?

The tested uvicorn listens on 127.0.0.1:

Code: Pascal  [Select][+][-]

1. INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)

[sstvmaster]

... ES instance running on WSL...

I'm on Windows only. And what is an "ES instance" and "WSL" ?

rgr

[PascalDragon]

... ES instance running on WSL...

I'm on Windows only. And what is an "ES instance" and "WSL" ?

ES instance = the Elastic Search server
WSL = Windows Services for Linux (aka "Linux on Windows")

[LeadGuit]

... ES instance running on WSL...

I'm on Windows only. And what is an "ES instance" and "WSL" ?

rgr

Ah ok.
ES Instance: Simply an instance of Elasticsearch - meaning "one running version of it somewhere"
WSL: Windows Subsystem for Linux - Microsoft integrated a linux kernel into Windows some time ago, running as very lightweight virtual machine under Windows. I do all my dayjob coding now in there, as the stuff I need runs like 99% smoothly and I can properly develop on Linux without having to dual-boot, a separate laptop or other shenanigans. Additional benefit, due to the forwarding mentioned in other posts here, my Browser on Windows can access the server running on this Linux as localhost. So if I am developing a webapp, I use VSCode's remote function to write the code, execute the command line of linux to run it, and my windows browser to access it. The Linux doesn't need any GUI facilities this way - best of both worlds imo.

For my problem in this thread this means - the architecture is like this:
Windows -> Developing Lazarus apps for fun, to learn a new language and quick GUI stuff
Linux inside Windows -> Dayjob coding, machine learning, SysAdmin automation etc

Which in turn means -> ES is hosted inside the Linux VM, since I need it for the dayjob - that enables convenience using the elasticsearch package with Python in my linux dev setup, as the default is "localhost", meaning when I write my data into the "production server", I simply input it's IP and off we go, while I can test, delete and adjust my logic on my local ES instance - just to be sure I don't accidentally change/delete something important. At the same time, I can easily use a chrome extension on windows to do simple ES queries to check if stuff gets done right - by querying localhost as well.
So what I tried to do for the original question was creating a unit for elasticsearch access - similarly as I would create a python module, to conveniently just call a couple functions to interact and write/read data. The problem seems to arise in the way Windows handles that port forwarding - as said, Chrome, Python, Curl etc all get properly forwarded to the Linux VM underneath, the FPC app does not - but only in the case of Elasticsearch. Other servers I run under my linux can be accessed as intended.
Unfortunately ES isn't supported on WSL (meaning - it runs, I've been working that way for over 2 years xD, but they don't give official support for it). So if you ask th eES guys a question - the answer is "it's not supported, sorry". The interesting part is merely, that it just concern that forwarding constellation - as mentioned, connecting to the ES Instance via the Linux VM's URL does work (with ES in an "expanded development" mode).

I hope that clears it all up a bit :-)

[sstvmaster]

Thank you for your explanation. So then i think if your elasticsearch server running as localhost (127.0.0.1), then you only have access from this machine (with curl it works).

So you must run the elasticsearch server with ip mode 0.0.0.0 so he can accept requests from outside.
see: https://discuss.elastic.co/t/network-host-0-0-0-0-is-not-working-in-elasticsearch-7-3-0/195697 and https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html

[LeadGuit]

Thank you for your explanation. So then i think if your elasticsearch server running as localhost (127.0.0.1), then you only have access from this machine (with curl it works).

So you must run the elasticsearch server with ip mode 0.0.0.0 so he can accept requests from outside.
see: https://discuss.elastic.co/t/network-host-0-0-0-0-is-not-working-in-elasticsearch-7-3-0/195697 and https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html

Exactly, that's was the "kind of solution" that I came up with a few posts above (the not recommended hack I talked about, since it exposes the ES to the whole network). But the problem is only circumvented, as I have to access it through the VMs IP instead of localhost. So to speak - it "kind of solved", but the problem is still there and now it's more of a bug hunt to find out why this problem exists :-)