Print Page - HTTP request over Unix Domain Socket

Programming => Networking and Web Programming => Topic started by: dubst3pp4 on December 02, 2017, 09:55:51 am

Title: HTTP request over Unix Domain Socket
Post by: dubst3pp4 on December 02, 2017, 09:55:51 am

Hello, I'm wondering if it is possible to send HTTP requests over the Unix Domain Socket? I want to query a service, which exposes an API via HTTP and Unix Domain Sockets, but I could not find out how to do this with TFPHTTPClient class... Any ideas?

Best regards,
Marc

Title: Re: HTTP request over Unix Domain Socket
Post by: Leledumbo on December 02, 2017, 05:49:03 pm

It requires additional support to connect over unix domain socket, which I don't think TFPHTTPClient has.

Title: Re: HTTP request over Unix Domain Socket
Post by: KemBill on December 02, 2017, 08:16:46 pm

I know "simple sockets", but if the only difference between UNIX and IP

Code: Pascal [Select][+]

server_sockaddr.sun_family = AF_UNIX;  

instead

Code: Pascal [Select][+]

server_sockaddr.sun_family = AF_INET;  

you can try to overload the method

Title: Re: HTTP request over Unix Domain Socket
Post by: dubst3pp4 on December 03, 2017, 05:47:09 pm

It seems that the sockets unit is all what I need to connect to a Unix socket. Maybe subclassing the TFPHTTPClient class to use a Unix socket instead of an Internet socket should work... All the base HTTP functionality is already in the class, so it would be great if I could use it...

Title: Re: HTTP request over Unix Domain Socket
Post by: KemBill on December 03, 2017, 06:16:59 pm

Are you sur you have nothing listening using netstat 127.0.0.1:80 ?

It seems you can inject a TSocketHandler class into TFPHTTPClient, so you just need to write a wrapper using TUnixSocket

Title: Re: HTTP request over Unix Domain Socket
Post by: dubst3pp4 on December 04, 2017, 11:56:44 am

Quote from: KemBill on December 03, 2017, 06:16:59 pm

Are you sur you have nothing listening using netstat 127.0.0.1:80 ?

It seems you can inject a TSocketHandler class into TFPHTTPClient, so you just need to write a wrapper using TUnixSocket

I'm trying to query the Docker daemon, which could be exposed to a real network host, but I want to avoid that.

I've played around for some time and connecting to the daemon is as easy as:

Code: Pascal [Select][+]

UnixSocket := TUnixSocket.Create('/var/run/docker.sock');

After that you can send the HTTP-Request with the WriteBuffer method and read data with the ReadBuffer method.

Now the interesting part:
the TFPHTTPClient class creates a TInetSocket instance for the private field FSocket, which is also of type TInetSocket, in the protected method ConnectToServer. There are no dependencies to the functionality of TInetSocket in any other methods, everywhere else just methods of the parent class TSocketStream of TInetSocket and TUnixSocket are used.

So if I could change FSocket from a private to a protected field and override the ConnectToServer method in a subclass, all should be fine. Unfortunately to change the private field in TFPHTTPClient, I would have to create a copy of the fphttpclient unit, wouldn't I? Any chance to take another route to solve this problem?

I also think it would be a good idea to contact the author of the unit (Michael Van Canneyt?) to propose such changes, so that the class is usable with Unix domain sockets, too (as more and more Linux services use HTTP over UDS, for example Docker and Node.js).

Title: Re: HTTP request over Unix Domain Socket
Post by: dubst3pp4 on December 04, 2017, 03:54:30 pm

I got it now working by changing the type of the FSocket field to FSocketStream. In my subclass I'm extending TFPHttpCustomClient by reimplementing the ConnectToServer method, which in turn creates an instance of TUnixSocket, while the original ConnectToServer creates a TInetSocket instance.

Maybe this is not the best solution, but it seems to work without problems. I would like to propose this change but don't know how or to whom... :-\

Title: Re: HTTP request over Unix Domain Socket
Post by: Leledumbo on December 04, 2017, 08:53:56 pm

Quote from: dubst3pp4 on December 04, 2017, 03:54:30 pm

Maybe this is not the best solution, but it seems to work without problems. I would like to propose this change but don't know how or to whom... :-\

Ask Michael, you already knew his name before.

Title: Re: HTTP request over Unix Domain Socket
Post by: dubst3pp4 on December 05, 2017, 05:57:37 am

Okay, I will send him an email :)

Title: Re: HTTP request over Unix Domain Socket
Post by: istoica on January 25, 2022, 11:10:24 am

Have you managed to complete this ?

Title: Re: HTTP request over Unix Domain Socket
Post by: MarkMLl on January 25, 2022, 11:15:16 am

Quote from: istoica on January 25, 2022, 11:10:24 am

Have you managed to complete this ?

Necroposting will get you in trouble these days.

What are you trying to do: specifically use HTTP or use unix-domain sockets in general? If the latter then see this recent thread https://forum.lazarus.freepascal.org/index.php/topic,57706.msg429320.html#msg429320

MarkMLl

Title: Re: HTTP request over Unix Domain Socket
Post by: istoica on January 25, 2022, 12:04:29 pm

Thank you MarkMLI, I try to do both actually, exactly like the author of this thread.

Make an HTTP request over unix domain sockets, there are APIs such as docker api or podman api that due to security reasons, they don't expose http servers over TCP

https://docs.podman.io/en/latest/_static/api.html

curl --unix-socket /run/podman/podman.sock http://d/v3.0.0/libpod/info

- The curl client has built-in support for it these days
- A famous JS http client library axios, has `socketPath` as argument to support it - https://axios-http.com/docs/req_config
- Unix sockets are very useful for these scenarios too, even Windows supports it from 2017 - https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows

So back to original, I simply want to use the simplicity of `TFPHTTPClient` using unix domain sockets. Tried to follow discoveries here, but arrived nowhere, segfault :)

Code: Pascal [Select][+]

unit Unit2;
 
{$mode ObjFPC}{$H+}
 
interface
 
uses
  Classes, SysUtils, fphttpclient, ssockets, httpdefs, uriparser, strutils;
 
type
 
  TFPExHTTPClient = Class(TFPCustomHTTPClient)
  private
    FUnixSocketPath: string;
    FSocket : TSocketStream;
    FIOTimeout: Integer;
    FConnectTimeout: Integer;
  Public
    Property UnixSocketPath: string Read FUnixSocketPath Write FUnixSocketPath;
    Procedure ConnectToServer(const AHost: String; APort: Integer; UseSSL : Boolean=False); override;
  end;
 
implementation
   
 
procedure TFPExHTTPClient.ConnectToServer(const AHost: String;
  APort: Integer; UseSSL : Boolean = False);
 
Var
  G : TSocketHandler;
 
 
begin
  If IsConnected Then
    DisconnectFromServer; // avoid memory leaks
  if (Aport=0) then
    if UseSSL then
      Aport:=443
    else
      Aport:=80;
  G:=GetSocketHandler(UseSSL);
  if FUnixSocketPath = '' then
    FSocket:=TInetSocket.Create(AHost,APort,G)
  else
    WriteLn(FUnixSocketPath);
    FSocket:=TUnixSocket.Create(FUnixSocketPath);
 
  try
    if FIOTimeout<>0 then
      FSocket.IOTimeout:=FIOTimeout;
    if FConnectTimeout<>0 then
      FSocket.ConnectTimeout:=FConnectTimeout;
   //     if TypeOf(FSocket) is TInetSocket then
    //  FSocket.Connect;
  except
    FreeAndNil(FSocket);
    Raise;
  end;
end;
 
end. 
 

Title: Re: HTTP request over Unix Domain Socket
Post by: MarkMLl on January 25, 2022, 12:21:58 pm

I can't help with the HTTP side of it... I'm not even going to start thinking about it since it's outside my field and I've got a lot of other stuff on my plate.

I suggest stepping into TUnixSocket.Create() and seeing what's going wrong, and also looking at the earlier discussion. It might be something silly like a malformed path or access rights to (the file-like named entity implementing) the path, or it might be that that object isn't fully implemented. In extremis read through the earlier thread I cited and try opening the socket as a simple text file.

Most of my experience with UD sockets has involved using them for datagrams, i.e. UDP-like rather than TCP-like. The problems were almost always on the server side (i.e. access rights etc. which necessitated my getting involved with POSIX Capabilities) but in the case of streams (TCP-like) there are also hazards when a connection has been shut down and not properly restarted.

MarkMLl

Title: Re: HTTP request over Unix Domain Socket
Post by: istoica on January 25, 2022, 12:57:25 pm

I have copy-pasted the entire fphttpclient unit and modified the following

Modified private `FSocket : TInetSocket;` to `FSocket : TUnixSocket;`
Added private `FUnixSocketPath: String` to `TFPCustomHTTPClient`
Added public `Property UnixSocketPath : String Read FUnixSocketPath Write FUnixSocketPath;`
Modified public `TFPCustomHTTPClient.ConnectToServer` and commented out `FSocket.Connect;` as there is no Connect for TUnixSocket - one could add dummy Connect to TUnixSocket

Now this beauty works properly, I get expected response

Quote

var
S : String;
begin

With TFPCustomHTTPClient.Create(nil) do
try
UnixSocketPath := '/tmp/podman.sock';
S := Get('http://d/v3.0.0/libpod/info');
WriteLn(S);
finally
Free;
end;

It does the job, but it is so ... unpleasant to copy-paste the entire unit, it would be so cool if this is built-in

Title: Re: HTTP request over Unix Domain Socket
Post by: MarkMLl on January 25, 2022, 01:13:55 pm

In that case you'll have to submit a patch or at least make a feature request providing your suggested implementation via https://gitlab.com/freepascal.org/lazarus/lazarus/-/issues

There was something I found I had to do to get basic sockets working... ah yes,

Code: Pascal [Select][+]

  (* This is a reimplementation of the standard Connect() since it doesn't
    support a unix-domain address and truncates it if told to cast it to an
    inet domain address structure.
 
    Test using e.g.  $ nc -lkU ~/dsocket  noting the -k to prevent the listener
    terminating when the writer (this program) closes the connection.
  *)
  Function Connect(Sock:longint;const addr: TUnixSockAddr;var SockIn,SockOut:text):Boolean;
 
 
    Function DoConnect(Sock:longint;const addr: TUnixSockAddr): Boolean;
 
    var
      res: longint;
    begin
      repeat
        res:=fpconnect(Sock,@Addr,SizeOF(TUnixSockAddr)); (* NOTA BENE *)
      until (res<>-1) or (SocketError <> EsockEINTR);
      DoConnect:= res = 0;
    end;
 
 
  begin
    Connect:=DoConnect(Sock,addr);
    If Connect then
       Sock2Text(Sock,SockIn,SockOut);
  end { Connect } ;
 

noting the different address type, which is of course much the same thing that you tackled in your earlier posting.

MarkMLl

Title: Re: HTTP request over Unix Domain Socket
Post by: istoica on January 25, 2022, 05:47:48 pm

Thank you, I've did my best, I am not that big expert in FPC design / idioms

https://gitlab.com/freepascal.org/fpc/source/-/merge_requests/151

Title: Re: HTTP request over Unix Domain Socket
Post by: czd on August 08, 2022, 11:19:32 pm

Hi,
Sorry to open this post again, but I cant convert the fphttpclient for UnixSocket, can someone has the modified versions of this fphttpclient unit.
Thanks.

Title: Re: HTTP request over Unix Domain Socket
Post by: MarkMLl on August 09, 2022, 09:16:40 am

Why? What problem are you having... error messages or run-time behaviour? In what way is the existing discussion (with attachments and links) unhelpful? Where's your example code?

MarkMLl

Title: Re: HTTP request over Unix Domain Socket
Post by: czd on August 09, 2022, 12:59:03 pm

Hi MarkMLI,
Soon, I realized that link https://gitlab.com/freepascal.org/fpc/source/-/merge_requests/151/diffs#diff-content-e2642c4dbb5c1d3691ddf9a8201235e1a41e1e0e (https://gitlab.com/freepascal.org/fpc/source/-/merge_requests/151/diffs#diff-content-e2642c4dbb5c1d3691ddf9a8201235e1a41e1e0e). I'll try from that source.
Thanks.

Title: Re: HTTP request over Unix Domain Socket
Post by: MarkMLl on August 09, 2022, 01:02:34 pm

Otherwise see some of the snippets I've posted in this thread. I went through the various possibilities and got all working (not necessarily using FPC library code) for piping (term used loosely) text between two different programs a few months ago.

But /please/ bear in mind that this is OS-specific etc., and that you really do need to include information on what you're using in the body of your message even if it's non-obvious why.

MarkMLl

Lazarus

Programming => Networking and Web Programming => Topic started by: dubst3pp4 on December 02, 2017, 09:55:51 am