Recent

Author Topic: TFPHTTPClient debugging  (Read 2279 times)

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: TFPHTTPClient debugging
« Reply #15 on: May 24, 2022, 10:00:05 am »
then should map the code against https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#4xx_client_errors
Those are descriptive enough.

It's a set, it won't fit ;)

For now, I did it like this:

Code: Pascal  [Select][+][-]
  1. procedure TRestClient.HTTPMethod(const AMethod, AURL: String; Stream: TStream;
  2.   const AllowedResponseCodes: array of Integer);
  3. var
  4.   p: TParameter;
  5.   s: string;
  6. begin
  7.   s := '';
  8.   if Parameters.Count > 0 then
  9.   begin
  10.     for p in Parameters do
  11.     begin
  12.       s += '&' + p.Name + '=';
  13.       if (not VarIsEmpty(p.Value)) and (not VarIsNull(p.Value)) then
  14.         s += VarToStr(p.Value);
  15.     end;
  16.     s[1] := '?';
  17.   end;
  18.  
  19.   inherited HTTPMethod(AMethod, AURL + s, Stream, [200, 201, 400, 401]);
  20. end;

And I'll add new errors when I encounter them.

Thaddy

  • Hero Member
  • *****
  • Posts: 14201
  • Probably until I exterminate Putin.
Re: TFPHTTPClient debugging
« Reply #16 on: May 24, 2022, 10:01:55 am »
Hey SymbolicFrank,
..
From the quick look that I gave the code, I think that TFPHTTPClient is not a complicated HTTP Client.
It does HTTP 1.0 or 1.1 and that's pretty much it. I think it does not do HTTP 2.0, nor does it do compressed or chunked transport. But I could be wrong on these last ones.
It pretty much sends what you can gather from RequestHeaders, RequestBody and that's pretty much it, no magic behind the curtain.
..
Cheers,
Gus

Exactly. It's for simple stuff, and it either works, or it doesn't.

I always used Synapse for client & server in the past, but as this is the default FP client, I used it this time. But it needs work to get it to do pretty basic stuff.

That's it.
What exactly do you miss in fcl-web? It is actually in the same family as synapse and some code is shared or derived (SSL).
I rewrote all my web client code to use fcl-web/net instead of synapse, because it is better maintainted and seems to allow more connections.
« Last Edit: May 24, 2022, 10:04:02 am by Thaddy »
Specialize a type, not a var.

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 1111
  • Professional amateur ;-P
Re: TFPHTTPClient debugging
« Reply #17 on: May 24, 2022, 10:05:03 am »
Hey SymbolicFrank,

Yes, and I did.

Ok, good, you handled the parameters, nice :)

The TFPHTTP client expects a server, a path and a file to get. For most modern sites the URL is a function to call and a set of parameters. 20 years ago that would have been totally fine. Today, not so much.

For the verb/method GET, most API sites need an endpoint, some sort of authentication and some parameters in some cases.
For the verbs/methods POST, PUT, DELETE, most API sites need an endpoint, some sort of authentication, some content on the RequestBody and maybe some parameters.

  • For the endpoint, it's the piece of the URL that comes after the site's domain and you put that on the URL string.
  • For the verbs themselves, apart from GET and POST, depending on how the site gets the others, usually it's via a Header added to the RequestHeaders.
  • For the authentication:
    - if it's basic, you have Username and Password properties.
    - if it's something else, you have the RequestHeaders and/or parameters.
  • For the RequestBody you have a stream type of thing, so you can put anything in there, even plain old JSON.

You usually then get JSON or XML as the result of calling one of the verbs/methods, so you can then process that elsewhere in your app.

From a modern perspective, I don't see what more you need in terms of added functionality to TFPHTTPClient.

If so, and I guess I could've missed something, could you please be a bit more specific on what you feel is missing?

And I did add and fix it all.

You've lost me here again. What did you add and fix?

This is just my feedback on what definitely could (and probably should) be improved.

Ermm, again, pardon me for asking, but I don't recall you mentioning or specifying any improvements.
What I recall is you mentioning a lot of things that are either wrong or missing according to your opinion.

Cheers,
Gus
Lazarus 3.99(main) FPC 3.3.1(main) Ubuntu 23.10 64b Dark Theme
Lazarus 3.0.0(stable) FPC 3.2.2(stable) Ubuntu 23.10 64b Dark Theme
http://github.com/gcarreno

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 1111
  • Professional amateur ;-P
Re: TFPHTTPClient debugging
« Reply #18 on: May 24, 2022, 10:15:01 am »
Hey SymbolicFrank,

Exactly. It's for simple stuff, and it either works, or it doesn't.

I was still writing an answer to the other reply and Thaddy responded to this one in the meantime.
And I have to say that I agree with him!!

So, maybe the problem is with the way you're trying to use the component.

From my experience there are 2 ways:
  • The simple way, but you forgo some control
  • The complete way and you have full control

Could you please give us some examples of your code so we could maybe point you in the right direction?

I always used Synapse for client & server in the past, but as this is the default FP client, I used it this time. But it needs work to get it to do pretty basic stuff.

I have the same experience as Thaddy in this aspect. I've used both synapse and Indy in the past for lack of an internal option.
But ever since they released TFPHTTPClient that I've been using it and figuring out the kinks when using SSL across multiple FPC versions.

So again, I beg of you, could you please give us some examples of you code and what issues you're stumbling on?

Cuz me and Thaddy have been rather happy with it for quite a while and here you come saying it doesn't work, so you confuse use, a lot!!

Cheers,
Gus
Lazarus 3.99(main) FPC 3.3.1(main) Ubuntu 23.10 64b Dark Theme
Lazarus 3.0.0(stable) FPC 3.2.2(stable) Ubuntu 23.10 64b Dark Theme
http://github.com/gcarreno

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: TFPHTTPClient debugging
« Reply #19 on: May 24, 2022, 10:58:27 am »
Hey SymbolicFrank,
..
So again, I beg of you, could you please give us some examples of you code and what issues you're stumbling on?

Cuz me and Thaddy have been rather happy with it for quite a while and here you come saying it doesn't work, so you confuse use, a lot!!

Cheers,
Gus

Hi Gus,

I think I did, see my post above.

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: TFPHTTPClient debugging
« Reply #20 on: May 24, 2022, 10:59:28 am »
Yes, and you can turn redirecting on. But it is the TFPHTTPClient that only allows 200 and raises an exception on everything else. try..except won't help, because the response isn't read at all. So if you get a 400+, you cannot see what the error was.
Thats wrong, it is possible, you can simply ask the client afterwards:
Code: Pascal  [Select][+][-]
  1. var
  2.   cli: TFPHTTPClient;
  3.   res: TStringStream;
  4. begin
  5.   cli := TFPHTTPClient.Create(nil);
  6.   res := TStringStream.Create;
  7.   try
  8.     try
  9.       cli.Get('http://google.com/ThisPathDoesNotExist', res);
  10.     except
  11.       res.Seek(0, soBeginning);
  12.       WriteLn(cli.ResponseStatusCode, ' ', cli.ResponseStatusText);
  13.       WriteLn(cli.ResponseHeaders.Text);
  14.       WriteLn(res.DataString);
  15.     end;
  16.   finally
  17.     cli.Free;
  18.     res.Free
  19.   end;
  20.   ReadLn;
  21. end.
This gives the complete HTTP response
Code: Text  [Select][+][-]
  1. 404 Not Found
  2. Content-Type: text/html; charset=UTF-8
  3. Referrer-Policy: no-referrer
  4. Content-Length: 1581
  5. Date: Tue, 24 May 2022 08:54:19 GMT
  6. Connection: close
  7.  

That said, it is incredibly clumsy, as it does not make use of the main feature of exceptions (that they are objects that can hold abitrary data), so you have to get your data from different sources, with the actual exception being so void of information that in the code above I did not even need to catch it but a general except block is enough.
Also this means that this does only work with an initialized HTTP client and not with the helper functions, and therefore requires a lot of boilerplate.
This would be so much more comfortable:
Code: Pascal  [Select][+][-]
  1. try
  2.   Res := TFPHTTPClient.Get(url);
  3. except on E: EInvalidHTTPResponse do
  4. begin
  5.   WriteLn(E.StatusCode, ' ', E.StatusText);
  6.   WriteLn(E.ResponseHeaders.Text);
  7.   WriteLn(E.ResponseBody);
  8. end;
  9. end;

So the current design is bad, no question, but at least it is possible to handle this sort of stuff

Thaddy

  • Hero Member
  • *****
  • Posts: 14201
  • Probably until I exterminate Putin.
Re: TFPHTTPClient debugging
« Reply #21 on: May 24, 2022, 11:01:51 am »
Another option is to use libcurl which has full http/2 support. It is not as easy to use as synapse or fcl-web, but it is not THAT difficult and comes with FPC as standard: ./packages/libcurl
In windows you need to obtain libcurl.dll. I fixed windows support a couple of years ago and Florian merged my patch.
« Last Edit: May 24, 2022, 11:04:53 am by Thaddy »
Specialize a type, not a var.

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: TFPHTTPClient debugging
« Reply #22 on: May 24, 2022, 11:04:33 am »
It's a set, it won't fit ;)

For now, I did it like this:

Code: Pascal  [Select][+][-]
  1. procedure TRestClient.HTTPMethod(const AMethod, AURL: String; Stream: TStream;
  2.   const AllowedResponseCodes: array of Integer);
  3. var
  4.   p: TParameter;
  5.   s: string;
  6. begin
  7.   s := '';
  8.   if Parameters.Count > 0 then
  9.   begin
  10.     for p in Parameters do
  11.     begin
  12.       s += '&' + p.Name + '=';
  13.       if (not VarIsEmpty(p.Value)) and (not VarIsNull(p.Value)) then
  14.         s += VarToStr(p.Value);
  15.     end;
  16.     s[1] := '?';
  17.   end;
  18.  
  19.   inherited HTTPMethod(AMethod, AURL + s, Stream, [200, 201, 400, 401]);
  20. end;

And I'll add new errors when I encounter them.
First, this code is potentially buggy, as you need to URL encode your parameters. Second, I think you shouldn't encode the parameters at this stage, the parameter encoding should be part of the URL with which the function gets called. I know that fpweb is missing some easy URI handling that makes the use of queries, fragments and co easy, but still this sort of stuff should already be part of the URI that gets in, it has nothing todo with HTTP and should therefore be not part of the HTTPClient class

Thaddy

  • Hero Member
  • *****
  • Posts: 14201
  • Probably until I exterminate Putin.
Re: TFPHTTPClient debugging
« Reply #23 on: May 24, 2022, 11:09:38 am »
Links for libcurl on Windows are here: https://curl.se/windows/
In most nixes it comes as standard, but if not you can obtain it from the same website: curl.se
Note that windows 10/11 come with curl.exe as standard, but without the libcurl.dll because Microsoft has statically linked it.
Also note that the example header still says it needs modifications for windows. This is not the case after my fix, except you may want to rename libcurl-x64.dll to libcurl.dll for 64 bit windows. Static linking works too with FPC.

Curl is very complete, stable, secure and very up-to-date. (At the cost of steep learning curve for programmers if you want all its features)
« Last Edit: May 24, 2022, 11:24:54 am by Thaddy »
Specialize a type, not a var.

Thaddy

  • Hero Member
  • *****
  • Posts: 14201
  • Probably until I exterminate Putin.
Re: TFPHTTPClient debugging
« Reply #24 on: May 24, 2022, 11:27:33 am »
Oh, wait!,
I see the libcurl.pp in the fpc distribution needs an update... I will create one.
Specialize a type, not a var.

MvC

  • New Member
  • *
  • Posts: 25
    • Free Pascal Core team member
Re: TFPHTTPClient debugging
« Reply #25 on: May 24, 2022, 11:29:52 am »
About "parameters":

As the maintainer of fphttpclient, I can only agree with warfly.

The HTTP protocol expects a URL.  How this URL is constructed is not its business.

But I agree that a class (or advanced record) for easy URL constructing would be useful, but it simply must not be part of the HTTPClient class.
I have made a note to add this to the URIParser or HTTPProtocol unit.

About exceptions:

I think it is indeed possible to improve EHTTTPClient to contain the error document, and I will add this.
It has already fields for status code and text, I will check whether they are already filled.

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: TFPHTTPClient debugging
« Reply #26 on: May 24, 2022, 11:30:02 am »
@Warfley: I agree. This is for testing, and I certainly am going to URLEncode them and add a lot of other code to make it more bullet-proof, before it is production ready.

I just wasn't sure if TFPHHTPClient was the way to go.

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1313
Re: TFPHTTPClient debugging
« Reply #27 on: May 24, 2022, 11:32:15 am »
About "parameters":

As the maintainer of fphttpclient, I can only agree with warfly.

The HTTP protocol expects a URL.  How this URL is constructed is not its business.

But I agree that a class (or advanced record) for easy URL constructing would be useful, but it simply must not be part of the HTTPClient class.
I have made a note to add this to the URIParser or HTTPProtocol unit.

About exceptions:

I think it is indeed possible to improve EHTTTPClient to contain the error document, and I will add this.
It has already fields for status code and text, I will check whether they are already filled.

Thanks!

 

TinyPortal © 2005-2018