Recent

Author Topic: TFPHTTPClient debugging  (Read 956 times)

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1077
TFPHTTPClient debugging
« on: May 20, 2022, 04:23:03 pm »
It works, it's pretty simple to get going, but it is hard to see what is happening and it likes raising exceptions (there are 15). Like this:

Code: Pascal  [Select][+][-]
  1. procedure TFPCustomHTTPClient.Get(const AURL: String; Stream: TStream);
  2. begin
  3.   HTTPMethod('GET',AURL,Stream,[200]);
  4. end;

The HTTPMethod is the main function to execute the request. Which ends up in:

Code: Pascal  [Select][+][-]
  1. Function TFPCustomHTTPClient.ReadResponse(Stream: TStream;
  2.   const AllowedResponseCodes: array of Integer; HeadersOnly: Boolean): Boolean;
  3.   //...
  4.   if not CheckResponseCode(FResponseStatusCode,AllowedResponseCodes) then
  5.     Raise EHTTPClient.CreateFmt(SErrUnexpectedResponse,[ResponseStatusCode]);

Where that "[200]" is the AllowedResponseCodes. So, if the result is not 200, you don't get to see the response, because it isn't read.

Also, seeing what message (and headers!) is actually sent is quite hard.

Is the intended use of this class that you override virtual methods until you can see what is happening?

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1077
Re: TFPHTTPClient debugging
« Reply #1 on: May 23, 2022, 03:09:33 pm »
How do you add parameters?

Thaddy

  • Hero Member
  • *****
  • Posts: 11897
Re: TFPHTTPClient debugging
« Reply #2 on: May 23, 2022, 07:04:08 pm »
200 is not the only legal response code. 301 and 302 are too (redirections, there are more...like for old security protocols) and quite very common nowadays.
https://en.wikipedia.org/wiki/HTTP_301
https://en.wikipedia.org/wiki/HTTP_302

So if you expect 200 rightaway you are making a programmer error.
Simply set AllowRedirects to true. Eventually you should get 200.
« Last Edit: May 23, 2022, 07:11:19 pm by Thaddy »
Black themes should be banned.

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 914
  • Professional amateur ;-P
Re: TFPHTTPClient debugging
« Reply #3 on: May 23, 2022, 08:00:46 pm »
Hey SymbolicFrank,

Also, seeing what message (and headers!) is actually sent is quite hard.

If you look at the code (https://gitlab.com/freepascal.org/fpc/source/-/blob/main/packages/fcl-web/src/base/fphttpclient.pp#L373) you can see that TFPHTTPClient exposes these, among many:

Code: Pascal  [Select][+][-]
  1.   TFPHTTPClient = Class(TFPCustomHTTPClient)
  2.   Published
  3.     { Shortened to make a point }
  4.     Property RequestHeaders;
  5.     Property RequestBody;
  6.     Property ResponseHeaders;
  7.     Property ResponseStatusCode;
  8.     Property ResponseStatusText;
  9.     Property Cookies;
  10.     Property Proxy;
  11.   end;

If you're looking to inspect the Request Headers before you perform a method, that would be RequestHeaders.
If you're looking to inspect the Request Body, usually empty if you don't place anything there, that would be RequestBody.
If you want to inspect the Response Headers after you've performed a method, that would be ResponseHeaders.

Cheers,
Gus
Lazarus 2.3.0(trunk) FPC 3.3.1(trunk) Ubuntu 21.10 64b Dark Theme
Lazarus 2.0.12(stable) FPC 3.2.2(stable) Ubuntu 21.10 64b Dark Theme
http://github.com/gcarreno

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 914
  • Professional amateur ;-P
Re: TFPHTTPClient debugging
« Reply #4 on: May 23, 2022, 08:05:35 pm »
Hey SymbolicFrank,

How do you add parameters?

If you're looking for a way to add parameters like one is used to on Postman, or some other HTTP Client object implementations you will not find any.

It's up to you to add the question mark symbol and the name, value pairs in the URL string that you pass to any method call.
Same with the anchor side of things: it's up to you to put the hash symbol and the name of the anchor if you want to have that in.

But I do agree that it would be a good Quality of Life type of feature that could be added without much effort.

Cheers,
Gus
Lazarus 2.3.0(trunk) FPC 3.3.1(trunk) Ubuntu 21.10 64b Dark Theme
Lazarus 2.0.12(stable) FPC 3.2.2(stable) Ubuntu 21.10 64b Dark Theme
http://github.com/gcarreno

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 914
  • Professional amateur ;-P
Re: TFPHTTPClient debugging
« Reply #5 on: May 23, 2022, 08:16:33 pm »
Hey SymbolicFrank,

AHHH, sorry, I completely forgot that you can also hook up on these events to have access to what's happening during the actual method call:

https://gitlab.com/freepascal.org/fpc/source/-/blob/main/packages/fcl-web/src/base/fphttpclient.pp#L373
Code: Pascal  [Select][+][-]
  1.   TFPHTTPClient = Class(TFPCustomHTTPClient)
  2.   Published
  3.     {Shortened to make a point}
  4.     Property OnRedirect;
  5.     Property OnPassword;
  6.     Property OnDataReceived;
  7.     Property OnDataSent;
  8.     Property OnHeaders;
  9.     Property OnGetSocketHandler;

Cheers,
Gus
« Last Edit: May 23, 2022, 09:38:47 pm by Gustavo 'Gus' Carreno »
Lazarus 2.3.0(trunk) FPC 3.3.1(trunk) Ubuntu 21.10 64b Dark Theme
Lazarus 2.0.12(stable) FPC 3.2.2(stable) Ubuntu 21.10 64b Dark Theme
http://github.com/gcarreno

Thaddy

  • Hero Member
  • *****
  • Posts: 11897
Re: TFPHTTPClient debugging
« Reply #6 on: May 23, 2022, 09:23:44 pm »
Indeed TFPCustomHTTPClient exposes more (inheritance) and even more from what TFPCustomHTTPClient inherits.
Follow the breadcrumps.
Black themes should be banned.

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1077
Re: TFPHTTPClient debugging
« Reply #7 on: May 24, 2022, 08:59:26 am »
200 is not the only legal response code. 301 and 302 are too (redirections, there are more...like for old security protocols) and quite very common nowadays.
https://en.wikipedia.org/wiki/HTTP_301
https://en.wikipedia.org/wiki/HTTP_302

So if you expect 200 rightaway you are making a programmer error.
Simply set AllowRedirects to true. Eventually you should get 200.
Yes, and you can turn redirecting on. But it is the TFPHTTPClient that only allows 200 and raises an exception on everything else. try..except won't help, because the response isn't read at all. So if you get a 400+, you cannot see what the error was.

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1077
Re: TFPHTTPClient debugging
« Reply #8 on: May 24, 2022, 09:02:13 am »
Hey SymbolicFrank,

Also, seeing what message (and headers!) is actually sent is quite hard.

If you look at the code (https://gitlab.com/freepascal.org/fpc/source/-/blob/main/packages/fcl-web/src/base/fphttpclient.pp#L373) you can see that TFPHTTPClient exposes these, among many:

Code: Pascal  [Select][+][-]
  1.   TFPHTTPClient = Class(TFPCustomHTTPClient)
  2.   Published
  3.     { Shortened to make a point }
  4.     Property RequestHeaders;
  5.     Property RequestBody;
  6.     Property ResponseHeaders;
  7.     Property ResponseStatusCode;
  8.     Property ResponseStatusText;
  9.     Property Cookies;
  10.     Property Proxy;
  11.   end;

If you're looking to inspect the Request Headers before you perform a method, that would be RequestHeaders.
If you're looking to inspect the Request Body, usually empty if you don't place anything there, that would be RequestBody.
If you want to inspect the Response Headers after you've performed a method, that would be ResponseHeaders.

Cheers,
Gus

Yes, but there are a lot of rules they have to conform to for them to be added to the message. So you have to override multiple methods to be able to see what is sent just before it is. That is the problem.  Ok, you can check the ResponseHeaders, unless you got something else than a 200.

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1077
Re: TFPHTTPClient debugging
« Reply #9 on: May 24, 2022, 09:09:58 am »
Hey SymbolicFrank,

How do you add parameters?

If you're looking for a way to add parameters like one is used to on Postman, or some other HTTP Client object implementations you will not find any.

It's up to you to add the question mark symbol and the name, value pairs in the URL string that you pass to any method call.
Same with the anchor side of things: it's up to you to put the hash symbol and the name of the anchor if you want to have that in.

But I do agree that it would be a good Quality of Life type of feature that could be added without much effort.

Cheers,
Gus

Yes, and I did.

The TFPHTTP client expects a server, a path and a file to get. For most modern sites the URL is a function to call and a set of parameters. 20 years ago that would have been totally fine. Today, not so much.

And I did add and fix it all. This is just my feedback on what definitely could (and probably should) be improved.

Thaddy

  • Hero Member
  • *****
  • Posts: 11897
Re: TFPHTTPClient debugging
« Reply #10 on: May 24, 2022, 09:28:22 am »
Yes, and you can turn redirecting on. But it is the TFPHTTPClient that only allows 200 and raises an exception on everything else. try..except won't help, because the response isn't read at all. So if you get a 400+, you cannot see what the error was.
No, TFpHttpClient does NOT throw exceptions on 301/302 if redirects are on. It will only throw an exception if the error is not a redirect or if the last in the chain is not 200. You can test this against e.g. google.com
"[200, 301,302]"  are AllowedResponseCodes
« Last Edit: May 24, 2022, 09:54:10 am by Thaddy »
Black themes should be banned.

Gustavo 'Gus' Carreno

  • Hero Member
  • *****
  • Posts: 914
  • Professional amateur ;-P
Re: TFPHTTPClient debugging
« Reply #11 on: May 24, 2022, 09:34:25 am »
Hey SymbolicFrank,

Yes, but there are a lot of rules they have to conform to for them to be added to the message.

I'm really sorry, you lost me here.
Could you please elaborate on what you call rules and what you call message?

So you have to override multiple methods to be able to see what is sent just before it is. That is the problem. 

From the quick look that I gave the code, I think that TFPHTTPClient is not a complicated HTTP Client.
It does HTTP 1.0 or 1.1 and that's pretty much it. I think it does not do HTTP 2.0, nor does it do compressed or chunked transport. But I could be wrong on these last ones.
It pretty much sends what you can gather from RequestHeaders, RequestBody and that's pretty much it, no magic behind the curtain.

So, when you say that you have to override multiple methods, I really don't understand your objective.

Ok, you can check the ResponseHeaders, unless you got something else than a 200.

If I'm not mistaken, apart from some 500 errors, you always get content back on the ResponseHeaders, so I don't understand your argument here, sorry.

In the end, what I think is that I really don't understand your debugging needs. And that's normal since you actually never specified them. You just stated that you can't debug it.

So please, in the quest to better help you, could you please give more specifics on what you're expectations for the debug are?
Could you provide us with an example of what you input and then what you're expecting to get back on such debug?
This would really help us give you a more precise and focused help.

Cheers,
Gus
Lazarus 2.3.0(trunk) FPC 3.3.1(trunk) Ubuntu 21.10 64b Dark Theme
Lazarus 2.0.12(stable) FPC 3.2.2(stable) Ubuntu 21.10 64b Dark Theme
http://github.com/gcarreno

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1077
Re: TFPHTTPClient debugging
« Reply #12 on: May 24, 2022, 09:51:45 am »
Yes, and you can turn redirecting on. But it is the TFPHTTPClient that only allows 200 and raises an exception on everything else. try..except won't help, because the response isn't read at all. So if you get a 400+, you cannot see what the error was.
No, TFpHttpClient does NOT throw exceptions on 301/302 if redirects are on. It will only throw an exception if the error is not a redirect or if the last in the chain is not 200. You can test this against e.g. google.com
"[200, 301,302]"  are the AllowedResponseCodes
Yes, that's what I said.

What I also said, was that it generates an exception if you get a 400+, and that you are unable to see the error message, because it isn't read.

Thaddy

  • Hero Member
  • *****
  • Posts: 11897
Re: TFPHTTPClient debugging
« Reply #13 on: May 24, 2022, 09:56:48 am »
then should map the code against https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#4xx_client_errors
Those are descriptive enough.
Black themes should be banned.

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 1077
Re: TFPHTTPClient debugging
« Reply #14 on: May 24, 2022, 09:57:16 am »
Hey SymbolicFrank,
..
From the quick look that I gave the code, I think that TFPHTTPClient is not a complicated HTTP Client.
It does HTTP 1.0 or 1.1 and that's pretty much it. I think it does not do HTTP 2.0, nor does it do compressed or chunked transport. But I could be wrong on these last ones.
It pretty much sends what you can gather from RequestHeaders, RequestBody and that's pretty much it, no magic behind the curtain.
..
Cheers,
Gus

Exactly. It's for simple stuff, and it either works, or it doesn't.

I always used Synapse for client & server in the past, but as this is the default FP client, I used it this time. But it needs work to get it to do pretty basic stuff.

That's it.

 

TinyPortal © 2005-2018