Recent

Author Topic: Problem with POST [BUG discovered]  (Read 1112 times)

Vodnik

  • Full Member
  • ***
  • Posts: 184
Problem with POST [BUG discovered]
« on: January 21, 2021, 07:21:33 pm »
Hello!
This is my first step attempt to use REST API.
I've used this example as a base:
https://wiki.lazarus.freepascal.org/fphttpclient#Posting_JSON
Here is my code:
Code: Pascal  [Select][+][-]
  1. program http;
  2. uses
  3.   classes,  // TStringList
  4.   sysutils,
  5.   fphttpclient, opensslsockets;
  6. var
  7.   Client: TFPHTTPClient;
  8.   Response: TStringStream;
  9. begin
  10.   try
  11.     writeln('Initializing HTTP client...');
  12.     Client := TFPHTTPClient.Create(nil);
  13.     Client.AllowRedirect := True;
  14.     writeln('HTTP client successfully initialized.');
  15.     writeln('Looking for REST SDK API...');
  16.     try
  17.       if Pos('Rest SDK API',Client.Get('https://10.10.111.211/restsdk'))<>0 then begin
  18.         writeln('Rest SDK API is detected.');
  19.         writeln('Registering application...');
  20.         Client.AddHeader('Content-Type','application/json; charset=utf-8');
  21.         Client.AddHeader('Cache-Control','no-cache');
  22.         Client.AddHeader('Host','10.10.111.211');
  23.         Client.RequestBody:=TRawByteStringStream.Create('{ "webhookURL" : "https://enilpc2l0jwu.x.pipedream.net", "application" : { "name" : "TestRestSdkClient", "token" : "0e59b20476fca9042496084c99a1dc8f00f55a963146be13f6a9d6db0ee309af", "busUnitName" : "DEFAULT" } }');
  24.         Response:= TStringStream.Create('');
  25.         try
  26.           Client.Post('https://10.10.111.211/restsdk/webapi/main/registerApplication',Response);
  27.           writeln('Response Code is ' + inttostr(Client.ResponseStatusCode));
  28.           writeln(Client.ResponseStatusText);
  29.           if Client.ResponseStatusCode=0 then begin
  30.             writeln('Application registered');
  31.           end else begin
  32.             writeln('Application not registered');
  33.           end;
  34.         except
  35.           on E: exception do
  36.             writeln(E.Message);
  37.         end;
  38.       end else writeln('Rest SDK API is not detected.');
  39.     except
  40.       on E: exception do
  41.         writeln(E.Message);
  42.     end;
  43.   finally
  44.     Client.RequestBody.Free;
  45.     Client.Free;
  46.     Response.Free
  47.   end;
  48.   writeln('Press ENTER to exit');
  49.   readln();
  50. end.
Problem is that I always get response code 400.
I guess that body format may be not good, but the same REST request from Postman works fine.
I tried to check this with Wireshark, but communication is encrypted.
Please help with ideas for debug.
« Last Edit: January 31, 2021, 02:27:11 pm by Vodnik »

Vodnik

  • Full Member
  • ***
  • Posts: 184
Re: Problem with POST
« Reply #1 on: January 24, 2021, 02:38:43 pm »
Just replacing Post procedure with simplePost solves the problem!
Can somebody explain the difference between them, please?

Vodnik

  • Full Member
  • ***
  • Posts: 184
Re: Problem with POST
« Reply #2 on: January 24, 2021, 05:18:30 pm »
Well, SimplePost just calls Post, setting KeepConnection to False:
Code: Pascal  [Select][+][-]
  1. class procedure TFPCustomHTTPClient.SimplePost(const URL: string;
  2.   const Response: TStream);
  3.  
  4. begin
  5.   With Self.Create(nil) do
  6.     try
  7.       KeepConnection := False;
  8.       Post(URL,Response);
  9.     finally
  10.       Free;
  11.     end;
  12. end;    

So how KeepConnection may cause "Bad request" response?
Is it the same as setting header "Connection":"keep-alive"?

Leledumbo

  • Hero Member
  • *****
  • Posts: 8305
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: Problem with POST
« Reply #3 on: January 25, 2021, 01:56:16 am »
Your endpoint either doesn't like the content-type or the host header. It's endpoint specific behavior so you must consult the server side documentation.

Vodnik

  • Full Member
  • ***
  • Posts: 184
Re: Problem with POST
« Reply #4 on: January 25, 2021, 03:51:59 pm »
You are right, problem is with the host header.
If I remove
       
Code: Pascal  [Select][+][-]
  1. Client.AddHeader('Host','10.10.111.211');
then I guess host header is created somehow automatically, and then my Post request succeeds.
Host header for this API is documented exactly as I use it.
I have compared the same Post requests to RequestBin.com from Lazarus and from Postman but I didn't find any difference between the host headers in that case.

Vodnik

  • Full Member
  • ***
  • Posts: 184
Re: Problem with POST [BUG discovered]
« Reply #5 on: January 31, 2021, 03:29:00 pm »
Well, dear friends, seems there is a bug in TFPCustomHTTPClient.SendRequest procedure.
It adds "Host" header without any check for existing one.
Look at line 30: header "Host" is added without any check.
Later (line 38) it adds all headers, defined by programmer.
I'm also upset with "Cookies" header (line 48), which is added empty in my case (I've cleared them with         Client.Cookies.Clear).
Resulting string S (with 2 "Hosts" headers, empty "Cookies" header and two CRLF at the end) causes exception when calling FSocket.WriteBuffer procedure.
Code: Pascal  [Select][+][-]
  1. procedure TFPCustomHTTPClient.SendRequest(const AMethod: String; URI: TURI);
  2.  
  3. Var
  4.   PH,UN,PW,S,L : String;
  5.   I : Integer;
  6.   AddContentLength : Boolean;
  7.  
  8. begin
  9.   S:=Uppercase(AMethod)+' '+GetServerURL(URI)+' '+'HTTP/'+FHTTPVersion+CRLF;
  10.   UN:=URI.Username;
  11.   PW:=URI.Password;
  12.   if (UserName<>'') then
  13.     begin
  14.     UN:=UserName;
  15.     PW:=Password;
  16.     end;
  17.   If (UN<>'') then
  18.     begin
  19.     S:=S+'Authorization: Basic ' + EncodeStringBase64(UN+':'+PW)+CRLF;
  20.     I:=IndexOfHeader('Authorization');
  21.     If I<>-1 then
  22.       RequestHeaders.Delete(i);
  23.     end;
  24.   if Assigned(FProxy) and (FProxy.Host<>'') then
  25.     begin
  26.     PH:=FProxy.GetProxyHeaders;
  27.     if (PH<>'') then
  28.       S:=S+PH+CRLF;
  29.     end;
  30.   S:=S+'Host: '+URI.Host;
  31.   If (URI.Port<>0) then
  32.     S:=S+':'+IntToStr(URI.Port);
  33.   S:=S+CRLF;
  34.   AddContentLength:=Assigned(RequestBody) and (IndexOfHeader('Content-Length')=-1);
  35.   If AddContentLength then
  36.     AddHeader('Content-Length',IntToStr(RequestBody.Size));
  37.   CheckConnectionCloseHeader;
  38.   For I:=0 to FRequestHeaders.Count-1 do
  39.     begin
  40.     l:=FRequestHeaders[i];
  41.     If AllowHeader(L) then
  42.       S:=S+L+CRLF;
  43.     end;
  44.   If AddContentLength then
  45.     FRequestHeaders.Delete(FRequestHeaders.IndexOfName('Content-Length'));
  46.   if Assigned(FCookies) then
  47.     begin
  48.     L:='Cookie: ';
  49.     For I:=0 to FCookies.Count-1 do
  50.       begin
  51.       If (I>0) then
  52.         L:=L+'; ';
  53.       L:=L+FCookies[i];
  54.       end;
  55.     if AllowHeader(L) then
  56.       S:=S+L+CRLF;
  57.     end;
  58.   FreeAndNil(FSentCookies);
  59.   FSentCookies:=FCookies;
  60.   FCookies:=Nil;
  61.   S:=S+CRLF;
  62.   if not Terminated then
  63.     FSocket.WriteBuffer(S[1],Length(S));
  64.   If Assigned(FRequestBody) and not Terminated then
  65.     FSocket.CopyFrom(FRequestBody,FRequestBody.Size);
  66. end;          
So workaround for "Host" is: never add "Host" header manually!
How to completely clear "Cookies", I don't know.
In older examples I can see method Client.Clear, which is not available in current version (Lazarus 2.0.10, FPC 3.2.0).
I expect this was the way to clear the body and headers completely.
« Last Edit: February 07, 2021, 07:04:18 pm by Vodnik »

PascalDragon

  • Hero Member
  • *****
  • Posts: 2747
  • Compiler Developer
Re: Problem with POST [BUG discovered]
« Reply #6 on: February 01, 2021, 09:15:49 am »
So workaround for "Host" is: never add "Host" header manually!

Please file a bug report so that the class will at least complain if someone adds Host manually.

How to completely clear "Cookies", I don't know.

Please file a bug report for this as well.

In older examples I can see method Client.Clear, which is not available in current version (Lazarus 2.0.10, FPC 3.2.0).
I expect this was the way to clear the body and headers completely.

I can't find a Clear method it TFPHTTPClient back till at least 2.6.4. I also doubt that it would have done what you expect as the headers are built upon the request.

Vodnik

  • Full Member
  • ***
  • Posts: 184
Re: Problem with POST [BUG discovered]
« Reply #7 on: February 01, 2021, 10:08:47 am »
I will do with bug report.
I can't find a Clear method it TFPHTTPClient back till at least 2.6.4. I also doubt that it would have done what you expect as the headers are built upon the request.
Seems you are right. This is method of httpsend unit from Synapse.
So I have no idea how to clean the http client object between responses in FPHTTPClient.
« Last Edit: February 01, 2021, 10:11:59 am by Vodnik »

lucamar

  • Hero Member
  • *****
  • Posts: 3591
Re: Problem with POST [BUG discovered]
« Reply #8 on: February 01, 2021, 12:39:01 pm »
How to completely clear "Cookies", I don't know.

Untested, but:
Code: Pascal  [Select][+][-]
  1. FreeAndNil(MyHTTPClient.Cookies)
should do the trick
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus/FPC 2.0.8/3.0.4 & 2.0.10/3.2.0 - 32/64 bits on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

Vodnik

  • Full Member
  • ***
  • Posts: 184
Re: Problem with POST [BUG discovered]
« Reply #9 on: February 01, 2021, 08:09:25 pm »
No, this causes Error: Can't take the address of constant expressions.

lucamar

  • Hero Member
  • *****
  • Posts: 3591
Re: Problem with POST [BUG discovered]
« Reply #10 on: February 01, 2021, 10:19:44 pm »
No, this causes Error: Can't take the address of constant expressions.

Then try the alternatives, either:
Code: Pascal  [Select][+][-]
  1. MyHTTPClient.Cookies.Free;
  2. MyHTTPClient.Cookies := Nil;
or:
Code: Pascal  [Select][+][-]
  1. { var Strings: TStrings; }
  2. Strings := MyHTTPClient.Cookies;
  3. Strings.Free;
  4. MyHTTPClient.Cookies := Nil;
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus/FPC 2.0.8/3.0.4 & 2.0.10/3.2.0 - 32/64 bits on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

PascalDragon

  • Hero Member
  • *****
  • Posts: 2747
  • Compiler Developer
Re: Problem with POST [BUG discovered]
« Reply #11 on: February 02, 2021, 09:06:18 am »
If you look at the code you'll see that accessing the getter of Cookies will recreate the FCookies field.

Vodnik

  • Full Member
  • ***
  • Posts: 184
Re: Problem with POST [BUG discovered]
« Reply #12 on: February 06, 2021, 09:04:14 pm »
If you look at the code you'll see that accessing the getter of Cookies will recreate the FCookies field.
Yes, if FCookies=Nil.
But if previous (GET) request returns Cookies, it will not be Nil.
And after Client.Cookies.Free it will not be Nil.
As a result, next POST request contains empty Cookies header, causing exception.

@lucamar
MyHTTPClient.Cookies := Nil    causes SIGSEGV error, I guess because FCookies is declared private.

The very bad solution that I have found to make my application working with REST API without errors is to call MyHTTPClient.Free after each request.
« Last Edit: February 06, 2021, 09:13:08 pm by Vodnik »

devEric69

  • Sr. Member
  • ****
  • Posts: 433
Re: Problem with POST [BUG discovered]
« Reply #13 on: February 07, 2021, 10:50:20 am »
Overall, from memory, we do not really need to explicitely destroy the cookie object (TCookie) itself, with fpWeb. It's fpWeb that does it for us: for example, when we want to close a web session, we have to send back its corresponding cookie empty, towards the browser, by calling:

Code: Pascal  [Select][+][-]
  1. Self.Session.Terminate;
  2. Self.Session.InitResponse(AResponse); { it returns an empty web session cookie to the client browser }

Or, we can put its value with ''.




However, inversely, we may need to create a cookie, if there are none coming from the client in the Cookies "channel", and that it's no longer contextually normal:

Code: Pascal  [Select][+][-]
  1. [snip]
  2. o_cookie:= AResponse.Cookies.FindCookie('fpWebSessionName'); { my fpWeb web session name; in Php, it would have been PHP_SESSID }
  3. if o_cookie=nil then begin
  4.     o_cookie:= AResponse.Cookies.Add; { ask for a new cookie }
  5.     o_cookie.Name:= 'fpWebSessionName';
  6. end
  7. else begin
  8.   if FbSession_AnonymousOrNamed_started then begin { i'm using an inherited from TCustomSession }
  9.     o_cookie.Value:= GetGUID; { or anything else }
  10.     o_cookie.Path:= '/';  { if it must cover all paths on our web Domain Name Server }
  11.   end
  12.   else If FbSession_AnonymousOrNamed_Terminated then begin
  13.     o_cookie.Value:= ''; { send an invalid web-cookie-session }
  14.     o_cookie.Path:= '';
  15. end;
  16. [snip]
  17.  
« Last Edit: February 07, 2021, 12:20:56 pm by devEric69 »
use: Linux 64 bits (Ubuntu 20.04 LTS).
Lazarus version: 2.0.4 (svn revision: 62502M) compiled with fpc 3.0.4 - fpDebug \ Dwarf3.

Vodnik

  • Full Member
  • ***
  • Posts: 184
Re: Problem with POST [BUG discovered]
« Reply #14 on: February 07, 2021, 08:08:19 pm »
Bug report 0038450: "Host" header is duplicated in POST request if one is already set.
« Last Edit: February 08, 2021, 02:04:59 pm by Vodnik »

 

TinyPortal © 2005-2018