Recent

Author Topic: Indy Http GET problem  (Read 1793 times)

xinyiman

  • Hero Member
  • *****
  • Posts: 1752
Indy Http GET problem
« on: August 05, 2018, 11:32:24 am »
Hi guys, I'm creating a program with lazarus and indy to get a data transfer through the http GET. In practice I need to make a site with authentication via json.

Basically a function checks the validity of the credentials and returns this string to me

Code: Pascal  [Select]
  1. { "token" : "ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGTU9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk11RnBzWFlGZ2hBWThJQThxaTBrLy9MQlBhU1loeTVCYzI1VzdWcTBFVGEyRGxSMkV3WkR5d01yVmJyekM4T2tPOW9SUzRuZ3hLTkVhNkdHR3JVdklnRk9wdVFuSGdyY3BOUG0xU25FTFFFOFdtdkg5dFdGYTE1cnJSZUZ1Uy9GZ2UzREY1cz0=" }
  2.  

But when I use this code to recover the data

Code: Pascal  [Select]
  1. function TNGITController.GetHTML(url: string): string;
  2. var
  3.    idhttp1:TIdHTTP;
  4.    xpto:tmemorystream;
  5.    ret: string;
  6. begin
  7.      ret:='';
  8.      idhttp1:=TIdHTTP.Create;
  9.      xpto:=tmemorystream.Create;
  10.      idhttp1.ReadTimeout:=5000;
  11.      idhttp1.Get(url,xpto);
  12.      xpto.Position:=0;
  13.      SetString(ret, PAnsiChar(xpto.Memory), xpto.Size);
  14.      xpto.Free;
  15.      idhttp1.Free;
  16.      result:=ret;
  17. end;
  18.  

I get this result

Code: Pascal  [Select]
  1. json login: HTTP/1.1 200 OK
  2. Connection: close
  3. Content-Length: 396
  4. Server: NGIT Bootstrap Application
  5. Set-Cookie: IDHTTPSESSIONID=5Ltho7JkF8YkQaw; path=/
  6.  
  7. { "token" : "ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGTU9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk11RnBzWFlGZ2hBWThJQThxaTBrLy9MQlBhU1l
  8.  

But it does not always happen, only occasionally. It is also fair to tell you that the token is sent via the AResponseInfo object: TIdHTTPResponseInfo

Like this: AResponseInfo.ContentText

Where am I doing wrong?
Ubuntu and Mac
Lazarus: 1.8.0
FPC: 3.0.4

xinyiman

  • Hero Member
  • *****
  • Posts: 1752
Re: Indy Http GET problem
« Reply #1 on: August 05, 2018, 12:25:56 pm »
I tried replacing the GetHTML function with some synapse code. And it does not seem to give any more problems. I would not want there to be any bugs in indy for reading HTML. I forgot the problem manifests itself on Mac OSX
Ubuntu and Mac
Lazarus: 1.8.0
FPC: 3.0.4

Thaddy

  • Hero Member
  • *****
  • Posts: 7183
Re: Indy Http GET problem
« Reply #2 on: August 05, 2018, 02:12:59 pm »
I just wrote some examples here http://forum.lazarus.freepascal.org/index.php/topic,42132.msg293506/topicseen.html#new
that do just what you want. Either synapse or fcl-web. 4 examples in total.
inline variables like in D10.3 are a bit like Brexit: if you are given the wrong information it sounds like a good idea. Every kid loves candy, but it makes you fat and your teeth will disappear.

xinyiman

  • Hero Member
  • *****
  • Posts: 1752
Re: Indy Http GET problem
« Reply #3 on: August 06, 2018, 12:18:50 pm »
Thank you, but with synapse I can do it. It amazes me that indy, however, has the problem described by me.
Ubuntu and Mac
Lazarus: 1.8.0
FPC: 3.0.4

Thaddy

  • Hero Member
  • *****
  • Posts: 7183
Re: Indy Http GET problem
« Reply #4 on: August 06, 2018, 01:15:01 pm »
Thank you, but with synapse I can do it. It amazes me that indy, however, has the problem described by me.
If (after)I have quite a lot of ice cold beers I will probably add Indy to the examples too. There isn't much difference and also at max. a 4 liner to handle the basics.
Maybe Remy beats me to it?
inline variables like in D10.3 are a bit like Brexit: if you are given the wrong information it sounds like a good idea. Every kid loves candy, but it makes you fat and your teeth will disappear.

engkin

  • Hero Member
  • *****
  • Posts: 2129
Re: Indy Http GET problem
« Reply #5 on: August 06, 2018, 03:20:53 pm »
Code: Pascal  [Select]
  1. json login: HTTP/1.1 200 OK
Is that a space between "json" and "login"?

Edit:
If that is line is the "status-line" then the server is not responding correctly. It should be something like:
Quote
HTTP/1.1 200 OK json login

What do I know!

@Thaddy, it is getting hot over there.
« Last Edit: August 06, 2018, 03:41:06 pm by engkin »

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 515
    • Lebeau Software
Re: Indy Http GET problem
« Reply #6 on: August 07, 2018, 09:34:19 pm »
Basically a function checks the validity of the credentials and returns this string to me

You are not sending any credentials, at least not with any HTTP authentication anyway.  Are they in your URL string instead?  That is not very secure, unless HTTPS is being used.

Also, why are you using the overloaded version of TIdHTTP.Get() that fills a TStream, rather than using the version of TIdHTTP.Get() that returns a String?

Code: Pascal  [Select]
  1. function TNGITController.GetHTML(url: string): string;
  2. var
  3.    idhttp1: TIdHTTP;
  4. begin
  5.   Result := '';
  6.   idhttp1 := TIdHTTP.Create;
  7.   try
  8.     idhttp1.ReadTimeout := 5000;
  9.     Result := idhttp1.Get(url);
  10.   finally
  11.      idhttp1.Free;
  12.   end;
  13. end;
  14.  

But when I use this code to recover the data
...
I get this result

I don't understand what you are describing.  You are clearly getting a success reply from the server with a JSON payload, so what is the actual problem you are having?  Please be more specific.

Are you suggesting that the entire HTTP message, not just its body data, is being stored in the TMemoryStream?  I guarantee that is not how TIdHTTP.get() works.

But it does not always happen, only occasionally.

WHAT doesn't always happen?

It is also fair to tell you that the token is sent via the AResponseInfo object: TIdHTTPResponseInfo

Like this: AResponseInfo.ContentText

How the server end is implemented is not relevant to this issue, unless the server is not populating the AResponseInfo correctly.

Where am I doing wrong?

We can't know, because you haven't described the problem clearly enough.
« Last Edit: August 07, 2018, 09:36:12 pm by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

xinyiman

  • Hero Member
  • *****
  • Posts: 1752
Re: Indy Http GET problem
« Reply #7 on: August 08, 2018, 08:18:39 am »

Also, why are you using the overloaded version of TIdHTTP.Get() that fills a TStream, rather than using the version of TIdHTTP.Get() that returns a String?


I tried both versions. In any case, the following problem occurs to me.
The function should always return this string

Code: Pascal  [Select]
  1.     { "token" : "ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGTU9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk11RnBzWFlGZ2hBWThJQThxaTBrLy9MQlBhU1loeTVCYzI1VzdWcTBFVGEyRGxSMkV3WkR5d01yVmJyekM4T2tPOW9SUzRuZ3hLTkVhNkdHR3JVdklnRk9wdVFuSGdyY3BOUG0xU25FTFFFOFdtdkg5dFdGYTE1cnJSZUZ1Uy9GZ2UzREY1cz0=" }

But arbitrarily every now and then (let's say once on 5/6) returns

Code: Pascal  [Select]
  1.     json login: HTTP/1.1 200 OK
  2.     Connection: close
  3.     Content-Length: 396
  4.     Server: NGIT Bootstrap Application
  5.     Set-Cookie: IDHTTPSESSIONID=5Ltho7JkF8YkQaw; path=/
  6.      
  7.     { "token" : "ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGTU9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk11RnBzWFlGZ2hBWThJQThxaTBrLy9MQlBhU1l
  8.      

Last result all into string. This is problem. I think the problem is into GET function because i use synapse GetHttp the result is always correct
Ubuntu and Mac
Lazarus: 1.8.0
FPC: 3.0.4

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 515
    • Lebeau Software
Re: Indy Http GET problem
« Reply #8 on: August 11, 2018, 03:05:30 am »
The function should always return this string

Code: Pascal  [Select]
  1.     { "token" : "ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGTU9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk11RnBzWFlGZ2hBWThJQThxaTBrLy9MQlBhU1loeTVCYzI1VzdWcTBFVGEyRGxSMkV3WkR5d01yVmJyekM4T2tPOW9SUzRuZ3hLTkVhNkdHR3JVdklnRk9wdVFuSGdyY3BOUG0xU25FTFFFOFdtdkg5dFdGYTE1cnJSZUZ1Uy9GZ2UzREY1cz0=" }

But arbitrarily every now and then (let's say once on 5/6) returns

Code: Pascal  [Select]
  1.     json login: HTTP/1.1 200 OK
  2.     Connection: close
  3.     Content-Length: 396
  4.     Server: NGIT Bootstrap Application
  5.     Set-Cookie: IDHTTPSESSIONID=5Ltho7JkF8YkQaw; path=/
  6.      
  7.     { "token" : "ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGTU9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk11RnBzWFlGZ2hBWThJQThxaTBrLy9MQlBhU1l
  8.      

That is simply not possible under normal conditions.  Especially the "json login: " portion, which is not even part of the HTTP protocol.  TIdHTTP DOES NOT store a response's headers in the output TStream, only the response's body.  The server would have to be sending that extra text in its response body, which would be a flaw on the server's part, not in TIdHTTP itself.  Which might also make sense, given that the base64 token in the erroneous response data doesn't match the base64 token in the correct response data.

To verify this, use a packet sniffer like Wireshark, or attach one of Indy's TIdLog... components (such as TIdLogFile) to the TIdHTTP.Intercept property.  That will log the raw data that TIdHTTP is receiving from the server.  Please post that data for review.
« Last Edit: August 11, 2018, 03:08:49 am by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

xinyiman

  • Hero Member
  • *****
  • Posts: 1752
Re: Indy Http GET problem
« Reply #9 on: October 24, 2018, 09:57:00 am »
Sorry for the delay, but at this link you find an example. If you compile it to the current state use both synapse and indy. Synapse is used only in unit_controller.pas in the GetHTML function.

http://forum.lazarus-ide.org/index.php/topic,42973.msg300168/topicseen.html#new

To replicate the problem, just change the function from this

Code: Pascal  [Select]
  1. function TNGITController.GetHTML(url: string): string;
  2. var
  3.    idhttp1:TIdHTTP;
  4.    xpto:tmemorystream;
  5.    ret: string;
  6. begin
  7.      {ret:='';
  8.      idhttp1:=TIdHTTP.Create;
  9.      xpto:=tmemorystream.Create;
  10.      idhttp1.ReadTimeout:=5000;
  11.      idhttp1.Get(url,xpto);
  12.      xpto.Position:=0;
  13.      SetString(ret, PAnsiChar(xpto.Memory), xpto.Size);
  14.      xpto.Free;
  15.      idhttp1.Free;
  16.      result:=ret;  }
  17.  
  18.      //in questo punto ho optato su synapse perchè il get di indy sembra non funzionare bene
  19.      //in alcuni casi toppa e non ho capito perchè, lo fa in maniera del tutto arbitraria
  20.  
  21.      ret:=RecuperaHtml(url);
  22.      result:=ret;
  23. end;
  24.  

to this
Code: Pascal  [Select]
  1. function TNGITController.GetHTML(url: string): string;
  2. var
  3.    idhttp1:TIdHTTP;
  4.    xpto:tmemorystream;
  5.    ret: string;
  6. begin
  7.      ret:='';
  8.      idhttp1:=TIdHTTP.Create;
  9.      xpto:=tmemorystream.Create;
  10.      idhttp1.ReadTimeout:=5000;
  11.      idhttp1.Get(url,xpto);
  12.      xpto.Position:=0;
  13.      SetString(ret, PAnsiChar(xpto.Memory), xpto.Size);
  14.      xpto.Free;
  15.      idhttp1.Free;
  16.      result:=ret;
  17. end;
  18.  
« Last Edit: October 24, 2018, 10:08:14 am by xinyiman »
Ubuntu and Mac
Lazarus: 1.8.0
FPC: 3.0.4