Indy Http GET problem

[xinyiman]

Hi guys, I'm creating a program with lazarus and indy to get a data transfer through the http GET. In practice I need to make a site with authentication via json.

Basically a function checks the validity of the credentials and returns this string to me

Code: Pascal  [Select][+][-]

1. { "token" : "ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGTU9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk11RnBzWFlGZ2hBWThJQThxaTBrLy9MQlBhU1loeTVCYzI1VzdWcTBFVGEyRGxSMkV3WkR5d01yVmJyekM4T2tPOW9SUzRuZ3hLTkVhNkdHR3JVdklnRk9wdVFuSGdyY3BOUG0xU25FTFFFOFdtdkg5dFdGYTE1cnJSZUZ1Uy9GZ2UzREY1cz0=" }
2.  

But when I use this code to recover the data

Code: Pascal  [Select][+][-]

1. function TNGITController.GetHTML(url: string): string;
2. var
3.    idhttp1:TIdHTTP;
4.    xpto:tmemorystream;
5.    ret: string;
6. begin
7.      ret:='';
8.      idhttp1:=TIdHTTP.Create;
9.      xpto:=tmemorystream.Create;
10.      idhttp1.ReadTimeout:=5000;
11.      idhttp1.Get(url,xpto);
12.      xpto.Position:=0;
13.      SetString(ret, PAnsiChar(xpto.Memory), xpto.Size);
14.      xpto.Free;
15.      idhttp1.Free;
16.      result:=ret;
17. end;
18.  

I get this result

Code: Pascal  [Select][+][-]

1. json login: HTTP/1.1 200 OK
2. Connection: close
3. Content-Length: 396
4. Server: NGIT Bootstrap Application
5. Set-Cookie: IDHTTPSESSIONID=5Ltho7JkF8YkQaw; path=/
6.  
7. { "token" : "ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGTU9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk11RnBzWFlGZ2hBWThJQThxaTBrLy9MQlBhU1l
8.  

But it does not always happen, only occasionally. It is also fair to tell you that the token is sent via the AResponseInfo object: TIdHTTPResponseInfo

Like this: AResponseInfo.ContentText

Where am I doing wrong?

[xinyiman]

I tried replacing the GetHTML function with some synapse code. And it does not seem to give any more problems. I would not want there to be any bugs in indy for reading HTML. I forgot the problem manifests itself on Mac OSX

[Thaddy]

I just wrote some examples here http://forum.lazarus.freepascal.org/index.php/topic,42132.msg293506/topicseen.html#new
that do just what you want. Either synapse or fcl-web. 4 examples in total.

[xinyiman]

Thank you, but with synapse I can do it. It amazes me that indy, however, has the problem described by me.

[Thaddy]

Thank you, but with synapse I can do it. It amazes me that indy, however, has the problem described by me.

If (after)I have quite a lot of ice cold beers I will probably add Indy to the examples too. There isn't much difference and also at max. a 4 liner to handle the basics.
Maybe Remy beats me to it?

[engkin]

Code: Pascal  [Select][+][-]

1. json login: HTTP/1.1 200 OK

Is that a space between "json" and "login"?

Edit:
If that is line is the "status-line" then the server is not responding correctly. It should be something like:

HTTP/1.1 200 OK json login

What do I know!

@Thaddy, it is getting hot over there.

[Remy Lebeau]

Basically a function checks the validity of the credentials and returns this string to me

You are not sending any credentials, at least not with any HTTP authentication anyway.  Are they in your URL string instead?  That is not very secure, unless HTTPS is being used.

Also, why are you using the overloaded version of TIdHTTP.Get() that fills a TStream, rather than using the version of TIdHTTP.Get() that returns a String?

Code: Pascal  [Select][+][-]

1. function TNGITController.GetHTML(url: string): string;
2. var
3.    idhttp1: TIdHTTP;
4. begin
5.   Result := '';
6.   idhttp1 := TIdHTTP.Create;
7.   try
8.     idhttp1.ReadTimeout := 5000;
9.     Result := idhttp1.Get(url);
10.   finally
11.      idhttp1.Free;
12.   end;
13. end;
14.  

But when I use this code to recover the data
...
I get this result

I don't understand what you are describing.  You are clearly getting a success reply from the server with a JSON payload, so what is the actual problem you are having?  Please be more specific.

Are you suggesting that the entire HTTP message, not just its body data, is being stored in the TMemoryStream?  I guarantee that is not how TIdHTTP.get() works.

But it does not always happen, only occasionally.

WHAT doesn't always happen?

It is also fair to tell you that the token is sent via the AResponseInfo object: TIdHTTPResponseInfo

Like this: AResponseInfo.ContentText

How the server end is implemented is not relevant to this issue, unless the server is not populating the AResponseInfo correctly.

Where am I doing wrong?

We can't know, because you haven't described the problem clearly enough.

[xinyiman]

Also, why are you using the overloaded version of TIdHTTP.Get() that fills a TStream, rather than using the version of TIdHTTP.Get() that returns a String?

I tried both versions. In any case, the following problem occurs to me.
The function should always return this string

Code: Pascal  [Select][+][-]

1.     { "token" : "ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGTU9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk11RnBzWFlGZ2hBWThJQThxaTBrLy9MQlBhU1loeTVCYzI1VzdWcTBFVGEyRGxSMkV3WkR5d01yVmJyekM4T2tPOW9SUzRuZ3hLTkVhNkdHR3JVdklnRk9wdVFuSGdyY3BOUG0xU25FTFFFOFdtdkg5dFdGYTE1cnJSZUZ1Uy9GZ2UzREY1cz0=" }

But arbitrarily every now and then (let's say once on 5/6) returns

Code: Pascal  [Select][+][-]

1.     json login: HTTP/1.1 200 OK
2.     Connection: close
3.     Content-Length: 396
4.     Server: NGIT Bootstrap Application
5.     Set-Cookie: IDHTTPSESSIONID=5Ltho7JkF8YkQaw; path=/
6.      
7.     { "token" : "ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGTU9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk11RnBzWFlGZ2hBWThJQThxaTBrLy9MQlBhU1l
8.      

Last result all into string. This is problem. I think the problem is into GET function because i use synapse GetHttp the result is always correct

[Remy Lebeau]

The function should always return this string

Code: Pascal  [Select][+][-]

1.     { "token" : "ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGTU9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk11RnBzWFlGZ2hBWThJQThxaTBrLy9MQlBhU1loeTVCYzI1VzdWcTBFVGEyRGxSMkV3WkR5d01yVmJyekM4T2tPOW9SUzRuZ3hLTkVhNkdHR3JVdklnRk9wdVFuSGdyY3BOUG0xU25FTFFFOFdtdkg5dFdGYTE1cnJSZUZ1Uy9GZ2UzREY1cz0=" }

But arbitrarily every now and then (let's say once on 5/6) returns

Code: Pascal  [Select][+][-]

1.     json login: HTTP/1.1 200 OK
2.     Connection: close
3.     Content-Length: 396
4.     Server: NGIT Bootstrap Application
5.     Set-Cookie: IDHTTPSESSIONID=5Ltho7JkF8YkQaw; path=/
6.      
7.     { "token" : "ZVRoQWN0aFF6UExkdmQ4VTVlcmd4VGkxM1duQVdpUFFydnBESGJ4eUhQTUtTUUhoNldGTU9sWlFscjhna0JZdXlaeTBiWFVaWSs5a1dVa0RtbnJtN1NPVm1RT25sd1VuLzdINlBHcUl4NWtoVjl0bFJJWXZ6blFDYytrMEdseStjRitwNFlRY1U4WWJBdjc2U1pXbk11RnBzWFlGZ2hBWThJQThxaTBrLy9MQlBhU1l
8.      

That is simply not possible under normal conditions.  Especially the "json login: " portion, which is not even part of the HTTP protocol.  TIdHTTP DOES NOT store a response's headers in the output TStream, only the response's body.  The server would have to be sending that extra text in its response body, which would be a flaw on the server's part, not in TIdHTTP itself.  Which might also make sense, given that the base64 token in the erroneous response data doesn't match the base64 token in the correct response data.

To verify this, use a packet sniffer like Wireshark, or attach one of Indy's TIdLog... components (such as TIdLogFile) to the TIdHTTP.Intercept property.  That will log the raw data that TIdHTTP is receiving from the server.  Please post that data for review.

[xinyiman]

Sorry for the delay, but at this link you find an example. If you compile it to the current state use both synapse and indy. Synapse is used only in unit_controller.pas in the GetHTML function.

http://forum.lazarus-ide.org/index.php/topic,42973.msg300168/topicseen.html#new

To replicate the problem, just change the function from this

Code: Pascal  [Select][+][-]

1. function TNGITController.GetHTML(url: string): string;
2. var
3.    idhttp1:TIdHTTP;
4.    xpto:tmemorystream;
5.    ret: string;
6. begin
7.      {ret:='';
8.      idhttp1:=TIdHTTP.Create;
9.      xpto:=tmemorystream.Create;
10.      idhttp1.ReadTimeout:=5000;
11.      idhttp1.Get(url,xpto);
12.      xpto.Position:=0;
13.      SetString(ret, PAnsiChar(xpto.Memory), xpto.Size);
14.      xpto.Free;
15.      idhttp1.Free;
16.      result:=ret;  }
17.  
18.      //in questo punto ho optato su synapse perchè il get di indy sembra non funzionare bene
19.      //in alcuni casi toppa e non ho capito perchè, lo fa in maniera del tutto arbitraria
20.  
21.      ret:=RecuperaHtml(url);
22.      result:=ret;
23. end;
24.  

to this

Code: Pascal  [Select][+][-]

1. function TNGITController.GetHTML(url: string): string;
2. var
3.    idhttp1:TIdHTTP;
4.    xpto:tmemorystream;
5.    ret: string;
6. begin
7.      ret:='';
8.      idhttp1:=TIdHTTP.Create;
9.      xpto:=tmemorystream.Create;
10.      idhttp1.ReadTimeout:=5000;
11.      idhttp1.Get(url,xpto);
12.      xpto.Position:=0;
13.      SetString(ret, PAnsiChar(xpto.Memory), xpto.Size);
14.      xpto.Free;
15.      idhttp1.Free;
16.      result:=ret;
17. end;
18.