[SOLVED] Indy HTTP - Persistent Connections & Data Compression

[JD]

Hi there everyone,

I have an Indy 10 based REST server that provides data to clients (browsers & desktop clients). The server has its KeepAlive property set to true so as to persist connections.

When a desktop client talks to the server, it creates a new thread and a new instance of IdHTTP as in the example below:

Code: Pascal  [Select][+][-]

1. function TRestThread.Get(AURL: string): string;
2. var
3.   AString: string;
4. begin
5.   //
6.   Result := EmptyStr;
7.   AString := EmptyStr;
8.   //
9.   IdHTTP := TIdHTTP.Create(nil);
10.   IdHTTP.Request.UserAgent        := 'MyApp';
11.   IdHTTP.Request.Connection       := 'keep-alive';
12.   IdHTTP.Request.ContentType      := 'application/json';
13.   IdHTTP.Request.ContentEncoding  := 'UTF-8';
14.   IdHTTP.Request.Accept           := 'application/json';
15.   IdHttp.Request.AcceptLanguage   := 'fr, fr-FR';
16.   IdHttp.HTTPOptions              := [hoForceEncodeParams];
17.   IdHttp.Request.AcceptEncoding   := 'gzip, deflate';
18.  
19.   //
20.   try
21.     // did the server send a gzip response?
22.     if Pos('gzip', IdHttp.Request.AcceptEncoding) > 0 then
23.     begin
24.       AString := IdHttp.Get(AURL, IndyTextEncoding_UTF8);
25.       Result := UTF8Decode(ZDecompressStr(AString));
26.     end
27.     else
28.       Result := UTF8Decode(IdHttp.Get(AURL, IndyTextEncoding_UTF8));
29.   finally
30.     IdHTTP.Free;
31.   end;
32. end;
33.  

Sometimes I have to call this GET procedure as much as 20 times to fill many screen controls. As you can see, IdHTTP is created/freed in the GET function.

I must confess I opted for this create-free method because of users with mobile devices who can move from one WiFi network to another just by walking around. It ensures that I don't have a fixed connection tunnel like TCP/IP which fails when the user's WiFi access changes. IdTCPClient/IdTCPServer combo will require reconnection when user's WiFi changes or even when the server has to be restarted. The downside of this IdHTTP / REST Server method is that it is slower than the IdTCPClient/IdTCPServer combo.

My question is this, does this create-free method for IdHTTP make persistent connections useless?

Thanks,

JD

[Remy Lebeau]

I have an Indy 10 based REST server that provides data to clients (browsers & desktop clients). The server has its KeepAlive property set to true so as to persist connections.

That does you no good in this situation since the code you showed is not taking advantage of persistent connections.

When a desktop client talks to the server, it creates a new thread and a new instance of IdHTTP as in the example below:

Setting the Request.Connection property to 'keep-alive' is wasted in your example since your HTTP object is being freed, and thus disconnecting, when the request is finished.

Also, UTF-8 is not a valid ContentEncoding.

Also, since you clearly want to support compressed responses, you should assign a TIdZLibCompressorBase-derived component, like TIdCompressorZLib, to the TIdHTTP.Compressor property, and let Indy decompress the data for you.  However, if you are going to do it manually, you should be looking at the Response.ContentEncoding property to detect a compressed response, not the Request.AcceptEncoding property.  And worse, you are indicating that you support both 'gzip' and 'deflate' compression, but you are not actually checking for 'deflate'-encoded responses, only 'gzip'.  They are two separate compressions!  All the more reason not to assign anything to Request.AcceptEncoding at all.  TIdHTTP handles that for you when a Compressor is assigned.

But, in any case, definitely DO NOT process compressed data as a String!  It is binary data, not textual data.  Charset handling will corrupt the compressed bytes.  Use the overloaded version of TIdHTTP.Get() that fills an output TStream instead, and then decompress that if compressed.

Sometimes I have to call this GET procedure as much as 20 times to fill many screen controls. As you can see, IdHTTP is created/freed in the GET function.

I must confess I opted for this create-free method because of users with mobile devices who can move from one WiFi network to another just by walking around. It ensures that I don't have a fixed connection tunnel like TCP/IP which fails when the user's WiFi access changes.

You do realize that HTTP runs on top of TCP/IP, don't you?  And your code would still fail if the user happens to change WiFi in the *middle* of an HTTP retrieval.

IdTCPClient/IdTCPServer combo will require reconnection when user's WiFi changes or even when the server has to be restarted.

So does TIdHTTP/TIdHTTPServer, since HTTP runs on top of TCP/IP.

The downside of this IdHTTP / REST Server method is that it is slower than the IdTCPClient/IdTCPServer combo.

Not really, since TIdHTTP derives from TIdTCPClientCustom, and TIdHTTPServer derives from TIdCustomTCPServer, so you still have all of the same TCP/IP semantics.

My question is this, does this create-free method for IdHTTP make persistent connections useless?

Yes.  To really take advantage of persistent connections, you have to keep the TIdHTTP object alive and reuse it for subsequent requests.  For instance, you could have a pool of TIdHTTP objects, and have your REST thread pull an available object from the pool, use it as needed, and then put it back in the pool when finished.

[JD]

Thanks for your reply, Remy.

That does you no good in this situation since the code you showed is not taking advantage of persistent connections.

Setting the Request.Connection property to 'keep-alive' is wasted in your example since your HTTP object is being freed, and thus disconnecting, when the request is finished.

That's what I thought was going on. Thanks for confirming it.

Also, since you clearly want to support compressed responses, you should assign a TIdZLibCompressorBase-derived component, like TIdCompressorZLib, to the TIdHTTP.Compressor property, and let Indy decompress the data for you.  However, if you are going to do it manually, you should be looking at the Response.ContentEncoding property to detect a compressed response, not the Request.AcceptEncoding property.  And worse, you are indicating that you support both 'gzip' and 'deflate' compression, but you are not actually checking for 'deflate'-encoded responses, only 'gzip'.  They are two separate compressions!  All the more reason not to assign anything to Request.AcceptEncoding at all.  TIdHTTP handles that for you when a Compressor is assigned.

But, in any case, definitely DO NOT process compressed data as a String!  It is binary data, not textual data.  Charset handling will corrupt the compressed bytes.  Use the overloaded version of TIdHTTP.Get() that fills an output TStream instead, and then decompress that if compressed.

OK. But I am not clear about how IdHTTP compression and decompression works automagically with TIdCompressorZLib. On the server, I can use the methods in IdZlib to handle compression/decompression. When I tried to use IdHTTP with TIdCompressorZLib, I kept getting errors concerning inflate (don't remember the exact term used).

For instance at the moment on the server side - CommandGet and CommandOther, here is how I handle compression of strings (before the transition to streams you suggested).

Code: Pascal  [Select][+][-]

1.   // SERVER SIDE CommandGet and CommandOther code snippet
2.  
3.   // Should serve the entire procedure so move compression stuff here
4.   if strJSON <> EmptyStr then
5.   begin
6.     //
7.     AResponseInfo.ResponseNo := 200;
8.     AResponseInfo.ContentType := 'application/json';
9.     AResponseInfo.CharSet := 'utf-8';
10.  
11.     // Does the client request demand a compressed response?
12.     //
13.     if Pos('gzip', ARequestInfo.AcceptEncoding) > 0 then
14.     begin
15.       AResponseInfo.ContentEncoding := 'gzip';
16.       //
17.       if Pos('MyApp', ARequestInfo.UserAgent) > 0 then
18.         strCompressedJSON := RUtils.ZCompressStr(strJSON, Zstream.TCompressionlevel.clMax)
19.       else  // browsers will be able to handle gzip compression
20.         strCompressedJSON := IdZlib.CompressString(strJSON, IdZlib.TCompressionLevel.clMax, zsGZip);
21.       AResponseInfo.ContentStream := TIdMemoryBufferStream.Create(Pointer(strCompressedJSON), Length(strCompressedJSON));
22.     end
23.     else
24.       AResponseInfo.ContentStream := TIdMemoryBufferStream.Create(Pointer(strJSON), Length(strJSON));
25.   end
26.   else
27.   begin
28.     AResponseInfo.ResponseNo  := 404;        
29.     AResponseInfo.ContentType := 'text/html';
30.     AResponseInfo.CharSet     := 'utf-8';
31.     AResponseInfo.ContentText := strNotFound;
32.   end;    // if strJSON <> EmptyStr then
33.  
34.   // Send the string to the client  (here it works with string compression method).
35.   // One Write for all decision branches
36.   AResponseInfo.WriteContent;
37.  

Can you please show me how I can change this to handle stream compression using Indy only on the server as well as the client?

Not really, since TIdHTTP derives from TIdTCPClientCustom, and TIdHTTPServer derives from TIdCustomTCPServer, so you still have all of the same TCP/IP semantics.

I know HTTP runs on top of TCP/IP. However I discovered that I did not have to explicitly Connect to the server like I do in TCP/IP before using GET/POST/PUT/DELETE. Maybe an implicit Connect is done, I did not try to find out.

Yes.  To really take advantage of persistent connections, you have to keep the TIdHTTP object alive and reuse it for subsequent requests.  For instance, you could have a pool of TIdHTTP objects, and have your REST thread pull an available object from the pool, use it as needed, and then put it back in the pool when finished.

I have one for an older application using TCP/IP. In this HTTP connection pool, does that mean I will explicitly have to call Connect/Disconnect when the pool is created/destroyed like for the TCP/IP pool? Sounds obvious I know but it helps to be clear. 

JD

[rvk]

Sometimes I have to call this GET procedure as much as 20 times to fill many screen controls.

I hope you don't call this procedure 20 times for just one screen. Because if you do it might be useful to redesign it and post ONE complete json with all the information you want and get ONE json back with that information. In that case compression will also be able to do it's work because compressing a few bytes isn't that optimal. Compression is more useful the bigger the information gets.

[JD]

Sometimes I have to call this GET procedure as much as 20 times to fill many screen controls.

I hope you don't call this procedure 20 times for just one screen. Because if you do it might be useful to redesign it and post ONE complete json with all the information you want and get ONE json back with that information. In that case compression will also be able to do it's work because compressing a few bytes isn't that optimal. Compression is more useful the bigger the information gets.

You read my mind  I'm actually working on that right now. That's why I needed the compression. The 20 calls model was a prototype.

JD

[Remy Lebeau]

OK. But I am not clear about how IdHTTP compression and decompression works automagically with TIdCompressorZLib.

What is unclear?  If you assign a Compressor to TIdHTTP, and then TIdHTTP detects a compressed response, it will automatically decompress the response for you.  The data returned to you (String or TStream) will be the uncompressed data.

On the server, I can use the methods in IdZlib to handle compression/decompression.

Yes (if the client requests compression).

When I tried to use IdHTTP with TIdCompressorZLib, I kept getting errors concerning inflate (don't remember the exact term used).

Then you are likely not using it correctly.  But you have not shown that code or the error, so I can't help you troubleshoot it.

For instance at the moment on the server side - CommandGet and CommandOther, here is how I handle compression of strings (before the transition to streams you suggested).

My suggestion to use a stream only applies to the client side.  And even then, that doesn't change the data itself, only how TIdHTTP delivers the data to you.  But, if you assign a Compressor to TIdHTTP, then you don't even need to change your client code, as the String-returning version of Get() will simply decompress the data internally before it converts the decompressed data to a String for output.

Why are you using two different ZLib compressors on the server side, though?  GZip is GZip regardless of the library used to implement it.  There is no reason to use different compressors.

Can you please show me how I can change this to handle stream compression using Indy only on the server as well as the client?

The server code you showed is fine, you don't really need to change it.  Though I would suggest you decide on one ZLib library or another and stick with it.

I know HTTP runs on top of TCP/IP. However I discovered that I did not have to explicitly Connect to the server like I do in TCP/IP before using GET/POST/PUT/DELETE. Maybe an implicit Connect is done, I did not try to find out.

Yes, TIdHTTP calls Connect() internally for you.

I have one for an older application using TCP/IP. In this HTTP connection pool, does that mean I will explicitly have to call Connect/Disconnect when the pool is created/destroyed like for the TCP/IP pool?

No.

[JD]

What is unclear?  If you assign a Compressor to TIdHTTP, and then TIdHTTP detects a compressed response, it will automatically decompress the response for you.  The data returned to you (String or TStream) will be the uncompressed data.

OK. Here is what I did.

The server side CommandGet code using stream compression is as follows. This works even in my browser 

Code: Pascal  [Select][+][-]

1.     // Does the client request demand a compressed response?
2.     if Pos('gzip', ARequestInfo.AcceptEncoding) > 0 then
3.     begin
4.       AResponseInfo.ContentEncoding := 'gzip';
5.       // Copy strJSON into a stream and then compress the stream
6.       // into AResponseInfo.ContentStream in preparation for sending to
7.       // clients
8.       AResponseInfo.ContentStream := TMemoryStream.Create;
9.       JSONStream := TStringStream.Create(strJSON);
10.       try
11.         JSONStream.Position := 0;
12.         IdZlib.CompressStreamEx(JSONStream, AResponseInfo.ContentStream, IdZlib.TCompressionLevel.clMax, zsGzip);
13.         AResponseInfo.ContentStream.Position := 0;
14.       finally
15.         JSONStream.Free;
16.       end;
17.     end
18.     else
19.       AResponseInfo.ContentStream := TIdMemoryBufferStream.Create(Pointer(strJSON), Length(strJSON));
20.  

The client side code is as follows:

Code: Pascal  [Select][+][-]

1. procedure TForm1.btnTestGzipClick(Sender: TObject);
2.   function Get(AURL: string): string;
3.   var
4.     AString: string;
5.     ResultStream: TStringStream;
6.     var IdHttp: TIdHTTP;
7.   begin
8.     //
9.     Result := EmptyStr;
10.     AString := EmptyStr;
11.     //
12.     IdHTTP                          := TIdHTTP.Create(nil);
13.     IdHTTP.Compressor               := TIdCompressorZlib.Create(IdHTTP);
14.     IdHTTP.Request.ContentType      := 'application/json';
15.     IdHTTP.Request.Accept           := 'application/json';
16.     IdHttp.Request.AcceptLanguage   := 'fr, fr-FR';
17.     IdHttp.HTTPOptions              := [hoForceEncodeParams];
18.     IdHttp.Request.AcceptEncoding   := 'gzip';
19.  
20.     //
21.     try
22.       // PART 1
23.       Result := IdHttp.Get(AURL, IndyTextEncoding_UTF8);    
24.  
25.       //// PART 2
26.       //// Get the data from the server
27.       // IdHttp.Get(AURL, IndyTextEncoding_UTF8);
28.       //// did the server send a gzip response?
29.       //if Pos('gzip', IdHttp.Response.ContentEncoding) > 0 then
30.       //begin
31.       //  ResultStream := TStringStream.Create('');
32.       //  IdHttp.Response.ContentStream := TMemoryStream.Create;
33.       //  //
34.       //  try
35.       //    IdHttp.Compressor.DecompressGZipStream(IdHttp.Response.ContentStream, ResultStream);
36.       //    ResultStream.Position := 0;
37.       //    Result := UTF8Decode(ResultStream.DataString);
38.       //  finally
39.       //    ResultStream.Free;
40.       //  end;
41.       //end
42.       //else
43.       //  Result := UTF8Decode(AString);
44.     finally
45.       IdHTTP.Compressor := nil;
46.       IdHTTP.Compressor.Free;
47.       IdHTTP.Free;
48.     end;
49.   end;
50.  
51. begin
52.   //
53.   ShowMessage(Get('http://localhost:80/countries'));
54. end;
55.  

You said HTTP.Compressor handle decompression (string/stream) internally. The results of PART 1 and of PART 2 (the commented out portion) are shown in the attached screenshots.

Where did I go wrong on the client side? The browser rendering of the same URL is perfect so the problem can only be from the way I'm using HTTP.Compressor.

Thanks a lot,

JD

[JD]

Very, very strange. I had a hunch so I fired up Delphi XE4 to try the same thing.

And lo and behold, it worked!    . All I had to do was to change

Result := IdHttp.Get(AURL, IndyTextEncoding_UTF8);

to

Result := IdHttp.Get(AURL);

It fails in Lazarus 1.6.2 BUT works in Delphi XE4. Reminds me of my last text encoding problem before GetMem had the idea to try it in his Delphi XE4.

The HTTP server was built using Lazarus 1.6.2.

What do I do now?

JD

[Remy Lebeau]

The server side CommandGet code using stream compression is as follows. This works even in my browser 

Looks fine on the server side, though I would use ContentText instead of ContentStream for the non-compresssed situation:

Code: [Select]

AResponseInfo.ContentText := strJSON;


The client side code is as follows:

I see three issues:

1. double-check to make sure that the TIdCompressorZLib.IsReady property is returning true.  If it is not, then TIdCompressorZLib is not loading the ZLib library correctly.  I did have to check in a bug fix earlier this year regarding this.

2. Don't set the TIdHTTP.Request.AcceptEncoding at all.  I told you that earlier.  Let TIdHTTP handle that assignment internally when a Compressor is assigned, based on whether the Compressor is actually ready for use or not.  By assigning the property manually, you are potentially telling the server that you will accept a compressed response even if the Compressor is not able to actually decompress it.

3. if all goes well with the Compressor then the data returned by Get() will always be uncompressed:

Code: Pascal  [Select][+][-]

1. function Get(AURL: string): string;
2. var
3.   IdHttp: TIdHTTP;
4. begin
5.   Result := EmptyStr;
6.   IdHTTP := TIdHTTP.Create(nil);
7.   try
8.     IdHTTP.Compressor := TIdCompressorZlib.Create(IdHTTP);
9.     IdHTTP.Request.Accept := 'application/json';
10.     IdHttp.Request.AcceptLanguage := 'fr, fr-FR';
11.     Result := IdHttp.Get(AURL, IndyTextEncoding_UTF8);    
12.   finally
13.     IdHttp.Free;
14.   end;
15. end;
16.  

Code: Pascal  [Select][+][-]

1. function Get(AURL: string): string;
2. var
3.   ResultStream: TMemoryStream;
4.   IdHttp: TIdHTTP;
5. begin
6.   Result := EmptyStr;
7.   ResultStream := TMemoryStream.Create;
8.   try
9.     IdHTTP := TIdHTTP.Create(nil);
10.     try
11.       IdHTTP.Compressor := TIdCompressorZlib.Create(IdHTTP);
12.       IdHTTP.Request.Accept := 'application/json';
13.       IdHttp.Request.AcceptLanguage := 'fr, fr-FR';
14.       IdHttp.Get(AURL, ResultStream);
15.     finally
16.       IdHTTP.Free;
17.     end;
18.     ResultStream.Position := 0;
19.     Result := ReadStringFromStream(ResultStream, -1, IndyTextEncoding_UTF8);
20.   finally
21.     ResultStream.Free;
22.   end;
23. end;
24.  

You said HTTP.Compressor handle decompression (string/stream) internally.

Yes, when things go correctly.

[JD]

It now works. Removing the line IdHttp.Request.AcceptEncoding   := 'gzip'; made the difference.

However, for the second function using streams, it was necessary to change
Result := ReadStringFromStream(ResultStream, -1, IndyTextEncoding_UTF8);
to
Result := ReadStringFromStream(ResultStream, -1);
so that the strings could be properly rendered.

As usual, we start on one subject and end up examining other areas as well . I have altered the title of this thread to better describe what we discussed.

Thanks a lot for your valuable input and code review.

Merry Christmas to you,

JD

EDIT: at 15h23 CET

Sorry but it seems I was too hasty in my conclusions. Refactoring the server side code and removing the following line from the server's CommandGet procedure
 
if Pos('gzip', ARequestInfo.AcceptEncoding) > 0

made me realise that that was why it had worked like I described earlier today. It was actually sending uncompressed data. Now I only want it to return compressed streams and my Lazarus client application fails to decompress the streams.

I updated my Indy intallation from indy.fulgan.com and tried it again. It failed!
I then discovered that IdHTTP.Compressor.IsReady returns FALSE but only in Lazarus. It returns TRUE in Delphi.

I apologize to all who have read the previous original post for this oversight. I was in a hurry to start the weekend.

JD

[JD]

So my search for a solution continues.

[rvk]

If  IdHTTP.Compressor.IsReady returns FALSE there doesn't seem to be any decompressor loaded. .IsReady checks for IdZLibHeaders.Loaded. I'm not sure how it's done in Indy and Delphi or Lazarus but the (de-)compression is done with idzlibheaders.pas.

If I look in there I see

Code: Pascal  [Select][+][-]

1. {
2. TODO:  Wait for Emb to decide how to approach ZLib for their 64-bit support
3. before we proceed at our end.
4. }
5. {$UNDEF STATICLOAD_ZLIB}

This is in Indy 10.6.2.0 (I used the one in onlinepackagemanager)

So I think the .obj may not be included in the compilation. It could be that zlib1.dll is needed for decompression in that case.

What Indy version do you have on Delphi where .isReady is true?
If it's also 10.6.2.0 could you check line 22-26 if it also has these comments mentioned above and the UNDEF for STATICLOAD_ZLIB?

Edit: Yep. I just confirmed that adding a zlib1.dll in your project-directory will result in IdHTTP.Compressor.IsReady being TRUE.
The latest version on indy.fulgan.com (Indy10_5388) still has that comment.

[JD]

What Indy version do you have on Delphi where .isReady is true?
If it's also 10.6.2.0 could you check line 22-26 if it also has these comments mentioned above and the UNDEF for STATICLOAD_ZLIB?

The default Indy version in Delphi XE4. It has to be in the 10.6 series.

Edit: Yep. I just confirmed that adding a zlib1.dll in your project-directory will result in IdHTTP.Compressor.IsReady being TRUE.
The latest version on indy.fulgan.com (Indy10_5388) still has that comment.

It worked!!!! Thanks a million.    FYI I'm using Indy10_5388 too.

Merry Christmas to you rvk.

JD