* * *

Author Topic: idHTTP and Accept-Encoding  (Read 1521 times)

RDL

  • New member
  • *
  • Posts: 33
idHTTP and Accept-Encoding
« on: March 11, 2018, 07:48:20 am »
Hi

Why if I specify Accept-Encoding: gzip
idHTTP sends Accept-Encoding: deflate, gzip, identity?

Code: Pascal  [Select]
  1. Headers:='Host: '+Domain+#13+
  2.                  'User-Agent: '+UserAgent+#13+
  3.                  'Accept: '+Accept+#13+
  4.                  'Accept-Encoding: gzip'+#13+
  5.                  'Cookie: '+Cookie+#13+
  6.                  'Connection: keep-alive'+#13+
  7.                  'Content-Type: application/x-www-form-urlencoded';
  8.  
  9. HTTP:=TIdHTTP.Create();
  10. HTTP.HTTPOptions:=[hoForceEncodeParams,hoKeepOrigProtocol];
  11. HTTP.ProtocolVersion:=pv1_1;
  12. HTTP.IOHandler:=SSL;
  13. HTTP.Compressor:=CompressorZLib;
  14. HTTP.HandleRedirects:=True;
  15. HTTP.Request.CustomHeaders.AddText(Headers);
  16. Response.ResponseText:=HTTP.Post(URL,PostStream,IndyTextEncoding_UTF8);

In debag proxy showing:

Accept-Encoding: deflate, gzip, identity

Why?

Lazarus 1.8.0, indy 5444 OPM, Linux platform
« Last Edit: March 11, 2018, 07:50:12 am by RDL »
Sorry for my english, google translation!

Remy Lebeau

  • Sr. Member
  • ****
  • Posts: 446
    • Lebeau Software
Re: idHTTP and Accept-Encoding
« Reply #1 on: March 12, 2018, 10:37:42 pm »
Why if I specify Accept-Encoding: gzip
idHTTP sends Accept-Encoding: deflate, gzip, identity?

Because those are the encodings that Indy natively supports when receiving a response from the server.  When you assign a Compressor to TIdHTTP, 'deflate' and 'gzip' are added to the Request.AcceptEncoding if they are not already present, and 'identity' is added to the Request.AcceptEncoding if not blank and 'identity' is not already present.  This behavior is by design.

If you don't want this behavior, you will have to unassign the Compressor from TIdHTTP, and then use the Request.CustomHeaders property instead of the Request.AcceptEncodings property to specify your own 'Accept-Encoding' value.  But then you will have to manually decompress the response data yourself if it is sent compressed, which means you can't use the overloaded version of TIdHTTP.Post() that returns a String, you will have to use the overloaded version that populates an output TStream instead, and then you can decompress it and decode the resulting bytes to a String.

Code: Pascal  [Select]
  1. HTTP.Request.CustomHeaders.AddText(Headers);

That is not the correct way to set the headers you are assigning custom values to.  Use the appropriate TIdHTTP.Request properties instead:

Code: Pascal  [Select]
  1. //HTTP.Request.Host := Domain; // done automatically by TIdHTTP
  2. HTTP.Request.UserAgent := UserAgent;
  3. HTTP.Request.Accept := Accept;
  4. //HTTP.Request.AcceptEncoding := 'gzip'; // let TIdHTTP.Compressor handle this
  5. HTTP.Request.Connection := 'keep-alive';
  6. HTTP.Request.ContentType := 'application/x-www-form-urlencoded';
  7. HTTP.Request.CustomHeaders.Values['Cookie'] := Cookie; // consider letting TIdCookieManager handle this instead

Also, since you are posting an 'application/x-www-form-urlencoded' request, your PostStream should be a TStringList, let TIdHTTP.Post() format its own internal stream, and set the Request.ContentType for you.
« Last Edit: March 12, 2018, 10:47:03 pm by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

RDL

  • New member
  • *
  • Posts: 33
Re: idHTTP and Accept-Encoding
« Reply #2 on: March 13, 2018, 05:08:32 am »
[Remy Lebeau]
Thank you. At me all has turned out.

I've got one more question.
Is it possible to somehow put Content-Length last line. For example:

Code: Pascal  [Select]
  1. POST http://URL/ HTTP/1.1
  2. Content-Type: application/x-www-form-urlencoded1
  3. Host: URL
  4. Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  5. Accept-Encoding: gzip, identity
  6. User-Agent: Mozilla/3.0 (compatible; Indy Library)
  7. Content-Length: 47
Sorry for my english, google translation!

Remy Lebeau

  • Sr. Member
  • ****
  • Posts: 446
    • Lebeau Software
Re: idHTTP and Accept-Encoding
« Reply #3 on: March 13, 2018, 06:51:09 pm »
Is it possible to somehow put Content-Length last line.

TIdHTTP calculates the Content-Length header for you.  You have no control over the order in which TIdHTTP sends its HTTP headers, nor should it usually matter since HTTP itself doesn't generally care about the order of headers.
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus