Can we wallgarden this, please?

[Gustavo 'Gus' Carreno]

Hey Y'All,

So... A friend of mine was trying to use TFPHTTPClient, on Windows, to talk XML to an Italian commercial fridge via HTTP, not HTTPS.
Since he was sure that XML would be returned, every time, he decided to use a TStringStream as the response body.
Alas, it appear that the crappy embedded web server, decides single side, that if the response is higher than 1024 bytes, it will always use Content-Encoding: gzip.

Pretty much, was this means is that, because the TStream descendant he was using has String in it's name, it will not treat a raw byte stream as a raw byte stream. It will definitely have some issues with #0, or any other character below space. Then, that mangled stream is then fed to gzip do be inflated, or not, cuz the gzip header contains a bunch of #0, which may be all mangled by the code that makes that TStream descendant behave like a string interpreter.

Enough background...
Getting back to the question on the subject: Would it be wise, practical or generally accepted to throw some kind of error when TFPHHTPClient gets a Content-Encoding: gzip but the class that has been created for the response body is a TStringStream?

I'm aware that I'm probably missing a lot of shit, mainly because I've been trying to debug this issue for the last 4 hours while being at the end of my awake period, so I'm not completely in control of my second neuron...
But I would still, greatly, appreciate a good answer to my question above.

As per usual, many thanks in advance!!

Cheers,
Gus

[Gustavo 'Gus' Carreno]

Hey Y'all,

I'm BAAAAACK!!!

So, upon further investigation, we found another nice little thing that this server has!!
The first one is that it completely trips balls if you add any character after the last >. Fair enough, we should always sanitise our inputs, so no biggie!! My friend was getting the input from a TMemo and there was a trailing CRLF.

Now, can you spot the second one?:

Code: Text  [Select][+][-]

1. Server: MQX HTTP - Freescale Embedded Web Server\r\n
2. Access-Control-Allow-Origin: *\r\n
3. Connection: close\r\n
4. Content-Type: text/html\n
5. Content-Encoding: gzip\r\n
6. Content-Length: 2004\r\n
7. \r\n
8. <gzip content>

And NO, that's not a typo!! That comes from the WireShark capture.

Soooo, now, I have another question: How forgiving is TFPHTTPClient with the consistency, or lack thereof, of the header's line endings?!

Again, many thanks in advance for any answer!!

Cheers,
Gus
P.S.: Old man shakes fist at Italian embedded programmers!!! 

[domasz]

Maybe fix the newlines?

Code: Pascal  [Select][+][-]

1. A := Pos(#13#10#13#10, Str);
2. if A < 0 then A := Length(Str);
3. B := Pos(#13#13, Str);
4. if B < 0 then B := Length(Str);
5. C := Pos(#10#10, Str);
6. if C < 0 then C := Length(Str);
7.  
8. A := Min(A,B);
9. A := Min(A,C);
10.  
11. Head := Copy(Str, 1, A);
12.  
13. Head := StringReplace(#13#10, '@@@@@', Head, [rfReplaceAll]);
14. Head := StringReplace(#13, '@@@@@', Head, [rfReplaceAll]);
15. Head := StringReplace(#10, '@@@@@', Head, [rfReplaceAll]);
16.  
17. Head := StringReplace('@@@@@', #13#10, Head, [rfReplaceAll]);

[paweld]

It will definitely have some issues with #0, or any other character below space. Then, that mangled stream is then fed to gzip do be inflated, or not, cuz the gzip header contains a bunch of #0, which may be all mangled by the code that makes that TStream descendant behave like a string interpreter.

A String in Pascal can contain all characters in the range 0-255, it is not PChar that #0 is the terminator. Therefore, I believe that the following statement is unnecessary and will cause unnecessary confusion:

Would it be wise, practical or generally accepted to throw some kind of error when TFPHHTPClient gets a Content-Encoding: gzip but the class that has been created for the response body is a TStringStream?

How forgiving is TFPHTTPClient with the consistency, or lack thereof, of the header's line endings?!

It doesn't matter - TStrings handles line breaks very well regardless of the OS and regardless of whether it is CR, LF, or CRLF.

[anse]

If the webserver takes request headers into account (which I would not really expect on a fridge), you may make it easier by disabling gzipped content:

Code: Pascal  [Select][+][-]

1. var c: TFPHTTPClient;
2. c.RequestHeaders.Values['Accept-Encoding'] := '';
3. // or explicitly forbid gzip
4. // https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Accept-Encoding
5. // c.RequestHeaders.Values['Accept-Encoding'] := 'identity;q=1, gzip;q=0';
6.  

[Mobius1]

I don't think TStringStream has issues, for me is a matter of finding the proper encoding.
The brand of this particular fridge is Danfoss, from Denmark. Them Vikings messing things.

The server framework expects a XML in the body but can't have any extra characters after the closing brackets, neither spaces nor line breaks, there was an extra CRLF after the bracket which was giving a generic error. Took me a while to find out.

On top of that, if the answer has more than 1024 bytes, the content will be sent as gzip. Regardless of any accept headers or accept-encoding headers, they don't matter.
But there's a problem: the response header has an error on the content-type value: every header is separated by \r\n or CRLF, except the content-type one which has only \n.

This is the cause of the issue because the next parameter is content-encoding: gzip, which tells fpHttp that the response is compressed. Since fpHttp couldn't parse the content-encoding header it doesn't know it should decompress the response.
We found this because the first 3 bytes of the response were $1F $8B $08 which are part of the gzip signature. At first I thought it was double compressed but after analysing the traffic from Wireshark it was only compressed once so fpHttp wasn't decompressing the response.