* * *

Author Topic: [SOLVED] Indy10 HTTPServer issues  (Read 12694 times)

JD

  • Hero Member
  • *****
  • Posts: 1608
[SOLVED] Indy10 HTTPServer issues
« on: May 27, 2016, 12:16:31 pm »
Hi there everyone,

Having previously worked exclusively with TCP/IP. I made a first attempt at creating Indy10 HTTPServer which serves json & html files to clients - browser & Indy10 client applications.

The IdHTTPServer parameters are shown below;
Code: Pascal  [Select]
  1. procedure TfrmServeur.FormCreate(Sender: TObject);
  2. begin
  3.   //
  4.   HTTPServer := TIdHTTPServer.Create(nil);
  5.   //
  6.   with HTTPServer do
  7.   begin
  8.     OnCommandGet      := @HTTPServerCommandGet;
  9.     OnCommandOther    := @HTTPServerCommandOther;
  10.     OnConnect         := @HTTPServerConnect;
  11.     OnCreateSession   := @HTTPServerCreateSession;
  12.     OnDisconnect      := @HTTPServerDisconnect;
  13.     OnSessionEnd      := @HTTPServerSessionEnd;
  14.     OnSessionStart    := @HTTPServerSessionStart;
  15.   end;
  16.  
  17.   // set the context class of the server to our custom TClientContext
  18.   // the server will create automatically our class instance
  19.   // when a client is connected and free it when it disconnects
  20.   HTTPServer.ContextClass := TClientContext;
  21. end;
  22.  

The IdHTTPServer is later activated by the code below:
Code: Pascal  [Select]
  1.   HTTPServer.Bindings.Clear;
  2.   //
  3.   for i := 0 to lbIPs.Items.Count - 1 do
  4.   if lbIPs.Checked[i] then
  5.   begin
  6.     Binding := HTTPServer.Bindings.Add;
  7.     Binding.IP := lbIPs.Items.Strings[i];
  8.     Binding.Port := 8080;  //StrToInt(edPort.Text);
  9.   end;
  10.   // Activate the server
  11.   HTTPServer.Active := True;
  12.  

The server-side handles GET commands with the method below:
Code: Pascal  [Select]
  1. procedure TfrmServeur.HTTPServerCommandGet(AContext: TIdContext;
  2.   ARequestInfo: TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);
  3. var
  4.   ServerMethod: TServerMethods;
  5.   strUser, strPassword,
  6.   strLoginProfile: string;
  7.   strJSON: string;
  8. begin
  9.   //
  10.   case Trim(ARequestInfo.URI) of     // ARequestInfo.Params     // http://localhost:8080/departement
  11.     '/pays', '/pays/'
  12.     :
  13.       begin
  14.         // ServerMethod
  15.         ServerMethod := TServerMethods.Create;
  16.         //
  17.         try
  18.           // send JSON list of countries from SQL query
  19.           strJSON := ServerMethod.GetResource(ARequestInfo.URI);
  20.           //
  21.           if strJSON <> EmptyStr then
  22.           begin
  23.             AResponseInfo.ResponseNo := 200;
  24.             AResponseInfo.ContentType := 'application/json';
  25.             AResponseInfo.CharSet := 'utf-8';
  26.             AResponseInfo.ContentText := strJSON;
  27.             AResponseInfo.ContentLength := 99999;  <------ NO, NO, NO!!
  28.           end    // if strJSON <> EmptyStr then
  29.           else
  30.           begin
  31.             //
  32.             AResponseInfo.ResponseNo := 200;        
  33.             AResponseInfo.ContentType := 'text/html';
  34.             AResponseInfo.CharSet := 'utf-8';
  35.             AResponseInfo.ContentText := strNotFound;      
  36.           end;
  37.         finally
  38.           ServerMethod.Free;
  39.         end;
  40.       end;
  41.     else
  42.       begin
  43.         // ServerMethod
  44.         ServerMethod := TServerMethods.Create;
  45.         //
  46.         try
  47.           //
  48.           AResponseInfo.ResponseNo := 404;
  49.           AResponseInfo.ContentType := 'text/html';
  50.           AResponseInfo.CharSet := 'utf-8';
  51.           AResponseInfo.ContentText := strNotFound;       // Not Found
  52.         finally
  53.           ServerMethod.Free;
  54.         end;
  55.       end;
  56.   end;      // case Trim(ARequestInfo.URI) of
  57. end;
  58.  
  59.  

The client-side IdHTTP create method is shown below:
Code: Pascal  [Select]
  1. //
  2. constructor THttpConnectionIndy.Create;
  3. begin
  4.   FIdHttp := TIdHTTP.Create(nil);
  5.   FIdHttp.HandleRedirects := True;
  6.   FIdHTTP.HTTPOptions := [hoForceEncodeParams];
  7.   FIdHTTP.Connect('localhost', 8080);  
  8. end;
  9.  

I have the following problems client-side
a) Browsers
    i) the text is not rendered correctly (see screenshot)
   ii) the entire JSON string is not sent unless AResponseInfo.ContentLength := 99999. I chose a large number to be able to send long strings

b) Indy client applications
    i) the entire JSON string is not sent unless AResponseInfo.ContentLength := 99999. I chose a large number to be able to send long strings

In discussions with Remy Lebeau, he said, and it makes perfect sense, that setting AResponseInfo.ContentLength to a predefined number will FORCE clients to keep reading from the socket until those bytes arrive, or a timeout/error occurs. Indy has a procedure called TIdHTTPResponseInfo.WriteHeader that computes AResponseInfo.ContentLength automatically.

Code: Pascal  [Select]
  1. procedure TIdHTTPResponseInfo.WriteHeader;
  2. var
  3.   ...
  4. begin
  5.   ...
  6.  
  7.   if (ContentLength = -1) and
  8.     ((TransferEncoding = '') or TextIsSame(TransferEncoding, 'identity'))
  9. then {do not localize}
  10.   begin
  11.     // Always check ContentText first
  12.     if ContentText <> '' then begin
  13.       ContentLength := CharsetToEncoding(CharSet).GetByteCount(ContentText);
  14. // <-- HERE
  15.     end
  16.     else if Assigned(ContentStream) then begin
  17.       ContentLength := ContentStream.Size;
  18.     end else begin
  19.       // TODO: should this be calculating the size of the HTML message
  20.       // that WriteContent() generates in this case?
  21.       ContentLength := 0;
  22.     end;
  23.   end;
  24.  
  25.   ...
  26. end;
  27.  

Remy assures me that it MUST be called before responses are sent but I still don't understand why the JSON responses are not being rendered properly in the browser & not being sent completely in browsers and indy clients.

Thanks for your assistance.

JD


« Last Edit: June 09, 2016, 09:56:07 pm by JD »
Windows (10, 7) - Lazarus 1.8/FPC 3.0.4, NewPascal, Delphi
Linux Mint Cinnamon  - Lazarus 1.8/FPC 3.0.4, NewPascal
Indy 10.6 series; mORMot; Zeos 7.2.1; SQLite, Firebird, PostgreSQL & MariaDB

Remy Lebeau

  • Sr. Member
  • ****
  • Posts: 350
    • Lebeau Software
Re: Indy10 HTTPServer issues
« Reply #1 on: May 27, 2016, 07:59:59 pm »
Code: [Select]
AResponseInfo.ContentLength := 99999;  <------ NO, NO, NO!!

Absolutely DO NOT do that.  The ContentLength must be set to the correct number of bytes, not characters, that will be sent:

Code: [Select]
AResponseInfo.ContentLength := IndyTextEncoding_UTF8.GetByteCount(strJSON);

Or set to -1 (the default) to let TIdHTTPServer calculate the value for you:

Code: [Select]
AResponseInfo.ContentLength := -1;

Code: [Select]
AResponseInfo.ResponseNo := 200;         
AResponseInfo.ContentType := 'text/html';
AResponseInfo.CharSet := 'utf-8';
AResponseInfo.ContentText := strNotFound;     

Why are returning a success response code (200) if the requested JSON cannot be retrieved?  I would suggest either 404 or even 5xx instead.

Code: [Select]
FIdHTTP.Connect('localhost', 8080); 

Do not call Connect() directly on a TIdHTTP.  Its various request methods, like Get() and Post(), will call Connect() internally for you when needed.  The correct way to use TIdHTTP in this situation is to use the TIdHTTP.Get() method:

Code: [Select]
strJSON := FIdHTTP.Get('http://localhost:8080/pays');

Which is also all the more reason NOT to have the server returns a 2xx responsse code if the JSON is not available.  It should return an error response code so Get() can raise an exception accordingly.

I have the following problems client-side
a) Browsers
    i) the text is not rendered correctly (see screenshot)

The screenshot actually shows the JSON text is encoded as UTF-8.  The browser is simply interpreting the UTF-8 bytes as ISO-8859-1/Latin-1 when displaying the bytes.  For instance, the Unicode character 'é' is encoded in UTF-8 as bytes $C3 $A9, which are the characters 'é' in ISO-8859-1.

That is a browser issue, not a TIdHTTPServer issue.  TIdHTTPServer should be sending a 'Content-Type: application/json; charset=utf-8' response header.  The browser is likely just ignoring the charset attribute, since technically the official MIME registration for JSON does not include a charset attribute.  Per JSON's MIME registration), the default byte encoding for JSON text is UTF-8, but the browser is likely ignoring that as well.  It is likely seeing the media type as a generic 'application/...' and displaying the raw bytes as-is, rather than seeing the media type is specifically 'application/json' and displaying it as UTF-8.

If you include a 'Content-Disposition' response header, you can force a browser to save the JSON to a file instead of display it as text.  Then you can verify the encoding of the file using any hex/text viewer that supports UTF-8.

Code: [Select]
AResponseInfo.ContentDisposition := 'attachment; filename=pays.json';

   ii) the entire JSON string is not sent unless AResponseInfo.ContentLength := 99999. I chose a large number to be able to send long strings

As we have discussed on the Embarcadero forum, I guarantee you that is NOT how TIdHTTPServer operates.  Something else is going on to mess with the transmission.  TIdHTTPServer does not utilize the ContentLength property when sending the ContentText.  Setting the ContentLength property in that case merely provides a value for the 'Content-Length' response header, nothing more.  See the source code in the TIdHTTPResponseInfo.WriteHeader()  and TIdHTTPResponseInfo.WriteContent() and methods in IdCustomHTTPServer.pas.

In discussions with Remy Lebeau, he said, and it makes perfect sense, that setting AResponseInfo.ContentLength to a predefined number will FORCE clients to keep reading from the socket until those bytes arrive, or a timeout/error occurs.

That is true, per requirement of the HTTP spec (see RFC 2616 Section 4.4 Message Length).

Remy assures me that it MUST be called before responses are sent

Yes, and TIdHTTPResponseInfo.WriteContent() calls WriteHeader() if it has not already been called:

Code: [Select]
procedure TIdHTTPResponseInfo.WriteContent;
begin
  if not HeaderHasBeenWritten then begin
    WriteHeader;
  end;
  ...
end;

TIdCustomHTTPServer.DoExecute() automatically calls WriteHeader() and WriteContent() as need, after relevant event handlers have been called to prepare the response properties.

but I still don't understand why the JSON responses are not being rendered properly in the browser

Actually, it is, just not the way you are expecting.  Remember, the 'application/...' media type is for, well, application data, not text data.  The browser is displaying a textual representation of something that is technically binary data, not text data.  And there is no official 'text/...' media type for JSON (see What is the correct JSON content type?).

& not being sent completely in browsers and indy clients.

That, I do not know yet.  We have not uncovered the real culprit yet.

Use a packet sniffer, like Wireshark or Fiddler, to verify the response is being sent correctly, and in the correct timing.  Maybe something sitting in between the client and TIdHTTPServer is hindering the transmission.

Use a debugger to step into the server code to see what it is really doing while processing your request and make sure it is preparing the response correctly.
« Last Edit: May 27, 2016, 08:21:23 pm by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) open source project - Admin, Developer

JD

  • Hero Member
  • *****
  • Posts: 1608
Re: Indy10 HTTPServer issues
« Reply #2 on: May 28, 2016, 08:30:44 pm »
Thanks for your reply Remy. I attach a test project to my reply along with a screen capture of the results to show that ContentLength is not being calculated automatically and the characters are not being displayed normally. The debugger did not pass through the WriteHeader procedure at all even after I had put breakpoints in the code!

This is exactly how my browser displays the data. I would be glad if you or anybody else familiar with Indy on this forum could take a look at it.

The test project is a composite of Indy HTTP Server & HTTPClient on localhost:8080.

Thanks,

JD
« Last Edit: May 28, 2016, 08:33:13 pm by JD »
Windows (10, 7) - Lazarus 1.8/FPC 3.0.4, NewPascal, Delphi
Linux Mint Cinnamon  - Lazarus 1.8/FPC 3.0.4, NewPascal
Indy 10.6 series; mORMot; Zeos 7.2.1; SQLite, Firebird, PostgreSQL & MariaDB

engkin

  • Hero Member
  • *****
  • Posts: 1758
Re: Indy10 HTTPServer issues
« Reply #3 on: May 29, 2016, 02:30:52 am »
Add {$codepage UTF8} to unit1.

JD

  • Hero Member
  • *****
  • Posts: 1608
Re: Indy10 HTTPServer issues
« Reply #4 on: May 29, 2016, 05:22:26 am »
Add {$codepage UTF8} to unit1.

Thanks for your reply engkin. I did as you suggested but it didn't work! I got the same results as before.

Code: [Select]
unit Unit1;

{$mode objfpc}{$H+}
{$codepage UTF8}

However I discovered that if I commented out AResponseInfo.CharSet := 'utf-8' and computed content length using AResponseInfo.ContentLength := IndyTextEncoding_UTF8.GetByteCount(strJSON) as in the code below:

Code: Pascal  [Select]
  1.  if strJSON <> EmptyStr then
  2.  begin
  3.    AResponseInfo.ResponseNo := 200;
  4.    AResponseInfo.ContentType := 'application/json';
  5.    //AResponseInfo.CharSet := 'utf-8';
  6.    AResponseInfo.ContentText := strJSON;
  7.    AResponseInfo.ContentLength := IndyTextEncoding_UTF8.GetByteCount(strJSON);
  8.  end    // if strJSON <> EmptyStr then
  9.  

then the client gets the entire JSON string! It is still not displayed correctly though.  :(

JD
« Last Edit: May 29, 2016, 05:29:04 am by JD »
Windows (10, 7) - Lazarus 1.8/FPC 3.0.4, NewPascal, Delphi
Linux Mint Cinnamon  - Lazarus 1.8/FPC 3.0.4, NewPascal
Indy 10.6 series; mORMot; Zeos 7.2.1; SQLite, Firebird, PostgreSQL & MariaDB

GetMem

  • Hero Member
  • *****
  • Posts: 2463
Re: Indy10 HTTPServer issues
« Reply #5 on: May 29, 2016, 02:43:18 pm »
Hi JD,

I just converted your project to Delphi XE4, works fine without any modification(see attachment 1). It must be another encoding issue(windows + indy + lazarus), very similar to this one:
http://forum.lazarus.freepascal.org/index.php/topic,32694.0.html
Your project also works with lazarus, if you convert the string to stream and send it as a stream(attachment 2). 

regards,
GetMem

JD

  • Hero Member
  • *****
  • Posts: 1608
Re: Indy10 HTTPServer issues
« Reply #6 on: May 29, 2016, 03:52:17 pm »
Hi JD,

I just converted your project to Delphi XE4, works fine without any modification(see attachment 1). It must be another encoding issue(windows + indy + lazarus), very similar to this one:
http://forum.lazarus.freepascal.org/index.php/topic,32694.0.html
Your project also works with lazarus, if you convert the string to stream and send it as a stream(attachment 2). 

regards,
GetMem

Thanks for your reply and your findings. How do I send the response as a stream on the Server side? In addition, how do I make a browser read the stream?

JD
Windows (10, 7) - Lazarus 1.8/FPC 3.0.4, NewPascal, Delphi
Linux Mint Cinnamon  - Lazarus 1.8/FPC 3.0.4, NewPascal
Indy 10.6 series; mORMot; Zeos 7.2.1; SQLite, Firebird, PostgreSQL & MariaDB

GetMem

  • Hero Member
  • *****
  • Posts: 2463
Re: Indy10 HTTPServer issues
« Reply #7 on: May 29, 2016, 04:31:59 pm »
Please try this:
1. Server:
Code: Pascal  [Select]
  1. procedure TForm1.HTTPServerCommandGet(AContext: TIdContext;
  2.   ARequestInfo: TIdHTTPRequestInfo; AResponseInfo: TIdHTTPResponseInfo);
  3. var
  4.   strJSON: string;
  5. begin
  6.   strJSON := String(PAYS);
  7.   if strJSON <> EmptyStr then
  8.   begin    
  9.     AResponseInfo.ContentType := 'application/json';
  10.     AResponseInfo.CharSet := 'utf-8';
  11.  
  12.     AResponseInfo.ContentStream := TMemoryStream.Create;
  13.     try
  14.       AResponseInfo.ContentStream.Write(Pointer(strJSON)^, Length(strJSON));
  15.       AResponseInfo.ContentStream.Position := 0;
  16.       AResponseInfo.WriteContent;
  17.     finally
  18.       AResponseInfo.ContentStream := nil;
  19.     end;
  20.   end  
  21.   else
  22.   begin
  23.     AResponseInfo.ResponseNo := 404;
  24.     AResponseInfo.ContentType := 'text/html';
  25.     AResponseInfo.CharSet := 'utf-8';
  26.     AResponseInfo.ContentText := 'Resource not found';
  27.   end;
  28. end;

2. Client:
Code: Pascal  [Select]
  1. procedure TForm1.btnGetDataClick(Sender: TObject);
  2. var
  3.   Str: String;
  4.   MS: TMemoryStream;
  5. begin
  6.   if Trim(memServer.Text) = EmptyStr then
  7.     exit
  8.   else
  9.   begin
  10.     HTTPClient := TIdHTTP.Create(nil);
  11.     try
  12.       ms := TMemoryStream.Create;
  13.       try
  14.         HTTPClient.HandleRedirects := True;
  15.         HTTPClient.HTTPOptions := [hoForceEncodeParams];
  16.         HTTPClient.Get(Trim(edtDomain.Text), MS);
  17.         MS.Position := 0;
  18.         SetLength(Str, Ms.Size);
  19.         MS.Read(Pointer(Str)^, Length(Str));
  20.         memClient.Lines.Text := Str;
  21.       finally
  22.         Ms.Free;
  23.       end;
  24.     finally
  25.       HTTPClient.Free;
  26.     end;
  27.   end;
  28. end;
« Last Edit: May 29, 2016, 09:36:16 pm by GetMem »

JD

  • Hero Member
  • *****
  • Posts: 1608
Re: Indy10 HTTPServer issues
« Reply #8 on: May 29, 2016, 06:45:19 pm »
 :D It worked! Including the rendering in the Firefox browser! Thanks a million GetMem.

JD
Windows (10, 7) - Lazarus 1.8/FPC 3.0.4, NewPascal, Delphi
Linux Mint Cinnamon  - Lazarus 1.8/FPC 3.0.4, NewPascal
Indy 10.6 series; mORMot; Zeos 7.2.1; SQLite, Firebird, PostgreSQL & MariaDB

JD

  • Hero Member
  • *****
  • Posts: 1608
Re: Indy10 HTTPServer issues
« Reply #9 on: May 29, 2016, 06:48:48 pm »
Does anyone else think we can put this up on the Lazarus wiki as an example for other programmers?

JD
Windows (10, 7) - Lazarus 1.8/FPC 3.0.4, NewPascal, Delphi
Linux Mint Cinnamon  - Lazarus 1.8/FPC 3.0.4, NewPascal
Indy 10.6 series; mORMot; Zeos 7.2.1; SQLite, Firebird, PostgreSQL & MariaDB

GetMem

  • Hero Member
  • *****
  • Posts: 2463
Re: Indy10 HTTPServer issues
« Reply #10 on: May 29, 2016, 08:09:30 pm »
@JD
I'm glad it's working but still, I would like to find out why the same exact project works with XE4 and not with Lazarus.

JD

  • Hero Member
  • *****
  • Posts: 1608
Re: Indy10 HTTPServer issues
« Reply #11 on: May 29, 2016, 08:37:22 pm »
@JD
I'm glad it's working but still, I would like to find out why the same exact project works with XE4 and not with Lazarus.

It has to be the encoding. It was driving me nuts. FPC 3+ will take some getting used to. I also have XE4 but I didn't test it inside Delphi because it does not have a recent version of Indy10 installed & I did not want to go to the trouble of updating it. Thank goodness you had it also.

We'll need Remy's input to get to the bottom of this.

JD
« Last Edit: May 29, 2016, 09:34:55 pm by JD »
Windows (10, 7) - Lazarus 1.8/FPC 3.0.4, NewPascal, Delphi
Linux Mint Cinnamon  - Lazarus 1.8/FPC 3.0.4, NewPascal
Indy 10.6 series; mORMot; Zeos 7.2.1; SQLite, Firebird, PostgreSQL & MariaDB

Graeme

  • Hero Member
  • *****
  • Posts: 1419
    • Graeme on the web
Re: Indy10 HTTPServer issues
« Reply #12 on: May 30, 2016, 05:42:23 pm »
Quote
It has to be the encoding. It was driving me nuts. FPC 3+ will take some getting used to.
Indeed, and as I mentioned in the FPC Mailing List - FPC 3.0 is what I consider a "preview release" and really should not be used in actual applications. Obviously the core developers says otherwise, but I disagree with them. I have found quite a few things that FPC 3.0 breaks, and easily causes data loss due to encoding conversions. And because the FPC 3.0 RTL is not fully Unicode ready (UTF-16 or UTF-8), there is not much we can do - other than submit RTL fixes for the missing parts. Then once FPC 3.2 is released [in one or two years time], then maybe FPC will be usable again. For now, I recommend staying with FPC 2.6.4 for actual applications. I strongly believe FPC 3.0 is now worse off than what Delphi's 2009 release was.
« Last Edit: May 30, 2016, 05:44:40 pm by Graeme »
--
fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal
http://fpgui.sourceforge.net/

Thaddy

  • Hero Member
  • *****
  • Posts: 4774
Re: Indy10 HTTPServer issues
« Reply #13 on: May 30, 2016, 06:49:19 pm »
@Graeme and Remy:

The issue at hand is much much simpler and has always been so:
The transport layer itself should not make any guesses about any encoding. It transports bytes for god's sake...
That's the issue with Indy and also the issue why it is slow and unreliable. It doesn't help, it tries to second-guess. Drop it Remy... Painful it may be but it is better for society. Or re-design that piece of sh*t in any meaningful, performant, standards compliant way without any suggestion that high(er) level components are in any way compliant with any standards except those of Indy itself.. Otherwise you can predict the bug reports and questions and work-arounds (favorite pass-time?) beforehand.

I haven't seen any Indy version that properly makes the distinction between the underlying protocols and its component representation at the proper level of abstraction. It's been a mess from the start.

Note for non-informed readers: I actually use Indy some times ;) ... But not for anything serious.
« Last Edit: May 30, 2016, 06:59:50 pm by Thaddy »
"Logically, no number of positive outcomes at the level of experimental testing can confirm a scientific theory, but a single counterexample is logically decisive."

GetMem

  • Hero Member
  • *****
  • Posts: 2463
Re: Indy10 HTTPServer issues
« Reply #14 on: May 30, 2016, 07:33:56 pm »
Quote
@Thaddy
Drop it Remy... Painful it may be but it is better for society. Or re-design that piece of sh*t in any meaningful, performant, standards compliant way without any suggestion that high(er) level components are in any way compliant with any standards except those of Indy itself.. Otherwise you can predict the bug reports and questions and work-arounds (favorite pass-time?) beforehand.
If you don't like Indy it's perfectly fine, but please stop spreading nonsense. I have a client/server application running for more then five years, I have never had any issue with it, not once. Besides @Remy helped a lot of people over the past 10 years(including me), so this kind of language is unacceptable! Thaddy you're a knowledgeable programmer, but sometimes you act like a grumpy old man.

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus