[SOLVED] LNET http client

[fabienwang]

Hello,

I have tested the httpclienttest example from lnet svn,
but i can't get the whole source from a webpage.
Example: when i take google.com page, it randomly stops before the end of the webpage.
It should stop at </script>.

I tried with lazarus.freepascal.org and it seems to work great, i get the source until </html>.

I tried to get different https pages and all i have is:

Response: 302 Found, data...
0...
Finished.
Disconnected.

Any help to make the example program working?

[Almindor]

Please specify lNet/Lazarus/FPC version and specific URLs as you put them in the URL bar.

There were issues with this, they are currently fixed in trunk.

[Dibo]

LHTTPClient by default send only few headers (I remember that wireshark sniffer show me only "host" and another when - I don't remember which). Maybe try add some other headers like:
lhttpclient.AddExtraHeader('Connection: keep-alive');
lhttpclient.AddExtraHeader('Keep-alive: 300');

[fabienwang]

lnet from svn revision 2577 (latest one)

Lazarus 0.9.30-0 r FPC 2.4.2 i386-linux-gtk 2

URLS that do work: (any HTTP)
URLS that do not work (any HTTPS)

for HTTPS pages, i have:

TLS handshake successful
Response: 200 OK, data...
0...
Finished.
Disconnected.

here is the function OnInput i changed:

Code: Pascal  [Select][+][-]

1. function TMainForm.HTTPClientInput(ASocket: TLHTTPClientSocket; ABuffer: pchar;
2.   ASize: dword): dword;
3. var
4.   oldLength: dword;
5. begin
6.   oldLength := Length(HTTPBuffer);
7.   setlength(HTTPBuffer,oldLength + ASize);
8.   move(ABuffer^,HTTPBuffer[oldLength + 1], ASize);
9.   AppendToMemo(MemoHTML, HTTPBuffer); //changed line
10.   //MemoHTML.Text := HTTPBuffer;
11.   //MemoHTML.SelStart := Length(HTTPBuffer);
12.   AppendToMemo(MemoStatus, IntToStr(ASize) + '...');
13.   Result := aSize; // tell the http buffer we read it all
14. end;

So it seems the HTTPBuffer is empty on HTTPS pages. Why?

[Almindor]

It was a problem with the contents of some pages not being utf8 compatible. I added a UTF8Encode to the HTTP demo and it works now. Oh and google's page really does miss </html> and other end tags

[fabienwang]

Thanks Almindor.
As mentioned, we have to be careful with encoding! =)
I put [SOLVED] in the topic.