[RESOLVED] Synapse HttpSend, zero-length content

[jamhitz]

I have been using Indy for quite a while, but now I have started learning Synapse. In the process, I am documenting my findings and will post them online to act as a user guide for new users in the future.

Using HttpSend, I have struggled to understand why some pages return no content at all when requested using THttpSend, but are perfectly alright when requested using a browser; even after sniffing out the headers and trying to make them as compatible as possible.

For example I have tried loading https://dspace.library.uvic.ca:8443/ using the following code:

Code: [Select]

interface

uses
  Classes, SysUtils, FileUtil, Forms, Controls, Graphics, Dialogs, StdCtrls,
  blcksock, httpsend, ssl_openssl;

type

  { Tfrm }

  Tfrm = class(TForm)
      btnSend: TButton;
      edtURL: TEdit;
      memHeader: TMemo;
      memBody: TMemo;
      procedure btnSendClick(Sender: TObject);
  end;

var
  frm: Tfrm;

implementation

{$R *.lfm}

{ Tfrm }

procedure Tfrm.btnSendClick(Sender: TObject);
var ht:THttpSend;
begin
    ht := THttpSend.Create;
    ht.UserAgent := 'Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/20100101 Firefox/5.0';
    ht.Headers.Add('Accept: text/html');
    //ht.KeepAlive := true;
    ht.Protocol := '1.1';
    try
    if ht.HTTPMethod('GET',edtUrl.Text) then
        begin
            if(ht.Document.Size > 0) then
            begin
                ht.Document.Position := 0;
        memBody.Lines.LoadFromStream(ht.Document);
            end
            else
              memBody.Lines.Text := 'Error';
        end;
        memHeader.Lines.Add(IntToStr(ht.ResultCode) + '  ' + ht.ResultString);
    finally
        memHeader.Lines.Text := ht.Headers.Text;
    ht.Free;
    end;
end;
Any ideas?

Thanks.

[Leledumbo]

For example I have tried loading https://dspace.library.uvic.ca:8443/

Get it?
Please take a look at this as well.

[Silvio Clécio]

And:

http://code.google.com/p/lazsolutions/wiki/Tip?wl=en

[jamhitz]

@Leledumbo and @silvioprog:

My checklist:

• I have OpenSSL installed
• I have the latest Synapse from trunk
• I can access *many* secure sites such (eg https://mail.google.com/mail/ and https://www.admissions.uci.edu:8443/cgi-bin/maa_uci.cgi) but not all (eg https://dspace.library.uvic.ca:8443/)

In Synapse, it just fails silently, in Lnet it returns an SSL error, loops perpetually (CallAction) and then segfaults when I break the code.

I don't know how relevant this is, but I noticed (using Firebug) that most of the sites that are failing in Lazarus (but succeeding in Firefox) are running off of Apache-Coyote/1.1 which is part of Tomcat, I think.  Are there any known issues? Here is my LNet code:

Code: [Select]

unit uMain;

{$mode objfpc}{$H+}

interface

uses
  Classes, SysUtils, FileUtil, Forms, Controls, Graphics, Dialogs, StdCtrls,
  lnet, lHttp, lHttpUtil, lNetSSL, UriParser;

type

  { Tfrm }

  Tfrm = class(TForm)
      btnSend: TButton;
      edtURL: TEdit;
      memHeader: TMemo;
      memBody: TMemo;
      procedure btnSendClick(Sender: TObject);
      procedure edtURLChange(Sender: TObject);
      procedure FormCreate(Sender: TObject);
  private
    { private declarations }
       isDone: Boolean;
       errMsg: String;
       aBuff: TMemoryStream;
       procedure HtDisconnect(ASocket: TlSocket);
       procedure HtDoneInput(ASocket: TlHttpClientSocket);
       procedure HtError(const Msg: string; aSocket: TLSocket);
       procedure HtProcessHeaders(ASocket: TLHttpClientSocket);
       function HtInput(ASocket:TLHttpClientSocket; ABuffer:pChar; ASize:Integer): Integer;
       procedure HtSSLAccept(aSocket: TLSocket);
  public
    { public declarations }
  end;

var
  frm: Tfrm;

implementation

{$R *.lfm}

{ Tfrm }

procedure Tfrm.HtDisconnect(ASocket: TLSocket);
begin
isDone := true;
end;

procedure Tfrm.HtDoneInput(ASocket: TLHTTPClientSocket);
begin
ASocket.Disconnect();
end;

procedure Tfrm.HtError(const Msg: string; aSocket: TLSocket);
begin
    isDone := true;
    errMsg := Msg;
    memHeader.Lines.Add('Error: ' + errMsg);
    //aSocket.SocketState;
end;

procedure Tfrm.HtProcessHeaders(ASocket: TLHTTPClientSocket);
begin
    memHeader.Lines.Add(IntToStr(HttpStatusCodes[aSocket.ResponseStatus]) + ' ' + HTTPTexts[aSocket.ResponseStatus] + ' ' + ASocket.ResponseReason);
end;

function Tfrm.HtInput(ASocket: TLHTTPClientSocket; ABuffer: pChar; ASize: Integer): Integer;
var newText:String;
begin
        aBuff.Write(ABuffer^,ASize);
end;

procedure Tfrm.HtSSLAccept(aSocket: TLSocket);
begin
        memHeader.Lines.Add('TLS Handshake successful');
end;

procedure Tfrm.btnSendClick(Sender: TObject);
var   ht:TlHttpClient;
        ssl: TlSslSession;
        url, uri, host: String;
        port: Word; isSSL:Boolean;
begin
ht := TlHttpClient.Create(nil);
        ssl := TlSslSession.Create(ht);
        aBuff := TMemoryStream.Create;
        url := edtUrl.Text;
        isSSl := DecomposeURL(url,host,uri,port);

        {ht.AddExtraHeader('UserAgent: Mozilla/5.0 (X11; Linux i686; rv:5.0) Gecko/20100101 Firefox/5.0');
        ht.AddExtraHeader('Accept: text/html');
        ht.AddExtraHeader('Accept-Language: en-us,en;q=0.5');
        ht.AddExtraHeader('Accept-Encoding: gzip, deflate');
ht.AddExtraHeader('Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7');
ht.AddExtraHeader('Connection: keep-alive');}

    memHeader.Clear;
    memHeader.Lines.Add('Host: ' + host + ', URI: ' + uri + ', Port' + IntToStr(port));

    ssl.SSLActive := isSSL;
    if isSSL then memHeader.Lines.Add('SSL Active');

    try
        ht.Session := ssl;
        ht.Host := host;
        ht.Method := hmGet;
        ht.Port := port;
        ht.URI := uri;
        ht.Timeout := 1;
        ssl.OnSSLAccept := @HtSSLAccept;
        ht.OnDisconnect := @HtDisconnect;
        ht.OnDoneInput := @HtDoneInput;
        ht.OnError := @HtError;
        ht.OnInput := @HtInput;
        ht.OnProcessHeaders := @HtProcessHeaders;
        ht.SendRequest;
        isDone := false;

        while not isDone do
        ht.CallAction;

        aBuff.Position := 0;
        memBody.Lines.LoadFromStream(aBuff);
    finally
        aBuff.Free;
        ht.Free;
    end;
end;
end.

[Silvio Clécio]

Hi @jamhitz,

1. Download the LazSolutions via SVN from link: http://lazsolutions.googlecode.com/svn

2. Open the demo ">lazsolutions>trunk>Demos>LSHTTPSend" and change the code of GetHTMLButton to:

Code: [Select]

procedure TMainForm.GetHTMLButtonClick(Sender: TObject);
 var
{$IFDEF UNIX}
  VStrTemp: TStringList;
{$ENDIF}
  VHTTPSend: THTTPSend;
begin
  try
    VHTTPSend := THTTPSend.Create;
    VHTTPSend.Sock.HTTPTunnelPort := '8443';
{$IFDEF UNIX}
    VStrTemp := TStringList.Create;
{$ENDIF}
    if not LSHTTPGetTextEx(VHTTPSend, HostEdit.Text,
{$IFDEF UNIX}
      VStrTemp
{$ENDIF}
{$IFDEF MSWINDOWS}
      HTMLMemo.Lines
{$ENDIF}) then
      ShowMessage(CUnableHTMLMsg);
{$IFDEF UNIX}
    HTMLMemo.Text := AnsiToUtf8(VStrTemp.Text);
{$ENDIF}
  finally
    VHTTPSend.Free;
{$IFDEF UNIX}
    VStrTemp.Free;
{$ENDIF}
  end;
end;
3. Copy "libeay32.dll" and "ssleay32.dll" to ">lazsolutions>trunk>Demos>LSHTTPSend" (download at: http://code.google.com/p/lazsolutions/source/browse/#svn%2Fmedia%2Flib%2Fwin32).

4. Put the link "dspace.library.uvic.ca" in HostEdit.

5. Enjoy! ^^

[jamhitz]

A bit of progress. I have tried LazSolutions. This time at least I get a response. But the response is a redirect to the very same secure server, not the actual content:

Code: [Select]

<html>
<head>
<title>Redirect to UvicDspace</title>
<meta http-equiv="refresh" content="2; URL=https://dspace.library.uvic.ca:8443/">
<meta name="keywords" content="automatic redirection">
</head>
<body>
If your browser doesn't automatically go there within a few seconds,
you may want to click
<a href="https://dspace.library.uvic.ca:8443/"> here UvicDspace</a>
</body>
</html>
I must be missing something, because I believe FPC/Lazarus should be able to do this.  I am not prepared to give up... just not yet.

[Silvio Clécio]

Hmm... I received this same page. I'll try to solve it. (tomorrow, here in Brazil is already a little late - 23:47h ... ).

[jamhitz]

I have managed to resolved the problem, with the assistance of the nice people of the Synapse forum. Apparently something changed regarding how OpenSSL version 1  supports auto-detection of the SSL version (as opposed to the earlier versions).  As such, the version needs to be explicitly provided:

In the synapse code add:

Code: [Select]

ht.Sock.SSL.SSLType := LT_SSLv3;
In LNet code, add:

Code: [Select]

ssl.Method := msSSLv3;

[BigChimp]

Maybe a good idea to add this to the wiki (if not already done) and mark the thread as SOLVED?

This will help other people in future from facing the same problems...

[jamhitz]

Sure.