TfpHTTPClient - website form login

[kapibara]

I needed to download some files for personal use, but failed to login to the webpage. No matter if I used TFPHttpClient, THttpSend or TIdHTTP.

It really doesn't matter what library i choose, but for now lets lets talk about TFPHttpClient since thats FreePascals own lib.

There was a discussion earlier, but the OP switched to HTTPSend of Synapse instead and never got fphttpclient going: https://forum.lazarus.freepascal.org/index.php/topic,35682.msg236310.html

SSL
I noticed that he had not included units for SSL in his code, but was trying to use HTTPS. Maybe that was the reason. I added opensslsockets, but there is also ssockets, sslsockets, ssl_openssl, ssl_openssl11 and ssl_openssl3. Are any of them needed here?

URL Encoding
So if the url encoding is wrong it won't work. I use fphttp.FormPost for formdata together with a stringlist for login parameters, not single string. Reason be that the formdata must be urlencoded and https://wiki.freepascal.org/fphttpclient says it is done automatically only if a stringlist is used.

I checked the returnvalues of HTTPEncode and EncodeURLElement and the dollar sign "$", "!" and "@" is not encoded but remains as is. Should it be like that? The login string 'ctl00$cph1$lg1$txtEmail' contains the "$".
On the other hand, the custom wiki functions found here encode all these signs: https://wiki.freepascal.org/URL_encoding/decoding

For the formdata below, I'm also not sure if to use "&" signs in front of the strings after the first one or if that is handled automatically or not needed at all.

The result of my efforts is a HTTP 200 code and a not logged in webpage in the Response from fphttpclient.
Trial code looks like this:

Code: Pascal  [Select][+][-]

1.  uses
2.   .. openssl, opensslsockets, fphttpclient, httpprotocol;
3.  
4. procedure TForm1.btnLoginFpHTTPClick(Sender: TObject);
5. const
6.   URL = 'https://eoddata.com';
7. var
8.   HTTP: TFPHTTPClient;
9.   page: string;
10.   formdata: TStrings;
11.   s:string;
12.   Response: TStringStream;
13. begin
14.   formdata:= TStringList.Create;
15.   formdata.values['ctl00$cph1$lg1$txtEmail'] := 'user@somemail.com';  //Username
16.   formdata.values['ctl00$cph1$ls1$txtPassword'] := 'p@ss=w$rd!';  //Password
17.   formdata.values['ctl00$cph1$ls1$chkRemember'] := '1';  //CheckBox, "Remember me"
18.   formdata.values['ctl00$cph1$ls1$btnLogin'] := 'Login';  //Login Button
19.  
20.   InitSSLInterface;
21.   Response:= TStringStream.Create('');
22.  
23.   HTTP:= TFPHTTPClient.Create(nil);
24.   try
25.     try
26.       HTTP.AllowRedirect := True;
27.       HTTP.AddHeader('User-Agent', 'Mozilla/5.0 (compatible; fpweb)');   //HTTP.UserAgent:='Mozilla/5.0 (X11; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0';
28.       HTTP.AddHeader('Content-Type', 'application/x-www-form-urlencoded');
29.       HTTP.KeepConnection:=True;
30.       //HTTP.Headers.Add('Accept-Encoding: identity');
31.       //HTTP.Protocol:='1.1';
32.       //Call GET before FormPost?
33.       //HTTP.Get(URL);
34.       HTTP.FormPost(URL, formdata, Response);
35.       Response.Position:=0;  //Needed for Loading below.
36.       Memo1.Lines.LoadFromStream(Response);
37.       Memo1.Lines.Add(HTTP.ResponseHeaders.Text);
38.     except
39.       on E: EHttpClient do
40.         ShowMessage(E.Message)
41.       else
42.         raise;
43.     end;
44.   finally
45.     Response.Free;
46.     HTTP.Free;
47.   end;
48.  

Also got this:

Code: Pascal  [Select][+][-]

1. Cache-Control: private
2. Content-Type: text/html; charset=utf-8
3. Server: Microsoft-IIS/7.5
4. Set-Cookie: ASP.NET_SessionId=qh5hpbbctlwlgngfh5jjvwfu; path=/; HttpOnly; SameSite=Lax
5. X-AspNet-Version: 4.0.30319
6. X-Powered-By: ASP.NET
7. Date: Sat, 31 Aug 2024 01:24:42 GMT
8. Content-Length: 36362
9.  

HTML code from the webpage:

Code: Text  [Select][+][-]

1. <div class="p">
2.         <div id="ctl00_cph1_lg1_ch1_d1" class="rc_bg_bl"><div id="ctl00_cph1_lg1_ch1_d2" class="rc_tl_bl"><div id="ctl00_cph1_lg1_ch1_d3" class="rc_tr_bl"><div id="d4" class="rc_c">
3.         <table class="rc_t">
4.                 <tr><td><div id="ctl00_cph1_lg1_ch1_divText">MEMBER LOGIN</div></td><td><div id="ctl00_cph1_lg1_ch1_divLink" class="hlink"></div></td><td align="right"><div id="ctl00_cph1_lg1_ch1_divIcon" class="hicon"></div></td></tr>
5.         </table>
6. </div></div></div></div>
7.         <div class="bbf" style="padding:6px;">
8.                 <div id="ctl00_cph1_lg1_pnlLogin">
9.        
10.                         <table cellpadding="4" cellspacing="0" width="100%" border="0" style="margin-bottom:10px;">
11.                           <tr>
12.                             <td>Email/Username:</td>
13.                             <td><input name="ctl00$cph1$lg1$txtEmail" type="text" id="ctl00_cph1_lg1_txtEmail" /></td>
14.                           </tr>
15.                           <tr>
16.                             <td>Password:</td>
17.                             <td><input name="ctl00$cph1$lg1$txtPassword" type="password" id="ctl00_cph1_lg1_txtPassword" /></td>
18.                           </tr>
19.                           <tr>
20.                             <td></td>
21.                             <td><input id="ctl00_cph1_lg1_chkRemember" type="checkbox" name="ctl00$cph1$lg1$chkRemember" /><label for="ctl00_cph1_lg1_chkRemember">Remember Me</label></td>
22.                           </tr>
23.                           <tr><td colspan="2"><span id="ctl00_cph1_lg1_lblMessage"><font color="Red"></font></span></td></tr>
24.                           <tr>
25.                             <td></td>
26.                             <td><input type="submit" name="ctl00$cph1$lg1$btnLogin" value="Login" id="ctl00_cph1_lg1_btnLogin" class="fancy" with="80px;" /></td>
27.                           </tr>                
28.                         </table>                       
29.                         <table cellpadding="4" cellspacing="0" width="100%" border="0">
30.                           <tr>
31.                           <td><a id="ctl00_cph1_lg1_lnkRegister" href="register.aspx">Register</a></td>
32.                           <td align="center"><a id="ctl00_cph1_lg1_btnResendEmail" href="javascript:__doPostBack(&#39;ctl00$cph1$lg1$btnResendEmail&#39;,&#39;&#39;)">Resend Confirmation</a></td>
33.                           <td align="right"><a id="ctl00_cph1_lg1_lnkForgot" href="support/Forgot.aspx">Forgot</a></td>
34.                           </tr>
35.                         </table>
36. </div>
37.         </div>
38. </div>

[TRon]

SSL
I noticed that he had not included units for SSL in his code, but was trying to use HTTPS. Maybe that was the reason. I added opensslsockets, but there is also ssockets, sslsockets, ssl_openssl, ssl_openssl11 and ssl_openssl3. Are any of them needed here?

No, they are not needed. including opensslsockets should be enough.

Other then that: javascript.

How the data is submitted and what requirements are needed usually depends on how the scripting handles things. And what you get back depends again on javascript. fphttpclient does not handle the scripting.

You can do a simple test. In your browser turn off javascript completely and try to login as usual. If that works you got a chance, otherwise you simply don't have any.

FWIW: usually if you are a member of such a site they provide an API. Sites uses that to prevent scraping.

edit: ... and they do offer an xml web-interface that you use for easy access to 'their' data, for the highest tier ofc ...

[kapibara]

I turned off all javascript in Firefox and logged in without problem. So thats good. Because of the javascript on download buttons I am sceptical to if its possible to download the file I wanted, but that is not so important.

This is mostly educational and I will update the wiki documentation when I understand how it works with form logins.

[TRon]

I turned off all javascript in Firefox and logged in without problem. So thats good. Because of the javascript on download buttons I am sceptical to if its possible to download the file I wanted, but that is not so important.

At first sight they try to do everything in their power to prevent you from doing so. Hence the different tiers, for soap/ftp access etc.

But, you can always try to see what the browser does and where the files are actually located and what data goes over the connection before the download actually takes place. Always a long-shot but if you are able to do the same in the browser without javascript then you should be able to replicate that process with fphttpclient (or wget,curl, etc)

This is mostly educational and I will update the wiki documentation when I understand how it works with form logins.

In that case take a closer look at the cookies as well. They need to match as well and are usually required to be present somewhere on the website/session.

Edit: if the main aim is to automatically download the files that you are allowed f.e. on the free tier then you could always opt to fallback using selenium webdriver (There is a header for pascal). I could have sworn you opted that yourself as well in one of your posts.

[kapibara]

I found a webpage made for testing form logins. Will try fphttpclient there when I get some more time:

https://practicetestautomation.com/practice-test-login/

Thanks for the help, Tron. You made me realize that the webpages innocent looking javascript was such a roadblock.