logon to a website using fphttpclient

[Nitorami]

I am an absolute newbie to web programming and don't know if what I am asking for is a trivial or a tricky task. Please forgive me if I am stupid.

So, using a very simple example from the wiki, I managed to establish a web connection and to download a http site as string, using TFPHttpClient.SimpleGet and OpenSSL. Yay ! My first internet connection outside a browser, in just ten lines of code !

Now I would like to archive a very long thread from a community I am member of. The thread is split into 30000 individual pages, and clicking through these in a browser would take me ages. So I try to automate this using TFPHttpClient.

However the thread is in the members only section and I receive an error 403. Nice! It is all logical.

So I need to logon. How can I do this ?

EDIT - Hm, I get the error 403 even when trying to access this forum's main page. So there may be other issues that just the logon. SimpleGet() however works on the lazarus forums.

[Thaddy]

allowredirects := true. But 403 means you have no Rights.

[n7800]

Now I would like to archive a very long thread from a community I am member of. The thread is split into 30000 individual pages, and clicking through these in a browser would take me ages. So I try to automate this using TFPHttpClient.

Be careful. Multiple requests to the server in a loop can alert administrators or DDoS protection systems. It will be especially unpleasant if you are blocked by IP, and you can no longer log in even through a browser ))

[n7800]

So I need to logon. How can I do this ?

As far as I know, you need to use "cookie" from your browser where you are already logged in.

[Nitorami]

Be careful. Multiple requests to the server in a loop can alert administrators or DDoS protection systems. It will be especially unpleasant if you are blocked by IP, and you can no longer log in even through a browser ))

Understood. I would run this slowly, and in case of problems would be able to contact the admin via personal message.

As to the "cookie" - I have no idea how to get this from the browser. But even before thinking about the logon, I wonder why even get() of the forum's main page gives me an error 403. It does not require a logon.

[rvk]

Now I would like to archive a very long thread from a community I am member of. The thread is split into 30000 individual pages, and clicking through these in a browser would take me ages. So I try to automate this using TFPHttpClient.

You need to provide the website, or at least the type of forum software which is used there.

Every forum software has it's own login procedure and there is no one way to do it.

BTW. Often forum software also has XML or feed options but I'm not sure if that also goes for complete threads.

[Nitorami]

The website is the old https://forums.winamp.com. It operates with vBulletin written in PHP.

[Thaddy]

Spoof the user agent and you get a correct response:

Code: Pascal  [Select][+][-]

1. {$mode delphi}{$ifdef windows}{$apptype console}{$endif}{$H+}
2. uses
3.   Sysutils,
4.   classes,
5.   fphttpclient,
6.   opensslsockets;
7. var
8.   Client: TFPHttpClient;
9.   Response : string;
10. begin
11.   Client := TFPHttpClient.Create(nil);
12.   writeln(client.httpversion);
13.   try
14.      // This is Edge
15.      Client.AddHeader('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5735.199 Safari/537.36 Edg/114.0.1823.67');
16.      Client.AllowRedirect := true;
17.     try
18.       Response := Client.Get('https://forums.winamp.com');
19.       writeln(Response);
20.      except
21.       on E:Exception do
22.         writeln(e.message);
23.     end;
24.   finally
25.     Client.Free;
26.   end;
27. end.

403 gone... old fashioned spoofing.
Used FPC trunk + OpenSSL 3 + Windows 11

Response:

[Nitorami]

Aha !
Thanks Thaddy. I still get the error 403 from here but I see where this is going. As I said, I am a noob on this and had to read about user agent spoofing and its legitimante and fraudulent uses first. Interesting.

Altogether, this seems to be more tricky than I thought. I may now adjust the UA string to make it work properly but then I'll still need to do the logon handshake with the server, and in the end run the risk of being mistaken as a bot and banned ?

[rvk]

Aha !
Thanks Thaddy. I still get the error 403 from here but I see where this is going.

I also get the 403 from the exact code from Thaddy.

With curl I do see that it uses Cloudflare. And even if you get the 403 back, there is content in the body. But that just says the usual "<title>Just a moment...</title>" from Cloudflare.

I haven't look any further yet... but the fact your dealing with Cloudflare should give some pointers.

[Gustavo 'Gus' Carreno]

Hey Nitorami,

If you wanna pursue the collection of knowledge that you'll get from doing your own tool, I'm all for it. I do like to re-invent the wheel when I'm trying to master a subject.

I also look for some less brain wrecking options, if only, to have them in my back pocket in case plan A through E go flop

I'm not sure what operating system you're using, it's not clear from this thread. But if you have access to wget, it would be a lot easier.

I can even say it would be a one liner Especially because wget is known for it's archival features. If you use the correct set of switches, it will login, or use login cookies from your browser and it will download those 30K message thread, as slow as you tell it, cuz even that has a switch. With what ever user agent you wanna mask it to.

Hope this gives you a last chance plan option, just in case fphttpclient proves to be a bit too challenging.

Cheers,
Gus

[Thaddy]

Well, if there is a correct openssl version... my code works and with proof.
Without the spoof I also get 403, btw.
I suspect that the server denies access with lower tls versions (maybe even tls1.2, but I have to research that). Which is common, correct and proper.
Always make sure your security is in order. Too many people here ignore that and so get punished for it.
Which I really satanic like.  Maybe will learn.

Security protocols are a fast moving target: what worked yesterday will possibly not work today.
And very few on this forum are up-to-date.

[Thaddy]

If you can't use openssl3.x try the lts version 1.1.1s, note the s.
The handshake for that website is tls1.3.

[Thaddy]

I haven't look any further yet... but the fact your dealing with Cloudflare should give some pointers.

Cloudflare is not the issue. It is 2 ways: the user-agent string and tls 1.3 support.
See my screenshot.

Attached captured page. Full page in zip.
Alas, I can do nothing more than that, because it works.

Capture code:

Code: Pascal  [Select][+][-]

1. {$mode delphi}{$ifdef windows}{$apptype console}{$endif}{$H+}
2. uses
3.   Windows,
4.   Sysutils,
5.   classes,
6.   fphttpclient,
7.   opensslsockets;
8. var
9.   Client: TFPHttpClient;
10.   //Response : string;
11.   Page:TStringlist;
12. begin
13.   Client := TFPHttpClient.Create(nil);
14.   Page := TStringlist.Create;
15.   try
16.      Client.AddHeader('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5735.199 Safari/537.36 Edg/114.0.1823.67');
17.      Client.AllowRedirect := true;
18.     try
19.       Page.text := Client.Get('https://forums.winamp.com');
20.       //writeln(Page.text);
21.       Page.SaveToFile('response.html');
22.       ShellExecute(0, 'open', 'response.html', nil, nil, SW_SHOWNORMAL);// x-platform: openurl and add interfaces and lcintf to uses. remove windows.
23.      except
24.       on E:Exception do
25.         writeln(e.message);
26.     end;
27.   finally
28.     Page.Free;
29.     Client.Free;
30.   end;
31. end.

[Nitorami]

@Gustavo: I am on windows 10, and have read about wget. Maybe I should give it a try. In general, I have always tried to do things on the bottom level, in pascal, which I simply know best. Where others used batch files, I wrote a pascal program.... not the most efficient way but it usually worked. Now with web protocols the complexity may well throw me off. We'll see.

It is not easy to get low level help on the Internet on these topics either. Most guidances assume a web browser in the first place - advices such as "clear the cache", "refresh your certificate store" etc.

@Thaddy: Complex as these things are, I would find it well possible that your code works in your place, but not in mine - due to different IP, different god-knows-which-machine settings, locale etc.
But I admit I used the first SSL version I could get hold of, from https://Indy.fulgan.com/SSL/. I found the site somewhere on the wiki. It seems these are 1.0 versions, so I'll try SSL3.