Recent

Author Topic: Download HTML  (Read 1151 times)

pcurtis

  • Sr. Member
  • ****
  • Posts: 351
Download HTML
« on: October 08, 2020, 11:11:29 am »
Dear All,

I can download a web pages HTML WITH THIS

Code: Pascal  [Select][+][-]
  1. function TForm1.SynapseDownload(URL: string): string;
  2. var
  3.   HTTPGetResult: Boolean;
  4.   HTTPSender: THTTPSend;
  5.   S: string;
  6. begin
  7.   Result := '';
  8.   HTTPSender := THTTPSend.Create;
  9.  
  10.   HTTPSender.Timeout:=10000;
  11.  
  12.   HTTPGetResult := HTTPSender.HTTPMethod('GET', URL);
  13.  
  14.   SetLength(S, HTTPSender.Document.Size);
  15.   HTTPSender.Document.Read(S[1], Length(S));
  16.   Result:=S;
  17.  
  18.   HTTPSender.Free;
  19. end;
  20.  

I have two questions

1. How can I tell the web server what browser I am (user agent)? For example "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0"

2. Can I fool the web server into thinking I followed a link on one of it's pages?

Thanks in advance.
« Last Edit: October 08, 2020, 11:28:09 am by pcurtis »
Windows 10 / Linux Mint 20
Laz 2.10.0
FPC 3.2.0

Handoko

  • Hero Member
  • *****
  • Posts: 3902
  • My goal: build my own game engine using Lazarus
Re: Download HTML
« Reply #1 on: October 08, 2020, 11:22:32 am »
1. How can I tell the web server what broswer I am?

Set the user agent property:
http://synapse.ararat.cz/doc/help/httpsend.THTTPSend.html#UserAgent

These are the common user agent strings:
https://www.networkinghowtos.com/howto/common-user-agent-list/

More about user agent:
https://en.wikipedia.org/wiki/User_agent

For example:
HTTPGetResult.UserAgent := 'Mozilla/5.0 (compatible; fpweb)';
« Last Edit: October 08, 2020, 11:29:11 am by Handoko »

pcurtis

  • Sr. Member
  • ****
  • Posts: 351
Re: Download HTML
« Reply #2 on: October 08, 2020, 11:29:41 am »
Cheers. Again  ;)
Windows 10 / Linux Mint 20
Laz 2.10.0
FPC 3.2.0

Handoko

  • Hero Member
  • *****
  • Posts: 3902
  • My goal: build my own game engine using Lazarus
Re: Download HTML
« Reply #3 on: October 08, 2020, 11:42:04 am »
2. Can I fool the web server into thinking I followed a link on one of it's pages?

Not very sure but I believe it has something to do with HTTP referer:
https://en.wikipedia.org/wiki/HTTP_referer

HTTP referer is a field in the Headers property:
http://synapse.ararat.cz/doc/help/httpsend.THTTPSend.html#Headers

pcurtis

  • Sr. Member
  • ****
  • Posts: 351
Re: Download HTML
« Reply #4 on: October 08, 2020, 05:55:35 pm »
Thanks again.
Windows 10 / Linux Mint 20
Laz 2.10.0
FPC 3.2.0

 

TinyPortal © 2005-2018