Recent

« previous next »
Pages: [1]

Author Topic: Is there any library that would make process of making a website scrapper easier  (Read 837 times)

Rave

  • Full Member
  • ***
  • Posts: 170
Is there any library that would make process of making a website scrapper easier
« on: August 09, 2024, 11:19:30 pm »
Basically I want to make a web scraper for one of the sites I love that has awful UI so that I can interact with it better. Is there any Lazarus/Free Pascal library that would make this process easier? I'd rather not parse HTML by hand.
Logged

Thaddy

  • Hero Member
  • *****
  • Posts: 15496
  • Censorship about opinions does not belong here.
Re: Is there any library that would make process of making a website scrapper easier
« Reply #1 on: August 10, 2024, 01:50:13 pm »
Our member benibela has a good library for that:
https://www.benibela.de/sources_en.html#internettools
Simple example that extracts all hrefs from a page:
Code: Pascal  [Select][+][-]
  1. uses simpleinternet, xquery;
  2.  
  3. var a: IXQValue;
  4. begin
  5.   for a in process('https://freepascal.org', '//a/@href') do
  6.     writeln(a.toString);
  7. end.
You need to undefine USE_PASDBLSTRUTILS_FOR_JSON in internettoolsconfig.inc.
« Last Edit: August 10, 2024, 01:52:43 pm by Thaddy »
Logged
My great hero has found the key to the highway. Rest in peace John Mayall.
Playing: "Broken Wings" in your honour. As well as taking out some mouth organs.
Pages: [1]
« previous next »
 

TinyPortal © 2005-2018