* * *

Author Topic: Determine of Redirect URL with Internet tools (by benibela)  (Read 2036 times)

ArndtB

  • New member
  • *
  • Posts: 12
    • Arndts Blog
Determine of Redirect URL with Internet tools (by benibela)
« on: January 07, 2018, 02:28:11 am »
Hello together,

at first many many thanks to BeniBela for his excellent work on Internet Tools.

I have a small question; for my http://arndtb.github.io/WoW-Addon-Updater/ I've trying to solve a little problem, but after hours of searching I have no clue, how I can solve that with Internet Tools.

To create entries for the Addon-database, the needed entries will be extracted from the website of an specified Addon-URL, for example https://www.curseforge.com/wow/addons/recount.

Now I want to determine the Download-URL for the ZIP-File.

Code: Pascal  [Select]
  1. function GetAddonURL(url: string): string;
  2. var
  3.   ReqAddonURL: IXQValue;
  4.   AddonURL: string;
  5.  
  6. begin
  7.   ReqAddonURL := process(url + '/download', '//a[@class="download__link"]/@href');
  8.   AddonURL := 'https://www.curseforge.com' + ReqAddonURL.toString;
  9.   Result := AddonURL;
  10. end;

This will give the following result:
https://www.curseforge.com/wow/addons/recount/download/2509332/file

But this page will be redirected, showing for example with curl on command line:

Code: [Select]
curl https://www.curseforge.com/wow/addons/recount/download/2509332/file -i
HTTP/1.1 307 Temporary Redirect

Date: Sat, 06 Jan 2018 21:41:19 GMT
Content-Length: 0
Connection: keep-alive
Set-Cookie: __cfduid=d00e032640c7a5584392814737cbce3951515274879; expires=Sun, 06-Jan-19 21:41:19 GMT; path=/; domain=.www.curseforge.com; HttpOnly
Cache-Control: private
Location: https://addons.cursecdn.com/files/2509/332/Recount-v7.3.2b.zip
X-AspNetMvc-Version: 5.2
X-Frame-Options: SAMEORIGIN
X-AspNet-Version: 4.0.30319
X-UA-Compatible: IE=edge,chrome=1
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=600
Server: cloudflare-nginx
CF-RAY: 3d91d739fc6d979e-FRA

My problem is: How can I extract the „Location“-Header-Entry with the File-URL (marked) with Internet Tools?

I’ve tried it with
Code: Pascal  [Select]
  1. process(AddonURL, 'extract(/, "https://([^&]+).zip")')
- but with no success.

Is there another way with Internet Tools, what I cannot see in the moment?

Thank you very much for your help and your ideas!

ArndtB

  • New member
  • *
  • Posts: 12
    • Arndts Blog
Re: Determine of Redirect URL with Internet tools (by benibela)
« Reply #1 on: January 08, 2018, 10:49:02 pm »
I think, I've found a possible solution on BeniBelas Guestbook-Page http://www.benibela.de/koobtseug_de.php#top (Post 404), I will put a quote here:

Quote
Hallo Benito,

zuerst einmal vielen Dank für die internet tools, soweit funktioniert alles sehr gut. Ich habe nur noch ein "kleines" Problem.

Es gibt Webseiten, auf denen weder die aktuelle Versionsnummer noch der Dateiname eines Downloads enthalten sind.

z. B. https://www.wowace.com/projects/wim-3

Der Download Link der Seite verweist auf:
https://www.wowace.com/projects/wim-3/files/latest

Beim Aufruf mit einem Webbrowser wird der Dateiname (momentan WIM-r524-alpha.zip) mitgeliefert. Das folgende kleine Beispiel setzt aber voraus, dass ich den Dateinamen bereits kenne. Wie kann ich den Dateinamen vorab ermitteln?

Code: Pascal  [Select]
  1. uses bbutils, simpleinternet;
  2. ..
  3. strSaveToFileUTF8("irgend ein PfadUndDateiName", retrieve("https://www.wowace.com/projects/wim-3/files/latest"));
  4.  

Danke und Gruß
Jürgen

Quote from: Benito
Nach dem Download ist die endgültige Adresse in defaultInternet.lastUrl. Der Dateiname kann dann beispielsweise so ermittelt werden:

Code: Pascal  [Select]
  1.   temp := retrieve('https://www.wowace.com/projects/wim-3/files/latest');
  2.   name := strCopyFrom(defaultInternet.lastUrl,strLastIndexOf(defaultInternet.lastUrl, '/')+1);
  3.   strSaveToFileUTF8(name, temp);

Wobei es hier der Weiterleitung falsch gefolgt ist. Ich habe eine neue, korrigierte Version hochgeladen.

I will try it and give feedback, when it was successful  :)

But: Other suggestions or ideas are also welcome  :)

ArndtB

  • New member
  • *
  • Posts: 12
    • Arndts Blog
Re: Determine of Redirect URL with Internet tools (by benibela)
« Reply #2 on: January 08, 2018, 11:45:27 pm »
I've tested the following code successfully - if anyone else have a better implementation, you're welcome  :)

Code: Pascal  [Select]
  1. function GetAddonURL(url: string): string;
  2. var
  3.   ReqAddonURL: IXQValue;
  4.   AddonURL: string;
  5.  
  6. begin
  7.   ReqAddonURL := process(url + '/download', '//a[@class="download__link"]/@href');
  8.   AddonURL := 'https://www.curseforge.com' + ReqAddonURL.toString;
  9.  
  10.   // Fix URL-Encoding (see RFC3986), replace space characters (#32) in URL by %20
  11.   AddonURL := StringReplace(AddonURL, #32, #37 + '20', [rfReplaceAll]);
  12.  
  13.   retrieve(AddonURL);
  14.   AddonURL := defaultInternet.lastUrl;
  15.   Result := AddonURL;
  16. end;
  17.  

BeniBela

  • Hero Member
  • *****
  • Posts: 628
    • homepage
Re: Determine of Redirect URL with Internet tools (by benibela)
« Reply #3 on: January 10, 2018, 11:03:49 pm »
I've tested the following code successfully - if anyone else have a better implementation, you're welcome  :)

Code: Pascal  [Select]
  1. function GetAddonURL(url: string): string;
  2. var
  3.   ReqAddonURL: IXQValue;
  4.   AddonURL: string;
  5.  
  6. begin
  7.   ReqAddonURL := process(url + '/download', '//a[@class="download__link"]/@href');
  8.   AddonURL := 'https://www.curseforge.com' + ReqAddonURL.toString;
  9.  

There is a function resolve-html to find the link destination:

Code: Pascal  [Select]
  1.  AddonURL := process(url + '/download', '//a[@class="download__link"]/resolve-html(.)').toString;


I've tested the following code successfully - if anyone else have a better implementation, you're welcome  :)

Code: Pascal  [Select]
  1.   // Fix URL-Encoding (see RFC3986), replace space characters (#32) in URL by %20
  2.   AddonURL := StringReplace(AddonURL, #32, #37 + '20', [rfReplaceAll]);
  3.  

That should happen automatically

ArndtB

  • New member
  • *
  • Posts: 12
    • Arndts Blog
Re: Determine of Redirect URL with Internet tools (by benibela)
« Reply #4 on: January 12, 2018, 05:45:44 pm »
Thank you for the improvements, BeniBela - it works perfect!  :)

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus