Recent

Author Topic: Trying to download file from google drive (unauthenticated)  (Read 1137 times)

void09

  • New member
  • *
  • Posts: 6
Trying to download file from google drive (unauthenticated)
« on: January 07, 2021, 02:48:30 pm »
Code: Pascal  [Select][+][-]
  1. So following this guide https://www.matthuisman.nz/2019/01/download-google-drive-files-wget-curl.html
  2.  I put this code together :
  3.  
  4. unit gdrivedown;
  5.  
  6. {$mode objfpc}{$H+}
  7.  
  8. interface
  9. uses fphttpclient,opensslsockets,sysutils,strutils;
  10.  
  11. procedure DownloadFromGDrive(const URL,filepath:String);
  12.  
  13. implementation
  14.  
  15. procedure DownloadFromGDrive(const URL,FilePath:String);
  16.  var Html,FileID,Token:string;
  17.       HttpClient:TFPHTTPClient;
  18.  
  19.    function GetDownloadId(const URL:String):String;
  20.     var p,i:word;
  21.       R:string;
  22.   begin
  23.     Result:=''; p:=0;
  24.     p:=pos('?id=',URL);
  25.     if p<>0 then p:=p+4 else begin p:=pos('/d/',URL); if p<>0 then p:=p+3; end;
  26.      i:=p;
  27.       while URL[i] in ['a'..'z','A'..'Z','0'..'9'] do begin
  28.         Result:=Result+URL[i];
  29.         if i< length(URL) then inc(i);
  30.       end;
  31.    end;
  32.  
  33.    function GetConfirmToken(const Html:String):String;
  34.     var p1,p2:word;
  35.    begin
  36.      p1:= pos('confirm=',Html);
  37.      if p1<>0 then begin
  38.         p2:=PosEx( '&amp',Html,p1);
  39.         if p2<>0 then Result:=copy(HTML,p1+8,p2-p1-8);
  40.         writeln('confirm Token is '+Result);
  41.      end;
  42.    end;
  43.  
  44. begin
  45.   HttpClient:= TFPHTTPClient.Create(nil);
  46.   with HttpClient do try
  47.      AddHeader('User-Agent','Wget/1.13.4 (linux-gnu)');
  48.      FileID := GetDownloadID(URL);
  49.      writeln('attempting to get '+'https://docs.google.com/uc?export=download&id='+FileID);
  50.      Html := get('https://docs.google.com/uc?export=download&id='+FileID);
  51.      Writeln('download id is',FileID);
  52.      Token:=GetConfirmToken(Html);
  53.        Cookies.SaveToFile('lol.txt') ;
  54.         writeln('trying to get '+'https://docs.google.com/uc?export=download&id='+FileID+'&confirm='+Token);
  55.         try
  56.         AllowRedirect:=True;
  57.         html:= get('https://docs.google.com/uc?export=download&id='+FileID+'&confirm='+Token);
  58.         except
  59.           on E: EHttpClient do
  60.           writeln(E.Message) else raise; end;
  61.         writeln('got codes '+inttostr(ResponseStatusCode));
  62.         Cookies.SaveToFile('lol2.txt') ;
  63.        finally
  64.        free;
  65.        end;
  66. end;
  67.  
  68.  
  69. end.
  70.  
  71.  

and on the main program I just call this with
Code: Pascal  [Select][+][-]
  1. program project1;
  2.   uses gdrivedown;
  3. var
  4.     url:string='https://drive.google.com/file/d/11C8MTF9Il7l3MZOwcqCWegqQMwhe7fsU/view?usp=sharing';
  5. begin
  6.   DownloadFromGDrive(url,'lol.mkv');
  7. end.      
                 

Problem is, on second http get call, I get "Maximum redirects reached (17)". I checked with the wget example in the url above and it does indeed redirects 4-5 times, but eventually downloads. No idea how I can get the redirect messages to show so I can debug this, first time using this http client.

Please tell me if this code is already available in a library somewhere or posted somewhere, and how to debug this. Thanks

This is the output from attempting to do this with wgeet: https://hastebin.com/siyiwikiho.rb
« Last Edit: January 07, 2021, 03:25:17 pm by void09 »

rvk

  • Hero Member
  • *****
  • Posts: 4465
Re: Trying to download file from google drive (unauthenticated)
« Reply #1 on: January 07, 2021, 03:43:24 pm »
You could create a HttpClient.OnRedirect to see how many times there is actually a redirect and what the actual headers per redirect were.

You can also set HttpClient.MaxRedirects to something really big to see if it eventually gets the file.
(The default is DefMaxRedirects = 16; )

void09

  • New member
  • *
  • Posts: 6
Re: Trying to download file from google drive (unauthenticated)
« Reply #2 on: January 07, 2021, 03:45:52 pm »
I set redirects to 250 and it still reaches them.. so it goes on forever.
I will try to see about this HttpClient.OnRedirect thing, never used.

rvk

  • Hero Member
  • *****
  • Posts: 4465
Re: Trying to download file from google drive (unauthenticated)
« Reply #3 on: January 07, 2021, 03:51:10 pm »
In the OnRedirect event you can examine ADest for the new destination/redirection.
You can check if that's the same as your wget example.

You can also examine HttpClient.ResponseHeaders to see the other header lines to see if something wrong there (per redirect) like cookies.

void09

  • New member
  • *
  • Posts: 6
Re: Trying to download file from google drive (unauthenticated)
« Reply #4 on: January 07, 2021, 04:23:05 pm »
Unfortunately I never worked with RedirectEvents and have no idea what i'm doing. Trying random code that I think might do it doesn't work in this case, if I don't understand what's going on. So can you please provide a sample code for this and a short explanation of what happens, or give me al link I can study, didn't find anything simple enough for beginners.

rvk

  • Hero Member
  • *****
  • Posts: 4465
Re: Trying to download file from google drive (unauthenticated)
« Reply #5 on: January 07, 2021, 04:41:03 pm »
You can do something like this:

Code: Pascal  [Select][+][-]
  1. type
  2.  
  3.   { TForm1 }
  4.  
  5.   TForm1 = class(TForm)
  6.     Button1: TButton;
  7.     Memo1: TMemo;
  8.     procedure Button1Click(Sender: TObject);
  9.   private
  10.  
  11.   public
  12.     procedure DoRedirect(Sender : TObject; Const ASrc : String; Var ADest: String);
  13.  
  14.   end;
  15.  
  16. var
  17.   Form1: TForm1;
  18.  
  19. implementation
  20.  
  21. {$R *.lfm}
  22.  
  23. { TForm1 }
  24.  
  25. uses FpHttpClient;
  26.  
  27. procedure TForm1.DoRedirect(Sender : TObject; Const ASrc : String; Var ADest: String);
  28. begin
  29.   Memo1.Lines.Add('===================');
  30.   Memo1.Lines.Add(ADest);
  31.   Memo1.Lines.Add('-------------------');
  32.   Memo1.Lines.Add(TfpHttpClient(Sender).ResponseHeaders.Text);
  33.   Memo1.Lines.Add('===================');
  34. end;
  35.  
  36. procedure TForm1.Button1Click(Sender: TObject);
  37. var HttpClient: TfpHttpClient;
  38. begin
  39.   HttpClient := TfpHttpClient.Create(nil);
  40.   try
  41.     HttpClient.OnRedirect:= @DoRedirect; // add this
  42.     // do your thing
  43.   finally
  44.     HttpClient.Free;
  45.   end;
  46. end;

Note that DoRedirect is assigned to HttpClient.OnRedirect.
Now, that procedure is executed on every redirect and it prints every header.
In the result you can look at the Location-header and cookies to see if it changes correctly.

void09

  • New member
  • *
  • Posts: 6
Re: Trying to download file from google drive (unauthenticated)
« Reply #6 on: January 07, 2021, 05:55:52 pm »
What if I have a simple program and not a form class variable. Would the DoRedirect procedure work if not a class method ?
If it wouldn't, what kind of object would it be suitable to put into ? make my customhttp client class inherited from TFPHttpCLient  and just add that method ?  Then I wouldn't need to use sender as it would have access to the httpclient object ?

About this:
  Memo1.Lines.Add(TfpHttpClient(Sender).ResponseHeaders.Text);

Sender : TObject in this case is the HttpClient variable, and you just mention the type of the object you are reffering to, because otherwise it's just a pointer and doesn't know how to deal with this  ?

Would appreciate if you can point to me to some easy to follow and understand tutorials/books/videos, as OOP is hard for me to understand and conceptualize/implement.

Zaher

  • Hero Member
  • *****
  • Posts: 611
    • parmaja.org
Re: Trying to download file from google drive (unauthenticated)
« Reply #7 on: January 07, 2021, 07:35:34 pm »
Try to inherit TFPHTTPClient and add method for OoRedirect

Code: [Select]
unit gdrivedown;
 
{$mode objfpc}{$H+}
 
interface

uses
  fphttpclient, opensslsockets, sysutils, strutils;

type

  { TMyHTTPClient }

  TMyHTTPClient = class(TFPHTTPClient)
  public
    Procedure DoRedirect(Sender : TObject; Const ASrc : String; Var ADest: String);
  end;
 
procedure DownloadFromGDrive(const URL,filepath:String);
 
implementation
 
procedure DownloadFromGDrive(const URL,FilePath:String);

 var Html,FileID,Token:string;
      HttpClient:TMyHTTPClient;
 
   function GetDownloadId(const URL:String):String;
    var p,i:word;
      R:string;
  begin
    Result:=''; p:=0;
    p:=pos('?id=',URL);
    if p<>0 then p:=p+4 else begin p:=pos('/d/',URL); if p<>0 then p:=p+3; end;
     i:=p;
      while URL[i] in ['a'..'z','A'..'Z','0'..'9'] do begin
        Result:=Result+URL[i];
        if i< length(URL) then inc(i);
      end;
   end;
 
   function GetConfirmToken(const Html:String):String;
    var p1,p2:word;
   begin
     p1:= pos('confirm=',Html);
     if p1<>0 then begin
        p2:=PosEx( '&amp',Html,p1);
        if p2<>0 then Result:=copy(HTML,p1+8,p2-p1-8);
        writeln('confirm Token is '+Result);
     end;
   end;
 
begin
  HttpClient:= TMyHTTPClient.Create(nil);
  with HttpClient do try
    OnRedirect := @DoRedirect;
     AddHeader('User-Agent','Wget/1.13.4 (linux-gnu)');
     FileID := GetDownloadID(URL);
     writeln('attempting to get '+'https://docs.google.com/uc?export=download&id='+FileID);
     Html := get('https://docs.google.com/uc?export=download&id='+FileID);
     Writeln('download id is',FileID);
     Token:=GetConfirmToken(Html);
       Cookies.SaveToFile('lol.txt') ;
        writeln('trying to get '+'https://docs.google.com/uc?export=download&id='+FileID+'&confirm='+Token);
        try
        AllowRedirect:=True;
        html:= get('https://docs.google.com/uc?export=download&id='+FileID+'&confirm='+Token);
        except
          on E: EHttpClient do
          writeln(E.Message) else raise; end;
        writeln('got codes '+inttostr(ResponseStatusCode));
        Cookies.SaveToFile('lol2.txt') ;
       finally
       free;
       end;
end;

{ TMyHTTPClient }

procedure TMyHTTPClient.DoRedirect(Sender: TObject; const ASrc: String; var ADest: String);
begin
  WriteLn('src: ' + ASrc);
  WriteLn(' to ' +  ADest);
end;
 
end.

void09

  • New member
  • *
  • Posts: 6
Re: Trying to download file from google drive (unauthenticated)
« Reply #8 on: January 14, 2021, 07:52:57 am »
Just an update to this, after many tries there is still no success in downloading a file from google drive.
If somebody experienced with httpclient can have a go at this, it would benefit the whole pascal community I think.
I believe functionality like this should exist in units already.

trev

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1275
  • Former Delphi 1-7, 10.2 User
Re: Trying to download file from google drive (unauthenticated)
« Reply #9 on: January 14, 2021, 08:06:57 am »
FPC provides a unit for Google's Drive API (it's already contained in 3.2.0). For using the API itself you should probably check Google's documentation as the types are generated from Google's service description. For using Google APIs in general with FPC you can take a look at Michael van Canneyt's article about it.
- Source PascalDragon :)
Lazarus 2.1 r64368 FPC 3.3.1 r48100 macOS 10.14.6 Xcode 11.3.1
Lazarus 2.1 r64455 3.3.1 r48688  macOS 11.2 aarch64 Xcode 12.4
Lazarus 2.1 r61574 3.3.1 r42318 FreeBSD 12.1 amd64 VMware VM
Lazarus 2.1 r61574 3.0.4 Ubuntu 20.04 Parallels VM
Lazarus 2.0.10 3.2.0 Win10 Parallels VM

void09

  • New member
  • *
  • Posts: 6
Re: Trying to download file from google drive (unauthenticated)
« Reply #10 on: January 14, 2021, 12:36:53 pm »
FPC provides a unit for Google's Drive API (it's already contained in 3.2.0). For using the API itself you should probably check Google's documentation as the types are generated from Google's service description. For using Google APIs in general with FPC you can take a look at Michael van Canneyt's article about it.
- Source PascalDragon :)

I think these APIs are meant for authenticated users and provide advanced functions for drive (like rclone uses for example). I just want to download a file from a link, easily doable with 2 wget commands, which I want to reproduce and can't. Thus asking assistance.

Edit: Zaher notified me he found the solution to this problem and will post it later. No need for further investigation.
« Last Edit: January 14, 2021, 11:16:31 pm by void09 »

Zaher

  • Hero Member
  • *****
  • Posts: 611
    • parmaja.org
Re: Trying to download file from google drive (unauthenticated)
« Reply #11 on: January 15, 2021, 08:06:43 pm »
We need to save cookies between urls, save the cookie depend on domain name that cookie related to it, that need a good manage to parse the url and cookie that not exists in fphttpclient, i did that trick not good but work fine in my code bellow, the idea to save NID value that come with cookies responds to the client (here our code) on google.com domain, then you need pass it again in the next redirect, also AUTO cookie  needs to pass to the next googleusercontent.com

Code: [Select]
uses
  SysUtils, Classes, StrUtils,
  fphttpclient, URIParser, opensslsockets, fgl;

type
  //https://serverfault.com/questions/153409/can-subdomain-example-com-set-a-cookie-that-can-be-read-by-example-com

  { TSessionCookies }

  TSessionCookies = class(TStringList)
  private
    FHost: String;
  public
    property Host: String read FHost;
  end;

  { TSessionsCookies }

  TSessionsCookies = class(specialize TFPGObjectList<TSessionCookies>)
  private
    function Find(AHost: String): TSessionCookies;
    function GetSessions(Index: String): TSessionCookies;
  public
    property Sessions[Index: String]: TSessionCookies read GetSessions; default;
  end;

  { TMyHTTPClient }

  TMyHTTPClient = class(TFPHTTPClient)
  public
    Count: Integer;
    Referer: String; //refere only set to first request
    Sessions: TSessionsCookies;
    constructor Create(AOwner: TComponent); override;
    destructor Destroy; override;
    function CheckResponseCode(ACode: Integer; const AllowedResponseCodes: array of Integer): Boolean; override;
    procedure HTTPMethod(const AMethod, AURL: String; Stream: TStream; const AllowedResponseCodes: array of Integer); override;
    procedure DoNormalRequest(const AURI: TURI; const AMethod: String; AStream: TStream; const AAllowedResponseCodes: array of Integer; AHeadersOnly, AIsHttps: Boolean); override;
    procedure DoRedirect(Sender: TObject; const ASrc: String; var ADest: String);
  end;

procedure DownloadFromGDrive(const URL, FileName: String);
function GetDomain(Host: string): string;

implementation

function GetDomain(Host: string): string;
var
  p: Integer;
begin
  Result := Host;
  if RightStr(Result, 1) = '/' then
    Result := LeftStr(Result, Length(Result) - 1);
  p := pos('.', Result);
  if p >0 then
  begin
    Result := MidBStr(Result, p + 1, MaxInt);
  end;
end;

{ TSessionsCookies }

function TSessionsCookies.Find(AHost: String): TSessionCookies;
var
  i: Integer;
begin
  Result := nil;
  for i := 0 to Count - 1 do
  begin
    if SameText(Items[i].Host, AHost) then
    begin
      Result := Items[i];
      break;
    end;
  end;
end;

function TSessionsCookies.GetSessions(Index: String): TSessionCookies;
begin
  Result := Find(Index);
  if Result = nil then
  begin
    Result := TSessionCookies.Create;
    Result.FHost := Index;
    Add(Result);
  end;
end;

{ TMyHTTPClient }

constructor TMyHTTPClient.Create(AOwner: TComponent);
begin
  inherited Create(AOwner);
  Sessions := TSessionsCookies.Create(True);
end;

function TMyHTTPClient.CheckResponseCode(ACode: Integer; const AllowedResponseCodes: array of Integer): Boolean;
begin
  Result := inherited CheckResponseCode(ACode, AllowedResponseCodes);
  if ACode = 302 then
    Result := True;
end;

destructor TMyHTTPClient.Destroy;
begin
  FreeAndNil(Sessions);
  inherited Destroy;
end;

procedure TMyHTTPClient.HTTPMethod(const AMethod, AURL: String; Stream: TStream; const AllowedResponseCodes: array of Integer);
var
  aHost: string;
  URI: TURI;
begin
  inherited;
  {$ifdef GDDownload_LOG}
  WriteLn('to ' + AURL);
  {$endif}
  Referer := AURL;
  URI := ParseURI(AURL);
  aHost := URI.Host;
  aHost := GetDomain(aHost);
  Sessions[aHost].Assign(Cookies);
  Count := 0;
end;

procedure TMyHTTPClient.DoNormalRequest(const AURI: TURI; const AMethod: String; AStream: TStream; const AAllowedResponseCodes: array of Integer; AHeadersOnly, AIsHttps: Boolean);
var
  aHost: String;
begin
  aHost := GetDomain(AURI.Host);

  Cookies.Assign(Sessions[aHost]);

  if Referer <> '' then
    AddHeader('referer', Referer);

  AddHeader('User-Agent', 'Wget/1.13.4 (linux-gnu)');
  AddHeader('Connection', 'Keep-Alive');

  {$ifdef GDDownload_LOG}
  WriteLn('##### ' + aHost + ' Count: ', Count, ' #####');

  WriteLn;
  WriteLn('-----------------Request Headers---------------------');
  WriteLn(RequestHeaders.Text);

  WriteLn('-----------------Sending Cookies---------------------');
  WriteLn(Cookies.Text);
  {$endif}

  inherited;

  {$ifdef GDDownload_LOG}
  WriteLn('-----------------Response Headers---------------------');
  WriteLn(ResponseHeaders.Text);

  WriteLn('-----------------Response Cookies---------------------');
  WriteLn(Cookies.Text);
  {$endif}

  if Trim(Cookies.Text) <> '' then
    Sessions[aHost].Assign(Cookies);
  Count := Count + 1;
end;

procedure TMyHTTPClient.DoRedirect(Sender: TObject; const ASrc: String; var ADest: String);
begin
 // Referer := ASrc; //nop
  {$ifdef GDDownload_LOG}
  WriteLn;
  WriteLn('---------------------- Redirect -----------------------');
  WriteLn('Redirect to ' + ADest);
  {$endif}
end;

procedure DownloadFromGDrive(const URL, FileName: String);
var
  FileID, Token: String;
  HttpClient: TMyHTTPClient;

  function GetDownloadId(const URL: String): String;
  var
    p, i: Word;
    R: String;
  begin
    Result := '';
    p := 0;
    p := pos('?id=', URL);
    if p <> 0 then p := p + 4
    else
    begin
      p := pos('/d/', URL);
      if p <> 0 then p := p + 3;
    end;
    i := p;
    while URL[i] in ['a'..'z', 'A'..'Z', '0'..'9'] do
    begin
      Result := Result + URL[i];
      if i < length(URL) then Inc(i);
    end;
  end;

  function GetConfirmToken(const Html: String): String;
  var
    p1, p2: Word;
  begin
    p1 := pos('confirm=', Html);
    if p1 <> 0 then
    begin
      p2 := PosEx('&amp', Html, p1);
      if p2 <> 0 then
        Result := copy(HTML, p1 + 8, p2 - p1 - 8);
    end;
  end;

var
  DownloadUrl: String;
  HtmlResult: TStringList;

begin
  HttpClient := TMyHTTPClient.Create(nil);
  HtmlResult := TStringList.Create;
  with HttpClient do
    try
      MaxRedirects := 5;
      OnRedirect := @DoRedirect;
      AddHeader('User-Agent', 'Wget/1.13.4 (linux-gnu)');
      FileID := GetDownloadID(URL);
      DownloadUrl := 'https://drive.google.com/uc?export=download&id=' + FileID;
      {$ifdef GDDownload_LOG}
      WriteLn('attempting to get ' + DownloadUrl);
      {$endif}
      HtmlResult.Text := Get(DownloadUrl);
      Token := GetConfirmToken(HtmlResult.Text);
      {$ifdef GDDownload_LOG}
      Writeln('Download ID: ', FileID);
      writeln('Confirm Token: ' + Token);
      Writeln;
      Writeln('==============================================');
      {$endif}

      try
        AllowRedirect := True;
        DownloadUrl := 'https://drive.google.com/uc?export=download&confirm=' + Token + '&id=' + FileID;
        Get(DownloadUrl, FileName);
      except
        on E: Exception do
        begin
          {$ifdef GDDownload_LOG}
          writeln(E.Message);
          {$endif}
          raise;
        end;
      end;
      {$ifdef GDDownload_LOG}
      writeln('got codes ' + IntToStr(ResponseStatusCode));
      {$endif}
    finally
      HtmlResult.Free;
      Free;
    end;
end;


rvk

  • Hero Member
  • *****
  • Posts: 4465
Re: Trying to download file from google drive (unauthenticated)
« Reply #12 on: January 24, 2021, 08:30:09 pm »
So, cookies are not saved when doing multiple calls (which seems to be confirmed by Zahers post).

Nasty bug which doesn't seem to be fixed yet.

https://forum.lazarus.freepascal.org/index.php?topic=40516.msg391714#msg391714

 

TinyPortal © 2005-2018