Recent

Author Topic: Reading images online to files with http://  (Read 416 times)

Birger52

  • Sr. Member
  • ****
  • Posts: 270
Reading images online to files with http://
« on: July 14, 2020, 11:33:49 pm »
I have a series of images, I need to read from online.
But I do not know if they are all there. Or rather, I know, that they are not all there, but I don't know which are not.
So I send a Head request, and based on the returned status, I can read the file or not.
There is however a problem - the returned status from the Head request is always 0 - which it should never be.

Anybody has any ideas or experience with this?

Code looks like this:
Code: Pascal  [Select][+][-]
  1. procedure TFetchForm.CheckImg(aPath,aCard,aImg:string);
  2. var
  3.   logStr, imgNavn, imgF, url : string;
  4.   Client : TFPHttpClient;
  5.   hds : TStringList;
  6. begin
  7.   imgF := aCard+aImg;
  8.   ImgNavn := aPath + imgF;
  9.   if FileExists(ImgNavn) then logStr := imgF + '  exists'
  10.   else begin // and this where img shold be downloaded ...
  11.     url := url1 + aCard + url2 + imgF;
  12.     hds := TStringList.Create;
  13.     Client := TFPHttpClient.Create(Self);
  14.     try
  15.       Client.Head(url, hds);
  16.     finally
  17.     end;
  18.     if Client.ResponseStatusCode = 200 then
  19.       try
  20.         Client.Get(url, ImgNavn);
  21.         logStr :=  imgF + '  created from download';
  22.       except
  23.         on E : Exception do
  24.           logStr := 'Error reading ' + url + '   ' + E.Message;
  25.       end else
  26.         logStr := imgF + ' not found on server  ' + url;// + '   ' + E.Message;
  27.     Client.Free;
  28.     hds.Free
  29.   end;
  30.   MemoLog.Lines.Add(logStr);
  31. end;
  32.  

Lazarus 2.0.8 FPC 3.0.4
Win7 64bit
Playing and learning - strictly for my own pleasure.

jamie

  • Hero Member
  • *****
  • Posts: 3537
Re: Reading images online to files with http://
« Reply #1 on: July 15, 2020, 01:11:29 am »
Yup. it returns no status other than 0.

You can use the contents of your string list, if it's empty then It didn't get anything...

I am not so sure if the Status Code is intended to be that of the server response code, it may simply be 0 for no errors.

 One way to know is use a bogus URL and see what the response is.
The only true wisdom is knowing you know nothing

Birger52

  • Sr. Member
  • ****
  • Posts: 270
Re: Reading images online to files with http://
« Reply #2 on: July 15, 2020, 10:39:06 am »
Response to a HEAD is to be the exact same as the response to a GET, but without the body.
(https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.3) - so it should be 200 or 304 if the url asked for exists, 404 if not (or redirection codes or whatever)
This is definitely a flaw.

I have tried reading the returned headers, but they do not contain any indication of the status code.
It returns all the headers (mostly status of the connection - not  the contens) Response 0 and raises an exception "unexcpected statuscode 404", instead of actually returning the statuscode.

Problem is, that the line Client.Get(url, ImgNavn); actually creates the image even when the image does not exist online, and the request actually fails.
And the odd thing is, that here TFPHttpClient actually returns the correct response code.

For now I have skipped checking, and simply delete the image again, if an image was not downloaded. This is not optimal.

Code: Pascal  [Select][+][-]
  1. procedure TFetchForm.CheckImg(aPath,aCard,aImg:string);
  2. var
  3.   logStr, imgNavn, imgF, url : string;
  4.   Client : TFPHttpClient;
  5. begin
  6.   imgF := aCard+aImg;
  7.   ImgNavn := aPath + imgF;
  8.   if FileExists(ImgNavn) and (TextfileSize(ImgNavn) > 0) then logStr := imgF + '  exists'
  9.   else begin // and this where img shold be downloaded ...
  10.     url := url1 + aCard + url2 + imgF;
  11.     Client := TFPHttpClient.Create(Self);
  12.     try
  13.       Client.Get(url, ImgNavn);
  14.       logStr :=  imgF + '  created from download';
  15.     except
  16.       on E : Exception do begin
  17.         if Client.ResponseStatusCode = 404 then
  18.           logStr := imgF + ' not found on server'
  19.         else logStr := 'Error reading ' + url;// + '   ' + E.Message;
  20.         DeleteFile(ImgNavn);
  21.       end;
  22.     end;
  23.     Client.Free;
  24.   end;
  25.   MemoLog.Lines.Add(logStr);
  26. end;
  27.  

Lazarus 2.0.8 FPC 3.0.4
Win7 64bit
Playing and learning - strictly for my own pleasure.

trev

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 946
  • Former Delphi 1-7, 10.2 User
Re: Reading images online to files with http://
« Reply #3 on: July 15, 2020, 12:10:55 pm »
As you're using Windows you could use the Microsoft WinInet API. This is what I use to download a web page. It should be easy to adapt to downloading your image files.

Code: Pascal  [Select][+][-]
  1. {$IFDEF WINDOWS}
  2. // Need to use Windows WinInet to avoid issue with HTTPS
  3. // needing two OpenSSL DLLs to be provided with application
  4. // if using TFPHttpClient.
  5. // The WinINet API also gets any connection and proxy settings
  6. // set by Internet Explorer. Blessing or curse?
  7.  
  8. function GetMicrochipPage(const Url: string): string;
  9. var
  10.   NetHandle: HINTERNET;
  11.   UrlHandle: HINTERNET;
  12.   Buffer: array[0..1023] of Byte;
  13.   BytesRead: dWord;
  14.   StrBuffer: UTF8String;
  15. begin
  16.   Result := '';
  17.   NetHandle := InternetOpen('Mozilla/5.0(compatible; WinInet)', INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0);
  18.  
  19.   // NetHandle valid?
  20.   if Assigned(NetHandle) then
  21.     Try
  22.       UrlHandle := InternetOpenUrl(NetHandle, PChar(Url), nil, 0, INTERNET_FLAG_RELOAD, 0);
  23.  
  24.       // UrlHandle valid?
  25.       if Assigned(UrlHandle) then
  26.         Try
  27.           repeat
  28.             InternetReadFile(UrlHandle, @Buffer, SizeOf(Buffer), BytesRead);
  29.             SetString(StrBuffer, PAnsiChar(@Buffer[0]), BytesRead);
  30.             Result := Result + StrBuffer;
  31.           until BytesRead = 0;
  32.         Finally
  33.           InternetCloseHandle(UrlHandle);
  34.         end
  35.       // o/w UrlHandle invalid
  36.       else
  37.         ShowMessage('Cannot open URL: ' + Url);
  38.     Finally
  39.       InternetCloseHandle(NetHandle);
  40.     end
  41.   // NetHandle invalid
  42.   else
  43.     raise Exception.Create('Unable to initialize WinInet');
  44. end;
  45. {$ENDIF}
  46.  

WinInet docs: https://docs.microsoft.com/en-us/windows/win32/wininet/about-wininet
o Lazarus v2.1.0 r63871, FPC v3.3.1 r46876, macOS 10.14.6 (with sup update), Xcode 11.3.1
o Lazarus v2.1.0 r61574, FPC v3.3.1 r42318, FreeBSD 12.1 amd64 (VMware Fusion VM)
o FPC 3.0.4, FreeBSD 12.2-STABLE r365646 amd64
o Lazarus v2.1.0 r61574, FPC v3.0.4, Ubuntu 18.04 (Parallels VM)

Birger52

  • Sr. Member
  • ****
  • Posts: 270
Re: Reading images online to files with http://
« Reply #4 on: July 15, 2020, 03:27:26 pm »
Thx.
I'll look at that.
Lazarus 2.0.8 FPC 3.0.4
Win7 64bit
Playing and learning - strictly for my own pleasure.

 

TinyPortal © 2005-2018