Recent

Author Topic: [SOLVED] FPHTTPClient + GitHub = :)  (Read 2345 times)

Trenatos

  • Sr. Member
  • ****
  • Posts: 454
  • Software developer - Open source contributor
    • MarcusFernstrom.com
[SOLVED] FPHTTPClient + GitHub = :)
« on: April 11, 2018, 05:49:30 am »
I'm working on a CLI tool, it's working nicely but I've run into a strange issue.

I'm using FPHTTPClient to download various files, works fine, except with GitHub.

This works: WriteLn( TFPHTTPClient.SimpleGet('https://httpbin.org/uuid') );
This doesn't: WriteLn( TFPHTTPClient.SimpleGet('https://api.github.com/repos/OpenBD/openbd-core/releases/latest') );

I have the OpenSSL DLLs and that part seems fine, but when I try hitting the GitHub API nothing seems to work.

ResponseStatusCode returns 0, and data just spits out a blank line.

I'm lost..  Anyone seen this behaviour before?

Code: Pascal  [Select]
  1.   FHTTPClient := TFPHTTPClient.Create(nil);
  2.   try
  3.     FHTTPClient.AllowRedirect:=true;
  4.     data := FHTTPClient.Get( 'http://api.github.com/repos/OpenBD/openbd-core/releases/latest' );
  5.     WriteLn(FHTTPClient.ResponseStatusCode);
  6.     WriteLn( data );
  7.   finally
  8.     FHTTPClient.Free;
  9.   end;

The url works fine in my browser, it works fine in Insomnia (Desktop REST utility)
« Last Edit: April 20, 2018, 05:54:49 am by Trenatos »

Cyrax

  • Hero Member
  • *****
  • Posts: 624
Re: FPHTTPClient + GitHub = :(
« Reply #1 on: April 11, 2018, 05:59:43 am »
Maybe setting user agent variable helps?

GetMem

  • Hero Member
  • *****
  • Posts: 3254
Re: FPHTTPClient + GitHub = :(
« Reply #2 on: April 11, 2018, 06:39:06 am »
Try this:
Code: Pascal  [Select]
  1. function GetJSON(const AURL: String; out AJSON: TJSONStringType): Boolean;
  2. var
  3.   Ms: TMemoryStream;
  4.   HTTPClient: TFPHTTPClient;
  5. begin
  6.   Result := False;
  7.   Ms := TMemoryStream.Create;
  8.   try
  9.     HTTPClient := TFPHTTPClient.Create(nil);
  10.     try
  11.       HTTPClient.AllowRedirect := True;
  12.       HTTPClient.AddHeader('User-Agent', 'Mozilla/5.0 (compatible; fpweb)');
  13.       HTTPClient.AddHeader('Content-Type', 'application/json');
  14.       HTTPClient.HTTPMethod('GET', AURL, MS, []);
  15.       if HTTPClient.ResponseStatusCode = 200 then
  16.       begin
  17.         if Ms.Size > 0 then
  18.         begin
  19.           MS.Position := 0;
  20.           SetLength(AJSON, MS.Size);
  21.           MS.Read(Pointer(AJSON)^, Length(AJSON));
  22.           Result := Length(AJSON) > 0;
  23.         end;
  24.       end;
  25.     except
  26.       Result := False;
  27.     end;
  28.   finally
  29.     Ms.Free;
  30.   end;
  31. end;
  32.  
  33. var
  34.   JSON: TJSonStringType;
  35. begin
  36.   if GetJSON('https://api.github.com/repos/OpenBD/openbd-core/releases/latest', JSON) then
  37.     Writeln(JSON);
  38. end.
  39.  

molly

  • Hero Member
  • *****
  • Posts: 2345
Re: FPHTTPClient + GitHub = :(
« Reply #3 on: April 12, 2018, 01:36:52 am »
Maybe setting user agent variable helps?
Not maybe. Github API demands it.

Trenatos

  • Sr. Member
  • ****
  • Posts: 454
  • Software developer - Open source contributor
    • MarcusFernstrom.com
Re: FPHTTPClient + GitHub = :(
« Reply #4 on: April 12, 2018, 02:57:03 pm »
AHA! I hadn't seen that molly, thanks!

I haven't had a chance to try out your solution yet GetMem but hopefully shortly.

Trenatos

  • Sr. Member
  • ****
  • Posts: 454
  • Software developer - Open source contributor
    • MarcusFernstrom.com
Re: FPHTTPClient + GitHub = :(
« Reply #5 on: April 13, 2018, 02:41:14 am »
molly, GetMem, I'm adding you to my "owe beer to" list  :)

Indeed correct, it works with the user agent header, I did follow the GitHub API info and use the application name

Trenatos

  • Sr. Member
  • ****
  • Posts: 454
  • Software developer - Open source contributor
    • MarcusFernstrom.com
Re: [SOLVED] FPHTTPClient + GitHub = :)
« Reply #6 on: April 19, 2018, 04:56:47 pm »
I spoke too soon.

Adding the user agent solved the problem on Windows, but I'm still having the same issue on OSX.

GetMem, your code gives the same result on Mac as my original code, can't connect to GitHub and a response code of 0.

I did add openssl dylib to the same directory as the executable (Though it's also installed on the system)

Trenatos

  • Sr. Member
  • ****
  • Posts: 454
  • Software developer - Open source contributor
    • MarcusFernstrom.com
Re: FPHTTPClient + GitHub = :)
« Reply #7 on: April 19, 2018, 05:13:55 pm »
Tested with https://httpbin.org/get and it does return and outputs the json data, but not with GitHub.


WriteLn( HTTPClient.ResponseHeaders.Text ); outputs nothing for GitHub
WriteLn( HTTPClient.ResponseStatusText ); also outputs nothing for GitHub

Phil

  • Hero Member
  • *****
  • Posts: 2750
Re: [SOLVED] FPHTTPClient + GitHub = :)
« Reply #8 on: April 19, 2018, 05:36:15 pm »
GetMem, your code gives the same result on Mac as my original code, can't connect to GitHub and a response code of 0.

I did add openssl dylib to the same directory as the executable (Though it's also installed on the system)

That code works fine here with macOS 10.13. What version of macOS is your Mac running?

Note OpenSSL has been deprecated for years on Mac. Mac built-in libraries work fine. See "Useful" source here for ns_url_request.pas unit.

https://macpgmr.github.io

Trenatos

  • Sr. Member
  • ****
  • Posts: 454
  • Software developer - Open source contributor
    • MarcusFernstrom.com
Re: FPHTTPClient + GitHub = :)
« Reply #9 on: April 19, 2018, 05:40:44 pm »
El Capitan - 10.11.5

Bizarre that it works for you but not for me  :/

Phil

  • Hero Member
  • *****
  • Posts: 2750
Re: FPHTTPClient + GitHub = :)
« Reply #10 on: April 19, 2018, 05:42:48 pm »
El Capitan - 10.11.5

I get the same error as you here on 10.11, so it's probably something out of date. Try the ns_url_request.pas unit - it works fine.

https://developer.apple.com/library/content/documentation/Security/Conceptual/cryptoservices/SecureNetworkCommunicationAPIs/SecureNetworkCommunicationAPIs.html#//apple_ref/doc/uid/TP40011172-CH13-SW3

Here's all you need to do:

Code: Pascal  [Select]
  1.         HTTPClient := TNSHTTPSendAndReceive.Create;
  2.         try
  3.           HTTPClient.Address := AURL;
  4.           Headers := TStringList.Create;
  5.           Headers.Add('User-Agent=Mozilla/5.0 (compatible; fpweb)');
  6.           Headers.Add('Content-Type=application/json');
  7.           if HTTPClient.SendAndReceive(nil, Ms, Headers) then
  8.  

Trenatos

  • Sr. Member
  • ****
  • Posts: 454
  • Software developer - Open source contributor
    • MarcusFernstrom.com
Re: FPHTTPClient + GitHub = :)
« Reply #11 on: April 20, 2018, 05:54:24 am »
Thanks a bunch Phil, that did the trick, I also used that to stream a file download, works like a charm.

I really appreciate the help  :)