Recent

Author Topic: Cannot connect to https://api.1nce.com using TFPHTTPClient  (Read 4352 times)

trev

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1014
  • Former Delphi 1-7, 10.2 User
Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
« Reply #15 on: July 18, 2020, 08:43:51 am »
On the Mac mini (details in sig) I have no problem with HTTPS connections using fphttpclient on Lazarus 2.0.6, 2.0.8 (FPC 3.0.4 or trunk) nor with 2.0.10 (FPC 3.2.0 or trunk).

Pre-FPC 3.2.0 I did have problems with HTTPS connections on FreeBSD and Ubuntu unless using FPC trunk.

Sorry, I have no idea why you're having the issue.
o Lazarus v2.1.0 r63871, FPC v3.3.1 r47164, macOS 10.14.6 (with sup update), Xcode 11.3.1
o Lazarus v2.1.0 r61574, FPC v3.3.1 r42318, FreeBSD 12.1 amd64 (VMware Fusion VM)
o FPC 3.0.4, FreeBSD 12.2-STABLE r365646 amd64
o Lazarus v2.1.0 r61574, FPC v3.0.4, Ubuntu 20.04 (Parallels VM)

wittbo

  • Full Member
  • ***
  • Posts: 140
Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
« Reply #16 on: August 11, 2020, 05:35:26 pm »
Sorry, I haven't been home for some time.

Today, I made a lot of tests and the result is: 
-  On connecting to www.ariva.de connection fails with message "Connect to www.ariva.de: 443 failed"
-  Failure only occurs with Lazarus 2.0.10 (fpc 3.2.0) on...
-  ...MacOS (High Sierra and Mojave)

No errors when
-   using Lazarus < 2.0.10 (fpc <= 3.0.4)
-   using Windows 10 (even with Laz 2.0.10 / fpc 3.2.0)
-   connecting to other sites (i.e google.com, ...)

I have no idea, what's going wrong, but it must be a https / SSL problem.
You can download my little program for testing purposes.

@Handoko:  Could you put your simple web page loader program at my disposal , so I can check if it is running with my environment? Thank you in advance.
-wittbo-
MBAir with MacOS 10.14.6 / Lazarus 2.0.10
iMac with MacOS 10.13.6 / Lazarus 2.0.2

Thaddy

  • Hero Member
  • *****
  • Posts: 10526
Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
« Reply #17 on: August 11, 2020, 05:54:43 pm »
Common problem. Dump anything below TLS 1.1. That is really the minimum that is acceptable.
So check your protocol settings.

Modern browsers (all of them) will drop unsafe protocols. You should do too.
I think that is the difference: 3.0.4 still had a fallback. 3.2.0 does not and complies to current standards.
BTW Recent OpenSSL does not even support the old protocols... not even SSL...unless compiled yourself. 8-)
If there are servers in the wild that still support those old and compromised  protocols they should be shut down.
« Last Edit: August 11, 2020, 06:07:28 pm by Thaddy »

wittbo

  • Full Member
  • ***
  • Posts: 140
Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
« Reply #18 on: August 11, 2020, 07:54:53 pm »
Sounds reasonable.

But what I don't understand, is, that...
-   ... Laz 2.0.10 / fpc 3.2.0 / openssl running on Windows 10 can load www.ariva.de without any problem
-   ... really each browser can open www.ariva.de without any problem.
-wittbo-
MBAir with MacOS 10.14.6 / Lazarus 2.0.10
iMac with MacOS 10.13.6 / Lazarus 2.0.2

trev

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1014
  • Former Delphi 1-7, 10.2 User
Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
« Reply #19 on: August 12, 2020, 04:09:10 am »
@wittbo - it seems to be the difference between FPC 3.0.4 and FPC 3.2.0/FPC 3.3.1 (trunk).

* Lazarus 2.0.10 with FPC 3.0.4 works
* Lazarus 2.0.10 with FPC 3.3.1 fails.

(Tests on macOS 10.14.6 using the unit below.)

Code: Pascal  [Select][+][-]
  1. unit Unit1;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. interface
  6.  
  7. uses
  8.   Classes, SysUtils, Forms, Controls, Graphics, Dialogs
  9.   , fphttpclient, LCLIntf, LCLType, StdCtrls; //, OpenSSLSockets; -- needed for 3.3.1
  10.  
  11. type
  12.  
  13.   { TForm1 }
  14.  
  15.   TForm1 = class(TForm)
  16.     Button1: TButton;
  17.     procedure Button1Click(Sender: TObject);
  18.   private
  19.  
  20.   public
  21.  
  22.   end;
  23.  
  24. var
  25.   Form1: TForm1;
  26.  
  27. implementation
  28.  
  29. {$R *.lfm}
  30.  
  31. function GetWebPage(const URL: string): string;
  32. var
  33.   Client: TFPHttpClient;
  34. begin
  35.   Client := TFPHttpClient.Create(nil);
  36.   Try
  37.     Client.AllowRedirect := true;
  38.     Client.AddHeader('User-Agent', 'Mozilla/5.0(compatible; fpweb)');
  39.     Result := Client.Get(URL);
  40.   except
  41.       on E: Exception do
  42.            ShowMessage('Retrieval of: ' + URL + LineEnding
  43.                        + 'Failed with error: ' + E.Message + LineEnding
  44.                        + 'HTTP code: ' + IntToSTr(Client.ResponseStatusCode)+ LineEnding);
  45.   end;
  46. end;
  47.  
  48. { TForm1 }
  49.  
  50. procedure TForm1.Button1Click(Sender: TObject);
  51. begin
  52.   ShowMessage(GetWebPage('https://www.ariva.de/'));
  53. end;
  54.  
  55. end.

Time to log a bug?
o Lazarus v2.1.0 r63871, FPC v3.3.1 r47164, macOS 10.14.6 (with sup update), Xcode 11.3.1
o Lazarus v2.1.0 r61574, FPC v3.3.1 r42318, FreeBSD 12.1 amd64 (VMware Fusion VM)
o FPC 3.0.4, FreeBSD 12.2-STABLE r365646 amd64
o Lazarus v2.1.0 r61574, FPC v3.0.4, Ubuntu 20.04 (Parallels VM)

Thaddy

  • Hero Member
  • *****
  • Posts: 10526
Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
« Reply #20 on: August 12, 2020, 05:48:06 am »
Note that the Pascal code now queries from high to low, whereas in the past it queried from low to high!
e.g. it first looks for tls 1.2 then tls 1.1 and if that's not there drops connection without further warning.. That's a good thing and not a bug.
I believe I have already posted an example here on the forum that basically drops ssl2/3/tls1 too. Check your code for any of those three. They should not be there.

Check browser string here:
https://developers.whatismybrowser.com/useragents/parse/?analyse-my-user-agent=yes
« Last Edit: August 12, 2020, 06:09:36 am by Thaddy »

trev

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1014
  • Former Delphi 1-7, 10.2 User
Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
« Reply #21 on: August 12, 2020, 06:40:56 am »
The web browsers I tried (SeaMonkey + Firefox + Safari) can retrieve the url, FPC 3.04 can retrieve the url, FPC 3.2.0/3.3.1 cannot retrieve the url.

Ergo, the problem is FPC 3.2.0/3.3.1.
o Lazarus v2.1.0 r63871, FPC v3.3.1 r47164, macOS 10.14.6 (with sup update), Xcode 11.3.1
o Lazarus v2.1.0 r61574, FPC v3.3.1 r42318, FreeBSD 12.1 amd64 (VMware Fusion VM)
o FPC 3.0.4, FreeBSD 12.2-STABLE r365646 amd64
o Lazarus v2.1.0 r61574, FPC v3.0.4, Ubuntu 20.04 (Parallels VM)

trev

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1014
  • Former Delphi 1-7, 10.2 User
Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
« Reply #22 on: August 12, 2020, 12:42:57 pm »
@wittbo

I've bene meaning to investigate using macOS native code for ages, so I spent some time today and came up with this which works:

Code: Pascal  [Select][+][-]
  1. procedure TForm1.Button2Click(Sender: TObject);
  2. var
  3.   URL: NSURL;
  4.   urlData : NSData;
  5.   urlRequest : NSUrlRequest;
  6.   urlResponse: NSURLResponse;
  7.   urlConnection: NSURLConnection;
  8.   error: NSError;
  9.   body: NSString;
  10. begin
  11.   URL := NSURL.URLWithString(NSSTR(PAnsiChar('https://www.ariva.de/')));
  12.   if(Url = Nil) then
  13.     ShowMessage('NSURL.URLWithString failed!');
  14.  
  15.   urlRequest := NSURLRequest.requestWithURL(URL);
  16.  
  17.   urlConnection := NSURLConnection.alloc.init;
  18.   urlData := urlConnection.sendSynchronousRequest_returningResponse_error(
  19.       urlRequest,
  20.       @urlResponse,
  21.       @error
  22.     );
  23.  
  24.   Body := NSString.alloc.initWithData(urlData,NSUTF8StringEncoding);
  25.  
  26.   ShowMessage(NSStringToString(Body));
  27.   ShowMessage(NSStringToString(urlData.description));
  28.   ShowMessage(NSStringToString(urlResponse.description));
  29. end;

I'm using FPC 3.3.1 (trunk) and had to add the missing initWithData function to the NSString class b editing /usr/local/share/fpcsrc/fpc-3.3.1/packages/cocoaint/src/foundation/NSString.inc to add the missing function as follows:

Code: Pascal  [Select][+][-]
  1. --- NSString.inc        (revision 45778)
  2. +++ NSString.inc        (working copy)
  3. @@ -105,6 +105,7 @@
  4.      function characterAtIndex (index: NSUInteger): unichar; message 'characterAtIndex:';
  5.      function init: instancetype; message 'init'; { NS_DESIGNATED_INITIALIZER }
  6.      function initWithCoder (aDecoder: NSCoder): instancetype; message 'initWithCoder:'; { NS_DESIGNATED_INITIALIZER }
  7. +    function initWithData(data: NSData; encoding: NSStringEncoding) : instancetype; message 'initWithData:encoding:';
  8.  
  9.      { Adopted protocols }
  10.      function copyWithZone (zone: NSZonePtr): id; message 'copyWithZone:';

This should also work with FPC 3.2.0.
« Last Edit: August 12, 2020, 12:48:12 pm by trev »
o Lazarus v2.1.0 r63871, FPC v3.3.1 r47164, macOS 10.14.6 (with sup update), Xcode 11.3.1
o Lazarus v2.1.0 r61574, FPC v3.3.1 r42318, FreeBSD 12.1 amd64 (VMware Fusion VM)
o FPC 3.0.4, FreeBSD 12.2-STABLE r365646 amd64
o Lazarus v2.1.0 r61574, FPC v3.0.4, Ubuntu 20.04 (Parallels VM)

Bi0T1N

  • Jr. Member
  • **
  • Posts: 50
Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
« Reply #23 on: August 12, 2020, 01:26:45 pm »
The web browsers I tried (SeaMonkey + Firefox + Safari) can retrieve the url, FPC 3.04 can retrieve the url, FPC 3.2.0/3.3.1 cannot retrieve the url.

Ergo, the problem is FPC 3.2.0/3.3.1.

Did you add opensslsockets (OpenSSL) or gnutlssockets (GNU TLS) to your uses list? (https://wiki.freepascal.org/User_Changes_3.2.0#fpHTTPClient_and_fpHTTPServer_Units)

It's also important to mention your installed OpenSSL version as there were many API changes between 1.0.2 and 1.1.x. I also don't know if both are fully supported by opensslsockets.
However, I assume that if no SSL version is defined in your code it uses SSLv23_method (1.0.2) or TLS_method (1.1.x) which negotiates the highest version mutually supported by the client and the server. Thus it should work once the library is successfully loaded.
« Last Edit: August 12, 2020, 01:35:31 pm by Bi0T1N »

Thaddy

  • Hero Member
  • *****
  • Posts: 10526
Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
« Reply #24 on: August 12, 2020, 01:34:07 pm »
Apart from that -which could cause an issue indeed -, and my previous remarks, set AllowRedirects to true. But I still bet you are using a deprecated protocol or an old version of OpenSSL. Happens all the time. Make sure - really sure - your code does not reference to SSL2/3/TLS1.0.

In 3.2.0 this should not even be necessary because the behavior has changed from newest to oldest automatically..

Basically... OpenSSL does not support SSL   8-) anymore and modern browsers behave the same. Maybe OpenTLS?

TLS 1.1 is the bare minimum and TLS 1.2 (or 1.3, which is current) is advised. If servers or browsers still allow fall-backs beyond that, the servers are not secure and not very well maintained.
https://en.wikipedia.org/wiki/Transport_Layer_Security
Read and weep.
« Last Edit: August 12, 2020, 01:53:42 pm by Thaddy »

trev

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1014
  • Former Delphi 1-7, 10.2 User
Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
« Reply #25 on: August 12, 2020, 03:07:58 pm »
Did you add opensslsockets (OpenSSL) or gnutlssockets (GNU TLS) to your uses list? (https://wiki.freepascal.org/User_Changes_3.2.0#fpHTTPClient_and_fpHTTPServer_Units)

Yep - see my code quoted in a previous post.

Quote
It's also important to mention your installed OpenSSL version as there were many API changes between 1.0.2 and 1.1.x. I also don't know if both are fully supported by opensslsockets.
However, I assume that if no SSL version is defined in your code it uses SSLv23_method (1.0.2) or TLS_method (1.1.x) which negotiates the highest version mutually supported by the client and the server. Thus it should work once the library is successfully loaded.

I have not defined a specific version of SSL/TLS (again, see code posted in previous post).

I played around with one of my web server's allowable protocols and it seems that if I enable TLSv1 then using FPC 3.2.0/3.3.1, fphttpclient can connect. If I disable TLSv1, then fphttpclient cannot connect.

Note that with both TLSv1 and TLSv1.1 protocols disabled, fphttpclient can connect only if I compile with FPC 3.0.4. It fails to connect if I compile with FPC 3.2.0/3.3.1.

Getting back to @wittbo's specific issue - www.ariva.de allows TLSv1.1, v1.2 and v1.3 - No TLSv1 and so no successful connect with FPC 3.2.0/3.3.1. Of course, no problem with FPC 3.0.4.

Still looks like an issue with FPC 3.2.0/3.3.1 to me.
o Lazarus v2.1.0 r63871, FPC v3.3.1 r47164, macOS 10.14.6 (with sup update), Xcode 11.3.1
o Lazarus v2.1.0 r61574, FPC v3.3.1 r42318, FreeBSD 12.1 amd64 (VMware Fusion VM)
o FPC 3.0.4, FreeBSD 12.2-STABLE r365646 amd64
o Lazarus v2.1.0 r61574, FPC v3.0.4, Ubuntu 20.04 (Parallels VM)

wittbo

  • Full Member
  • ***
  • Posts: 140
Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
« Reply #26 on: August 12, 2020, 05:55:05 pm »
Thanks to all for the interesting comments; since I am a leisure time programmer only, I could not understand each detail. So the recent statement from trev is the logical result of all your efforts. What I will do, is...
... open a bug report
... try (for macos platform only) the NSURL solution introduced by Trev. @Trev: which units must the uses clause contain?
-wittbo-
MBAir with MacOS 10.14.6 / Lazarus 2.0.10
iMac with MacOS 10.13.6 / Lazarus 2.0.2

Bi0T1N

  • Jr. Member
  • **
  • Posts: 50
Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
« Reply #27 on: August 12, 2020, 05:59:45 pm »
Still looks like an issue with FPC 3.2.0/3.3.1 to me.
For me it looks like an issue specific to MacOS (or your setup) because here it's working fine with Free Pascal Compiler version 3.3.1-r45857 and OpenSSL Binaries Win-64 1.1.1g.
Just for the case that it loads an arbitrary OpenSSL library somewhere in your path try to print the OpenSSL version with:
Code: Pascal  [Select][+][-]
  1.  writeln(SSLeayversion(0));
  2.  writeln(OpenSSLGetVersion(0));
whereas the second function doesn't print anything for me but should print the same information. (see #37555)

wittbo

  • Full Member
  • ***
  • Posts: 140
Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
« Reply #28 on: August 12, 2020, 07:19:10 pm »
Absolutely correct.

As I wrote above, this is a macOS / Laz 2.0.10 / fcp 3.2.0 issue only;
but independant from the macOS version (tested on High Sierra and Mojave).

On Win10 / Laz 2.0.10 / fcp 3.2.0 no errors.

--> macOS problem:  compiler or openSSL library?

@Bi0T1N:  I could not found the noted calls for the  SSL version (may be for windows only); openSSL is part of the macOS operating system. Open macOS Terminal and type openssl and then type version. This command returns LibreSSL 2.6.5.

Quote
OpenSSLGetVersion(0)
-wittbo-
MBAir with MacOS 10.14.6 / Lazarus 2.0.10
iMac with MacOS 10.13.6 / Lazarus 2.0.2

Bi0T1N

  • Jr. Member
  • **
  • Posts: 50
Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
« Reply #29 on: August 12, 2020, 10:37:06 pm »
As I wrote above, this is a macOS / Laz 2.0.10 / fcp 3.2.0 issue only;
but independant from the macOS version (tested on High Sierra and Mojave).
Sorry, seems I didn't paid enough attention to this.

@Bi0T1N:  I could not found the noted calls for the  SSL version (may be for windows only); openSSL is part of the macOS operating system. Open macOS Terminal and type openssl and then type version. This command returns LibreSSL 2.6.5.

Quote
OpenSSLGetVersion(0)
You need to add openssl to the uses list to use the previous mentioned functions.

However, your LibreSSL version is kinda old but it should still work fine with the website as long as it supports TLS1.2 and the needed ciphers. According to the SSL Server Test even OpenSSL 1.0.1l can connect. Maybe try to connect to it through the provided command line utilities and if this works the issue is somewhere in FPC. %)

 

TinyPortal © 2005-2018