Lazarus

Programming => Networking and Web Programming => Topic started by: jeff on June 26, 2020, 08:37:11 am

Title: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: jeff on June 26, 2020, 08:37:11 am
Hi,
as the title says, the following code doesn't work, I get the error message 'Connect to api.1nce.com:443 failed'. The ssl libraries are installed, and I can connect to https websites except for this one. I tried debugging the fphttpclient unit, and the exception fires at line 624, which is FSocket.Connect; in TFPCustomHTTPClient.ConnectToServer procedure. I'm stuck here. Any suggestion how to fix this?

Code: Pascal  [Select][+][-]
  1. SMSGTW:=TFPHTTPClient.Create(nil);
  2. try
  3.    s:=SMSGTW.Get('https://api.1nce.com/management-api/swagger-ui.html');
  4.    Log.Append(s);
  5. except
  6.    on E: Exception do
  7.       ShowMessage(E.Message);
  8.    else raise;
  9. end;
  10.  
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: paweld on June 26, 2020, 09:14:41 am
Code: Pascal  [Select][+][-]
  1. uses
  2.   fphttpclient, opensslsockets;
  3.  
  4. procedure TForm1.Button1Click(Sender: TObject);
  5. var
  6.   SMSGTW: TFPHTTPClient;
  7.   s: String;
  8. begin
  9.   SMSGTW:=TFPHTTPClient.Create(nil);
  10.   try
  11.     s:=SMSGTW.Get('https://api.1nce.com/management-api/swagger-ui.html');
  12.     ShowMessage(s);
  13.   except
  14.     on E: Exception do
  15.     ShowMessage(E.Message);
  16.   end;
  17. end;
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: yus on June 26, 2020, 09:33:18 am
I had a problem with SSL in a multi-threaded application.
Try add
Code: Pascal  [Select][+][-]
  1. InitSSLInterface;
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: jeff on June 26, 2020, 10:34:09 am
Thanks, but it didn't help.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: Cyrax on June 26, 2020, 10:36:44 am
Does that website require special handling e.g it needs API key (or something close to) to passed via cookies?
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: jeff on June 26, 2020, 11:08:28 am
This website has an API that can be used with http calls. I have already implemented this in Java but I need it in freepascal too for a small utility. I didn't want to upload the whole project here, so the code above is only a stripped down example. The site 'https://api.1nce.com/management-api/swagger-ui.html' is a regular website (you can check it in a browser) but it demonstrates the problem as it cannot be connected to with fphttpclient. I don't know if this is due to a bug or I'm doing something wrong, to be honest I'm not really good at network related stuff.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: yus on June 26, 2020, 12:08:35 pm
I tried your code, it works. I get the HTML page of the specified URL.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: jeff on June 26, 2020, 12:30:54 pm
Quote from: yus on June 26, 2020, 12:08:35 pm
I tried your code, it works. I get the HTML page of the specified URL.
What is your free pascal version?
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: jeff on June 26, 2020, 12:57:51 pm
I did a clean lazarus install, and now it is working. Thanks for your help.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: wittbo on July 16, 2020, 06:12:41 pm
I have just the same problem with this URL:  https://www.ariva.de.
Created a little testprogram like your example and could reproduce the error with the URL of jeff.

I believe this has something to do with the OpenSSLSockets unit, since this problem occurs when using Lazarus 2.0.10. In all previous versions of Lazarus, there was no problem loading the webpage and the sslsocket support was automatically compiled into the program.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: Handoko on July 16, 2020, 06:32:43 pm
I had similar or maybe the same issue when using Lazarus 2.0.8. The problem has been solved after I use Lazarus 2.0.10. The problem only happened when connecting to secure protocol (https) not http.

I heard there were some improvements (that can solve the SSL issue) in FPC 3.2.0. Lazarus 2.0.10 is using FPC 3.2.0. Maybe you can try to upgrade your Lazarus or FPC.

To make it works, Now in Lazarus 2.0.10 I have to put opensslsockets in the uses clause and no need to call InitSSLInterface.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: wittbo on July 18, 2020, 05:30:29 am
In my case this is problem, which occurs with 2.0.10 only.
Deinstalled 2.0.10; reinstalled 2.0.8; compile, run:    no problem
Deinstalled 2.0.8; reinstalled 2.0.10; compile, run:   problem 'Connect to www.ariva.de:443 failed'
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: Handoko on July 18, 2020, 06:25:37 am
I am writing a simple web page loader program. I tried to connect arriva front page. I managed to download the html correctly. I'm using Lazarus 2.0.10.

The problem happen only when calling:
IpHtmlPanel1.SetHtml(HTML);

It seems it failed to render the page that contain iframe, see the image below.

Previously, when using Lazarus 2.0.8 it always failed to connect any https sites. Lazarus 2.0.10 fixes the https issue.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: dbannon on July 18, 2020, 06:32:26 am
Wittbo, from your sig, you are using Mac ? and you have not mentioned the FPC version you are using.

On Linux, with FPC3.0.4 there were several ssl problems, one involved FPC looking for a particular name for the ssl libraries that was not normally there.  If you can run strace on the Mac, you might be able to see which ssl libraries your app is trying to use.

One rather silly solution to that problem on Linux was to install the dev library, it made a symlink that did have the name that FPC was looking for.  But this problem went away with FPC320.

Davo
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: wittbo on July 18, 2020, 07:13:49 am
Sorry, my platform is MacBook Air with macOS 10.14.6.
And, as I mentioned, the errors only occurs with Lazarus 2.0.10 (which includes fpc 3.2.0). With all previous Lazarus versions (which include fpc 3.0.4), there is no problem.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: trev on July 18, 2020, 08:43:51 am
On the Mac mini (details in sig) I have no problem with HTTPS connections using fphttpclient on Lazarus 2.0.6, 2.0.8 (FPC 3.0.4 or trunk) nor with 2.0.10 (FPC 3.2.0 or trunk).

Pre-FPC 3.2.0 I did have problems with HTTPS connections on FreeBSD and Ubuntu unless using FPC trunk.

Sorry, I have no idea why you're having the issue.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: wittbo on August 11, 2020, 05:35:26 pm
Sorry, I haven't been home for some time.

Today, I made a lot of tests and the result is: 
-  On connecting to www.ariva.de connection fails with message "Connect to www.ariva.de: 443 failed"
-  Failure only occurs with Lazarus 2.0.10 (fpc 3.2.0) on...
-  ...MacOS (High Sierra and Mojave)

No errors when
-   using Lazarus < 2.0.10 (fpc <= 3.0.4)
-   using Windows 10 (even with Laz 2.0.10 / fpc 3.2.0)
-   connecting to other sites (i.e google.com, ...)

I have no idea, what's going wrong, but it must be a https / SSL problem.
You can download my little program for testing purposes.

@Handoko:  Could you put your simple web page loader program at my disposal , so I can check if it is running with my environment? Thank you in advance.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: Thaddy on August 11, 2020, 05:54:43 pm
Common problem. Dump anything below TLS 1.1. That is really the minimum that is acceptable.
So check your protocol settings.

Modern browsers (all of them) will drop unsafe protocols. You should do too.
I think that is the difference: 3.0.4 still had a fallback. 3.2.0 does not and complies to current standards.
BTW Recent OpenSSL does not even support the old protocols... not even SSL...unless compiled yourself. 8-)
If there are servers in the wild that still support those old and compromised  protocols they should be shut down.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: wittbo on August 11, 2020, 07:54:53 pm
Sounds reasonable.

But what I don't understand, is, that...
-   ... Laz 2.0.10 / fpc 3.2.0 / openssl running on Windows 10 can load www.ariva.de without any problem
-   ... really each browser can open www.ariva.de without any problem.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: trev on August 12, 2020, 04:09:10 am
@wittbo - it seems to be the difference between FPC 3.0.4 and FPC 3.2.0/FPC 3.3.1 (trunk).

* Lazarus 2.0.10 with FPC 3.0.4 works
* Lazarus 2.0.10 with FPC 3.3.1 fails.

(Tests on macOS 10.14.6 using the unit below.)

Code: Pascal  [Select][+][-]
  1. unit Unit1;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. interface
  6.  
  7. uses
  8.   Classes, SysUtils, Forms, Controls, Graphics, Dialogs
  9.   , fphttpclient, LCLIntf, LCLType, StdCtrls; //, OpenSSLSockets; -- needed for 3.3.1
  10.  
  11. type
  12.  
  13.   { TForm1 }
  14.  
  15.   TForm1 = class(TForm)
  16.     Button1: TButton;
  17.     procedure Button1Click(Sender: TObject);
  18.   private
  19.  
  20.   public
  21.  
  22.   end;
  23.  
  24. var
  25.   Form1: TForm1;
  26.  
  27. implementation
  28.  
  29. {$R *.lfm}
  30.  
  31. function GetWebPage(const URL: string): string;
  32. var
  33.   Client: TFPHttpClient;
  34. begin
  35.   Client := TFPHttpClient.Create(nil);
  36.   Try
  37.     Client.AllowRedirect := true;
  38.     Client.AddHeader('User-Agent', 'Mozilla/5.0(compatible; fpweb)');
  39.     Result := Client.Get(URL);
  40.   except
  41.       on E: Exception do
  42.            ShowMessage('Retrieval of: ' + URL + LineEnding
  43.                        + 'Failed with error: ' + E.Message + LineEnding
  44.                        + 'HTTP code: ' + IntToSTr(Client.ResponseStatusCode)+ LineEnding);
  45.   end;
  46. end;
  47.  
  48. { TForm1 }
  49.  
  50. procedure TForm1.Button1Click(Sender: TObject);
  51. begin
  52.   ShowMessage(GetWebPage('https://www.ariva.de/'));
  53. end;
  54.  
  55. end.

Time to log a bug?
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: Thaddy on August 12, 2020, 05:48:06 am
Note that the Pascal code now queries from high to low, whereas in the past it queried from low to high!
e.g. it first looks for tls 1.2 then tls 1.1 and if that's not there drops connection without further warning.. That's a good thing and not a bug.
I believe I have already posted an example here on the forum that basically drops ssl2/3/tls1 too. Check your code for any of those three. They should not be there.

Check browser string here:
https://developers.whatismybrowser.com/useragents/parse/?analyse-my-user-agent=yes
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: trev on August 12, 2020, 06:40:56 am
The web browsers I tried (SeaMonkey + Firefox + Safari) can retrieve the url, FPC 3.04 can retrieve the url, FPC 3.2.0/3.3.1 cannot retrieve the url.

Ergo, the problem is FPC 3.2.0/3.3.1.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: trev on August 12, 2020, 12:42:57 pm
@wittbo

I've bene meaning to investigate using macOS native code for ages, so I spent some time today and came up with this which works:

Code: Pascal  [Select][+][-]
  1. procedure TForm1.Button2Click(Sender: TObject);
  2. var
  3.   URL: NSURL;
  4.   urlData : NSData;
  5.   urlRequest : NSUrlRequest;
  6.   urlResponse: NSURLResponse;
  7.   urlConnection: NSURLConnection;
  8.   error: NSError;
  9.   body: NSString;
  10. begin
  11.   URL := NSURL.URLWithString(NSSTR(PAnsiChar('https://www.ariva.de/')));
  12.   if(Url = Nil) then
  13.     ShowMessage('NSURL.URLWithString failed!');
  14.  
  15.   urlRequest := NSURLRequest.requestWithURL(URL);
  16.  
  17.   urlConnection := NSURLConnection.alloc.init;
  18.   urlData := urlConnection.sendSynchronousRequest_returningResponse_error(
  19.       urlRequest,
  20.       @urlResponse,
  21.       @error
  22.     );
  23.  
  24.   Body := NSString.alloc.initWithData(urlData,NSUTF8StringEncoding);
  25.  
  26.   ShowMessage(NSStringToString(Body));
  27.   ShowMessage(NSStringToString(urlData.description));
  28.   ShowMessage(NSStringToString(urlResponse.description));
  29. end;

I'm using FPC 3.3.1 (trunk) and had to add the missing initWithData function to the NSString class b editing /usr/local/share/fpcsrc/fpc-3.3.1/packages/cocoaint/src/foundation/NSString.inc to add the missing function as follows:

Code: Pascal  [Select][+][-]
  1. --- NSString.inc        (revision 45778)
  2. +++ NSString.inc        (working copy)
  3. @@ -105,6 +105,7 @@
  4.      function characterAtIndex (index: NSUInteger): unichar; message 'characterAtIndex:';
  5.      function init: instancetype; message 'init'; { NS_DESIGNATED_INITIALIZER }
  6.      function initWithCoder (aDecoder: NSCoder): instancetype; message 'initWithCoder:'; { NS_DESIGNATED_INITIALIZER }
  7. +    function initWithData(data: NSData; encoding: NSStringEncoding) : instancetype; message 'initWithData:encoding:';
  8.  
  9.      { Adopted protocols }
  10.      function copyWithZone (zone: NSZonePtr): id; message 'copyWithZone:';

This should also work with FPC 3.2.0.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: Bi0T1N on August 12, 2020, 01:26:45 pm
Quote from: trev on August 12, 2020, 06:40:56 am
The web browsers I tried (SeaMonkey + Firefox + Safari) can retrieve the url, FPC 3.04 can retrieve the url, FPC 3.2.0/3.3.1 cannot retrieve the url.

Ergo, the problem is FPC 3.2.0/3.3.1.

Did you add opensslsockets (OpenSSL) or gnutlssockets (GNU TLS) to your uses list? (https://wiki.freepascal.org/User_Changes_3.2.0#fpHTTPClient_and_fpHTTPServer_Units)

It's also important to mention your installed OpenSSL version as there were many API changes between 1.0.2 and 1.1.x. I also don't know if both are fully supported by opensslsockets.
However, I assume that if no SSL version is defined in your code it uses SSLv23_method (1.0.2) or TLS_method (1.1.x) which negotiates the highest version mutually supported by the client and the server. Thus it should work once the library is successfully loaded.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: Thaddy on August 12, 2020, 01:34:07 pm
Apart from that -which could cause an issue indeed -, and my previous remarks, set AllowRedirects to true. But I still bet you are using a deprecated protocol or an old version of OpenSSL. Happens all the time. Make sure - really sure - your code does not reference to SSL2/3/TLS1.0.

In 3.2.0 this should not even be necessary because the behavior has changed from newest to oldest automatically..

Basically... OpenSSL does not support SSL   8-) anymore and modern browsers behave the same. Maybe OpenTLS?

TLS 1.1 is the bare minimum and TLS 1.2 (or 1.3, which is current) is advised. If servers or browsers still allow fall-backs beyond that, the servers are not secure and not very well maintained.
https://en.wikipedia.org/wiki/Transport_Layer_Security
Read and weep.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: trev on August 12, 2020, 03:07:58 pm
Quote from: Bi0T1N on August 12, 2020, 01:26:45 pm
Did you add opensslsockets (OpenSSL) or gnutlssockets (GNU TLS) to your uses list? (https://wiki.freepascal.org/User_Changes_3.2.0#fpHTTPClient_and_fpHTTPServer_Units)

Yep - see my code quoted in a previous post.

Quote
It's also important to mention your installed OpenSSL version as there were many API changes between 1.0.2 and 1.1.x. I also don't know if both are fully supported by opensslsockets.
However, I assume that if no SSL version is defined in your code it uses SSLv23_method (1.0.2) or TLS_method (1.1.x) which negotiates the highest version mutually supported by the client and the server. Thus it should work once the library is successfully loaded.

I have not defined a specific version of SSL/TLS (again, see code posted in previous post).

I played around with one of my web server's allowable protocols and it seems that if I enable TLSv1 then using FPC 3.2.0/3.3.1, fphttpclient can connect. If I disable TLSv1, then fphttpclient cannot connect.

Note that with both TLSv1 and TLSv1.1 protocols disabled, fphttpclient can connect only if I compile with FPC 3.0.4. It fails to connect if I compile with FPC 3.2.0/3.3.1.

Getting back to @wittbo's specific issue - www.ariva.de allows TLSv1.1, v1.2 and v1.3 - No TLSv1 and so no successful connect with FPC 3.2.0/3.3.1. Of course, no problem with FPC 3.0.4.

Still looks like an issue with FPC 3.2.0/3.3.1 to me.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: wittbo on August 12, 2020, 05:55:05 pm
Thanks to all for the interesting comments; since I am a leisure time programmer only, I could not understand each detail. So the recent statement from trev is the logical result of all your efforts. What I will do, is...
... open a bug report
... try (for macos platform only) the NSURL solution introduced by Trev. @Trev: which units must the uses clause contain?
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: Bi0T1N on August 12, 2020, 05:59:45 pm
Quote from: trev on August 12, 2020, 03:07:58 pm
Still looks like an issue with FPC 3.2.0/3.3.1 to me.
For me it looks like an issue specific to MacOS (or your setup) because here it's working fine with Free Pascal Compiler version 3.3.1-r45857 and OpenSSL Binaries Win-64 1.1.1g (http://wiki.overbyte.eu/arch/openssl-1.1.1g-win64.zip).
Just for the case that it loads an arbitrary OpenSSL library somewhere in your path try to print the OpenSSL version with:
Code: Pascal  [Select][+][-]
  1.  writeln(SSLeayversion(0));
  2.  writeln(OpenSSLGetVersion(0));
whereas the second function doesn't print anything for me but should print the same information. (see #37555 (https://bugs.freepascal.org/view.php?id=37555))
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: wittbo on August 12, 2020, 07:19:10 pm
Absolutely correct.

As I wrote above, this is a macOS / Laz 2.0.10 / fcp 3.2.0 issue only;
but independant from the macOS version (tested on High Sierra and Mojave).

On Win10 / Laz 2.0.10 / fcp 3.2.0 no errors.

--> macOS problem:  compiler or openSSL library?

@Bi0T1N:  I could not found the noted calls for the  SSL version (may be for windows only); openSSL is part of the macOS operating system. Open macOS Terminal and type openssl and then type version. This command returns LibreSSL 2.6.5.

Quote
OpenSSLGetVersion(0)
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: Bi0T1N on August 12, 2020, 10:37:06 pm
Quote from: wittbo on August 12, 2020, 07:19:10 pm
As I wrote above, this is a macOS / Laz 2.0.10 / fcp 3.2.0 issue only;
but independant from the macOS version (tested on High Sierra and Mojave).
Sorry, seems I didn't paid enough attention to this.

Quote from: wittbo on August 12, 2020, 07:19:10 pm
@Bi0T1N:  I could not found the noted calls for the  SSL version (may be for windows only); openSSL is part of the macOS operating system. Open macOS Terminal and type openssl and then type version. This command returns LibreSSL 2.6.5.

Quote
OpenSSLGetVersion(0)
You need to add openssl to the uses list to use the previous mentioned functions.

However, your LibreSSL version is kinda old but it should still work fine with the website as long as it supports TLS1.2 and the needed ciphers. According to the SSL Server Test (https://www.ssllabs.com/ssltest/analyze.html?d=api.1nce.com&s=2001%3a1608%3a6%3a13%3a0%3a0%3a0%3a5&latest) even OpenSSL 1.0.1l can connect. Maybe try to connect to it through the provided command line utilities and if this works the issue is somewhere in FPC. %)
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: trev on August 13, 2020, 01:11:08 am
Quote from: wittbo on August 12, 2020, 05:55:05 pm
@Trev: which units must the uses clause contain?

I've included the relevant details below.

Code: Pascal  [Select][+][-]
  1. unit Unit1;
  2.  
  3. {$mode objfpc}{$H+}
  4. {$modeswitch objectivec1}
  5. {$linkframework foundation}
  6.  
  7. interface
  8.  
  9. uses
  10. Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls,  fphttpclient, OpenSSLSockets, CocoaAll, CocoaUtils;
  11. [...]

Don't forget you need to edit .../packages/cocoaint/src/foundation/NSString.inc as noted in my earlier post and then recompile FPC 3.2.0. I use the following script to recompile:

Code: Text  [Select][+][-]
  1. #!/bin/sh
  2. make clean all FPC=/usr/local/lib/fpc/3.2.0/ppcx64 OS_TARGET=darwin CPU_TARGET=x86_64 OPT="-XR/Library/Developer/CommandLineTools//SDKs/MacOSX.sdk/"
  3. make install FPC=/usr/local/lib/fpc/3.2.0/ppcx64 OS_TARGET=darwin CPU_TARGET=x86_64
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: trev on August 13, 2020, 01:38:24 am
Quote from: Bi0T1N on August 12, 2020, 10:37:06 pm
According to the SSL Server Test (https://www.ssllabs.com/ssltest/analyze.html?d=api.1nce.com&s=2001%3a1608%3a6%3a13%3a0%3a0%3a0%3a5&latest) even OpenSSL 1.0.1l can connect.

wittbo's problematic website is www.ariva.de (not api.1nce.com). That website only accepts TLSv1.2 and TLSv1.3 protocols.

The SSL libraries on macOS 10.14.6 are not OpenSSL but LibreSSL. The default unversioned /usr/lib/libssl.dylib is linked to /usr/lib/libssl.35.dylib (LibreSSL v2.2.7). There's also /usr/lib/libssl.43.dylib (LibreSSL v2.5.5) and /usr/lib/libssl.44.dylib (LibreSSL v2.6.5).

My protocol tests indicate that FPC 3.2.0/3.3.1 fails to connect to websites that only accept TLSv1.2 or above whereas FPC 3.0.4 does connect to websites that only accept TLSv1.2 and above. I'm not sure how much clearer I can make it.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: wittbo on August 13, 2020, 11:54:54 am
@Trev:
Just tested your solution for macOS (s.a.), edited the NSString.inc file; when updating the compiler, there are the following error messages:
make: *** No rule to make target `clean'.  Stop.
make: *** No rule to make target `install'.  Stop.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: trev on August 13, 2020, 03:00:21 pm
You are doing this from the directory in which your FPC source is?
You will almost certainly need to use sudo as well.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: wittbo on August 13, 2020, 03:49:36 pm
Sorry, Trev, I'm not very experienced in such operations; in fact I did it the first time.

I did not change the path after having edited NSString.inc. That was it.
Being back to /usr/local/share/fpcsrc/3.2.0 everything was running perfectly.

Thank you very much for your help. Now I have a workaround for fpc 3.2.0 for MacOS, it's OK for me. Nevertheless I will open a bug report.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: trev on August 13, 2020, 04:12:26 pm
You're welcome!
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: Bi0T1N on August 13, 2020, 09:12:01 pm
Quote from: trev on August 13, 2020, 01:38:24 am
Quote from: Bi0T1N on August 12, 2020, 10:37:06 pm
According to the SSL Server Test (https://www.ssllabs.com/ssltest/analyze.html?d=api.1nce.com&s=2001%3a1608%3a6%3a13%3a0%3a0%3a0%3a5&latest) even OpenSSL 1.0.1l can connect.

wittbo's problematic website is www.ariva.de (not api.1nce.com). That website only accepts TLSv1.2 and TLSv1.3 protocols.
The first post uses the link I've used. However, both support TLSv1.2 and TLSv1.3 only.

Quote from: trev on August 13, 2020, 01:38:24 am
Quote from: Bi0T1N on August 12, 2020, 10:37:06 pm
According to the SSL Server Test (https://www.ssllabs.com/ssltest/analyze.html?d=api.1nce.com&s=2001%3a1608%3a6%3a13%3a0%3a0%3a0%3a5&latest) even OpenSSL 1.0.1l can connect.

The SSL libraries on macOS 10.14.6 are not OpenSSL but LibreSSL. The default unversioned /usr/lib/libssl.dylib is linked to /usr/lib/libssl.35.dylib (LibreSSL v2.2.7). There's also /usr/lib/libssl.43.dylib (LibreSSL v2.5.5) and /usr/lib/libssl.44.dylib (LibreSSL v2.6.5).

My protocol tests indicate that FPC 3.2.0/3.3.1 fails to connect to websites that only accept TLSv1.2 or above whereas FPC 3.0.4 does connect to websites that only accept TLSv1.2 and above. I'm not sure how much clearer I can make it.
According to this (https://github.com/graemeg/freepascal/blob/master/packages/openssl/src/openssl.pas#L5629) it's no longer allowed to load the unversioned one but I also don't know which one is loaded then as the DLLVersions array doesn't contain any of your numbers.
So it would be good to add the code I've posted above to see which version really gets loaded - maybe an ancient version which doesn't support TLS1.2 or higher.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: trev on August 14, 2020, 03:39:00 am
@BiOT1N

FPC 3.2.0 reports using OpenSSL 0.9.8zh 14 January 2016

FPC 3.0.4 reports using LibreSSL 2.2.7 (The default unversioned /usr/lib/libssl.dylib or /usr/lib/libssl.35.dylib  ).

I found /usr/lib/libssl.0.9.7.dylib and /usr/lib/libssl.0.9.8.dylib which I somehow inexplicably missed the first time,
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: trev on August 14, 2020, 03:54:52 am
Reverting the "fix" for Bug ID 36484 fixes the issue on FPC 3.2.0.

Updated Bug ID 36484 (https://bugs.freepascal.org/view.php?id=36484).
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: trev on August 14, 2020, 11:06:48 am
The better solution than reverting the original fix (tested on trunk):

Code: Pascal  [Select][+][-]
  1. --- openssl.pas (revision 45778)
  2. +++ openssl.pas (working copy)
  3. @@ -111,7 +111,7 @@
  4.    { ADD NEW ONES WHEN THEY APPEAR!
  5.      Always make .so/dylib first, then versions, in descending order!
  6.      Add "." .before the version, first is always just "" }
  7. -  DLLVersions: array[1..19] of string = ('', '.1.1', '.11', '.10', '.1.0.6', '.1.0.5', '.1.0.4', '.1.0.3',
  8. +  DLLVersions: array[1..26] of string = ('', '.48', '.47', '.46', '.45', '.44', '.43', '.35', '.1.1', '.11', '.10', '.1.0.6', '.1.0.5', '.1.0.4', '.1.0.3',
  9.                                          '.1.0.2', '.1.0.1','.1.0.0','.0.9.8',
  10.                                          '.0.9.7', '.0.9.6', '.0.9.5', '.0.9.4',
  11.                                          '.0.9.3', '.0.9.2', '.0.9.1');
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: trev on August 15, 2020, 10:36:14 am
Updated because (at least on macOS) the libssl and libcrypto version numbers are out of sync:

Code: Pascal  [Select][+][-]
  1. --- openssl.pas (revision 45778)
  2. +++ openssl.pas (working copy)
  3. @@ -111,7 +111,7 @@
  4.    { ADD NEW ONES WHEN THEY APPEAR!
  5.      Always make .so/dylib first, then versions, in descending order!
  6.      Add "." .before the version, first is always just "" }
  7. -  DLLVersions: array[1..19] of string = ('', '.1.1', '.11', '.10', '.1.0.6', '.1.0.5', '.1.0.4', '.1.0.3',
  8. +  DLLVersions: array[1..29] of string = ('', '.48', '.47', '.46', '.45', '.44', '.43', '.42', '.41',
  9.                                             '.38', '.35', '.1.1', '.11', '.10', '.1.0.6', '.1.0.5',
  10.                                             '.1.0.4', '.1.0.3', '.1.0.2', '.1.0.1', '.1.0.0', '.0.9.8',
  11.                                             '.0.9.7', '.0.9.6', '.0.9.5', '.0.9.4', '.0.9.3', '.0.9.2',
  12.                                             '.0.9.1');

>> EDIT <<

This is no longer a good idea in light of further information - please see the Wiki article OpenSSL, LibreSSL, Secure Transport, Network Framework (https://wiki.lazarus.freepascal.org/macOS_Programming_Tips#OpenSSL.2C_LibreSSL.2C_Secure_Transport.2C_Network_Framework) for why. 
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: wittbo on August 15, 2020, 11:59:02 am
Would it work, if I modify openssl.pas on my mac (laz 2.0.10/ fpc 3.2.0) as you described?
Or has something more to be done? Recompile the compiler, ...?
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: trev on August 15, 2020, 12:36:19 pm
You need to make the patch as described and then recompile FPC (like you did before). Should work fine with FPC 3.2.0.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: wittbo on August 16, 2020, 09:53:53 am
Perfect. It's working as desired. Laz 2.0.10, fpc 3.2.0  MacOS Mojave
Thanks for help.
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: trev on August 16, 2020, 09:58:28 am
Happy to have helped :)
Title: Re: Cannot connect to https://api.1nce.com using TFPHTTPClient
Post by: trev on March 06, 2021, 11:59:53 am
This is no longer a good idea in light of further information - please see the Wiki article OpenSSL, LibreSSL, Secure Transport, Network Framework (https://wiki.lazarus.freepascal.org/macOS_Programming_Tips#OpenSSL.2C_LibreSSL.2C_Secure_Transport.2C_Network_Framework) for why. 
TinyPortal © 2005-2018