Recent

Author Topic: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me  (Read 24984 times)

MISV

  • Hero Member
  • *****
  • Posts: 772
In my

/usr/lib directory I have

libcrypto.dylib, libcrypto.0.9.7.dylib, libcrypto.0.9.8.dylib, libcrypto.35.dylib, libcrypto.0.41.dylib
libssl.dylib, libssl.0.9.7.dylib, libssl.0.9.8.dylib, libssl.35.dylib, libssl.0.41.dylib

If I use right click - Ge info on libcrypto.dylib I see it points to libcrypto.35.dylib (and similar for libssl.dylib)

Online I have found out that is a LibreSSL and not OpenSSL

...

The case currently is I can not crawl SSL sites using Indy (not those I have tested anyway) ... I do have a fallback to native (but that follows redirects silently which I do not want - I want to keep track of redirects)

How do you guys use Indy on Mac with OpenSSL/LibreSSL?

I am thinking maybe pointing Indy explicitly to OpenSSL 0.9.8 dylibs (which I believe is actually newer/patched versions of OpenSSL but have kept their file names for not breaking compability)



Hansaplast

  • Hero Member
  • *****
  • Posts: 674
  • Tweaking4All.com
    • Tweaking4All
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #1 on: September 19, 2018, 04:38:28 pm »
I'm using Synapse on macOS 10.13.6 and 10.14 (latest public beta), and it works just fine with the default installed SSL libs.
Maybe switch to Synapse? (I have no experience with Indy)

MISV

  • Hero Member
  • *****
  • Posts: 772
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #2 on: September 19, 2018, 05:44:26 pm »
What openssl/libressl libs do you have in your /usr/lib/ folder?

MISV

  • Hero Member
  • *****
  • Posts: 772
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #3 on: September 19, 2018, 07:37:54 pm »
I also sent you a PM - but  I do not see it in my sent PMs list - but I have sen you an URL - if you can scan that using Synapse


Maybe Synapse added support for LibreSSL? But knowing what .dylib files you have in your usr/lib/ would help clarify that :)

(I rather use built-in openssl/libressl instead of shipping  those files myself)

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1311
    • Lebeau Software
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #4 on: September 19, 2018, 07:47:48 pm »
The case currently is I can not crawl SSL sites using Indy (not those I have tested anyway) ...

What EXACTLY happens when you try?  Are you getting an exception that Indy can't load the dylibs?  Or are you simply getting an SSL/TLS error?  Please be more specific.

I am thinking maybe pointing Indy explicitly to OpenSSL 0.9.8 dylibs (which I believe is actually newer/patched versions of OpenSSL but have kept their file names for not breaking compability)

Indy doesn't provide an option to specify the exact filenames, only the folder. It then runs through a list of pre-defined filenames until it finds a match. Changing that logic would require altering Indy's source code and recompiling.

If you want to use a specific version of OpenSSL (1.0.2 is the latest supported by Indy), you should put those dylibs in a folder of your choosing, then specify that folder path to Indy via the IdOpenSSLSetLibPath() function at app startup. Apple has deprecated support for OpenSSL on OSX, so you have to deploy your own OpenSSL dylibs with your app if you want to use newer versions of OpenSSL than what Apple still provides for backwards compatibility.

Or, maybe try using a Homebrew version of OpenSSL: Using the OpenSSL library with macOS Sierra

In the meantime, I've opened a feature request to add LibreSSL support to Indy: https://github.com/IndySockets/Indy/issues/231
« Last Edit: September 19, 2018, 08:10:18 pm by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

MISV

  • Hero Member
  • *****
  • Posts: 772
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #5 on: September 19, 2018, 09:30:42 pm »
Here are the following exceptions I get with HEAD/GET requests to multiple https sites

SSLOptions.SSLVersions := [sslvSSLv23];
SSLOptions.Method := sslvSSLv23;
= EIdOSSLUnderlyingCryptoError

SSLOptions.SSLVersions := [sslvTLSv1_2];
SSLOptions.Method := sslvTLSv1_2;
= EIdOSSLUnderlyingCryptoError

SSLOptions.SSLVersions := [sslvTLSv1_1];
SSLOptions.Method := sslvTLSv1_1;
= EIdOSSLUnderlyingCryptoError

SSLOptions.SSLVersions := [sslvTLSv1];
SSLOptions.Method := sslvTLSv1;
= EIdOSSLUnderlyingCryptoError

Since my own website exposes the problem, I can email you example URL to test against (not sure this forum PM works, but I can PM as well)


...



I suspect it would work if I shipped OpenSSL myself. I was just hoping to be able to use system provided libraries


...


Possible solution... Searching for 0.9.8 files in /usr/lib/ and copy them out to another directory. Then point Indy to that. Not quite sure that would be completely legal though. But at least a possibility to consider.

From searching the net it seems new versions of Mac OS do not include OpenSSL anymore bu only LibreSSL
« Last Edit: September 20, 2018, 11:45:49 am by MISV »

Hansaplast

  • Hero Member
  • *****
  • Posts: 674
  • Tweaking4All.com
    • Tweaking4All
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #6 on: September 20, 2018, 12:18:42 pm »
I also sent you a PM - but  I do not see it in my sent PMs list - but I have sen you an URL - if you can scan that using Synapse


Maybe Synapse added support for LibreSSL? But knowing what .dylib files you have in your usr/lib/ would help clarify that :)

(I rather use built-in openssl/libressl instead of shipping  those files myself)


Stupid forum filesize limitation made me type this again  >:(

I have a test project for you (about 600 Kb - which includes the Synapse files), so if you PM me your email address, I can email it to you.


As for the lib files - I could not find a single libressl file, just the regular openssl files.

macOS 10.14 (latest public beta on a clean VM):


libssl.0.9.7.dylib
libssl.0.9.8.dylib
libssl.35.dylib
libssl.43.dylib
libssl.44.dylib
libssl.dylib -> libssl.35.dylib


libcrypto.0.9.7.dylib
libcrypto.0.9.8.dylib
libcrypto.35.dylib
libcrypto.41.dylib
libcrypto.42.dylib
libcrypto.dylib -> libcrypto.35.dylib


macOS 10.13.6:


libssl.0.9.7.dylib
libssl.0.9.8.dylib
libssl.35.dylib
libssl.43.dylib
libssl.dylib -> libssl.35.dylib


libcrypto.0.9.7.dylib
libcrypto.0.9.8.dylib
libcrypto.35.dylib
libcrypto.41.dylib
libcrypto.dylib -> libcrypto.35.dylib

rvk

  • Hero Member
  • *****
  • Posts: 6056
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #7 on: September 20, 2018, 12:35:14 pm »
Synapse just uses whatever libssl.dylib is pointing to.
So libssl.dylib should point to something that is OpenSSL "compatible" (which I assume LibreSSL/libssl.35.dylib is).

Code: Pascal  [Select][+][-]
  1.   {$IFNDEF MSWINDOWS}
  2.     {$IFDEF DARWIN}
  3.     DLLSSLName: string = 'libssl.dylib';
  4.     DLLUtilName: string = 'libcrypto.dylib';
  5.     {$ELSE}
  6.      {$IFDEF OS2}
  7.       {$IFDEF OS2GCC}
  8.     DLLSSLName: string = 'kssl.dll';

Don't point to a specific version in your code, otherwise you'll end up changing it regularly.

B.T.W. What version is that  libssl.35.dylib? I see that LibreSSL only goes up to 2.8.0.
Maybe it's a version which doesn't support the new TLS.


Edit: I see LibreSSL 2.5.5 has libssl-43.dll for Windows. So I guess libssl-35 is quite old.
libssl-35 seems to be from the 2.2.x branche which is from 2015, maybe 2016.
These might really have problems with the new TLS.
Try to find a newer LibreSSL version for your OS and install that.
« Last Edit: September 20, 2018, 12:41:32 pm by rvk »

MISV

  • Hero Member
  • *****
  • Posts: 772
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #8 on: September 20, 2018, 12:48:54 pm »
So looks like Indy and Synapse does the same. However, I don't think LibreSSL/OpenSSL are fully compaible.

...

I am trying to use what Mac OS already comes with. I would rather not require my users to install LibreSSL

My mac libssl.dylib points to libssl.35.dylib file

(however, not sure you can trust the version numbers, on my Mac 0.9.8 openssl was seemingly patched by Apple many times meaning the actual version is somewhat newer - long time ago I researched those details so I may be mistaken)

Hansaplast

  • Hero Member
  • *****
  • Posts: 674
  • Tweaking4All.com
    • Tweaking4All
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #9 on: September 20, 2018, 12:58:03 pm »
Just did a test, seems libssl.35.dylib is indeed LibreSSL 2.2.7 (Synapse provides this info).
Obviously libssl.43 and .44 are newer versions, but macOS by default links to libssl.35.

rvk

  • Hero Member
  • *****
  • Posts: 6056
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #10 on: September 20, 2018, 01:23:38 pm »
I'm not sure if the newer/newest versions of LibreSSL are 1:1 compatible with OpenSSL.

But I do know that even OpenSSL from 2015 has problems connecting to some newer websites where only certain TLS versions are allowed.

After installing newer LibreSSL it should point to those versions and otherwise you could adjust the symlink yourself.

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1311
    • Lebeau Software
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #11 on: September 20, 2018, 06:44:28 pm »
Here are the following exceptions I get with HEAD/GET requests to multiple https sites

What are the DETAILS provided by those exceptions?  What do their ErrorCode and Message properties say exactly?

The fact that you are even getting this far means Indy was able to find and load libssl and libcrypto libraries into memory and locate enough of the "OpenSSL" functions it is interested in using.  Which means LibreSSL is at least API-compatible to OpenSSL as far as Indy cares about (which makes sense, since LibreSSL is forked from OpenSSL 1.0.1g).  Though that doesn't necessarily guarantee that LibreSSL exposes everything that Indy needs to communicate with modern websites.

Synapse just uses whatever libssl.dylib is pointing to.

So does Indy, by default.  And when it can't find that, it starts looking for version-specific files next (see the code in Indy's LoadSSLCryptoLibrary() function in the IdSSLOpenSSLHeaders unit).

Just did a test, seems libssl.35.dylib is indeed LibreSSL 2.2.7 (Synapse provides this info).

What does Indy's OpenSSLVersion() function in the IdSSLOpenSSL unit report?
« Last Edit: September 20, 2018, 06:50:15 pm by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

rvk

  • Hero Member
  • *****
  • Posts: 6056
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #12 on: September 20, 2018, 07:17:11 pm »
OP also mentions "Since my own website exposes the problem...".

If it does work on other sites using SSL with LibreSSL, my guess is that 2.2.7 is too old for the TLS used at that site. Only way to fix that is updating LibreSSL.

MISV

  • Hero Member
  • *****
  • Posts: 772
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #13 on: September 20, 2018, 11:59:15 pm »
OpenSSLVersion reports = OpenSSL 0.9.8zh 14 jan 2016
(so seems OpenSSL lib was loaded even though the symlink points to LibreSSL? Under all circumstances I am unsure if OpenSSL is still provided in clean macOS installs)

SSLOptions.SSLVersions := [sslvSSLv23];
SSLOptions.Method := sslvSSLv23;
= EIdOSSLUnderlyingCryptoError
- ErrorCode: 336032814
- Message: Error connecting with SSL. error 1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 protocol alert version

SSLOptions.SSLVersions := [sslvTLSv1_2];
SSLOptions.Method := sslvTLSv1_2;
= EIdOSSLUnderlyingCryptoError
- ErrorCode: 336151598
- Message: Error connecting with SSL. error 1409442E:SSL routines:SSL3_READ_BYTES:tlsv1 protocol alert version

SSLOptions.SSLVersions := [sslvTLSv1_1];
SSLOptions.Method := sslvTLSv1_1;
= EIdOSSLUnderlyingCryptoError
- ErrorCode: 336151598
- Message: Error connecting with SSL. error 1409442E:SSL routines:SSL3_READ_BYTES:tlsv1 protocol alert version


...

I guess next question is why the LibreSSL is not used when I can see it is pointed to. But if LibreSSL is not fully supported then it is of course much better loading OpenSSL! Anyhow the domain I am testing on is not using CloudFlare SSL (I found out they would fail with 0.9.8zh long time ago), but I guess a similar SSL solution then.

...

HansaPlast has told me he can load the same https website with Synapse, so I will be testing that
« Last Edit: September 21, 2018, 09:50:20 am by MISV »

Hansaplast

  • Hero Member
  • *****
  • Posts: 674
  • Tweaking4All.com
    • Tweaking4All
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #14 on: September 21, 2018, 10:23:20 am »

To get you started with Synapse; I just toss the entire Synapse library in my project directory (usually in a separate directory, but you'd have to add that directory to the paths in your project options).


Code: Pascal  [Select][+][-]
  1. uses ... ssl_openssl, httpsend, blcksock, synautil ... ;
  2.  
  3. ...
  4.  
  5.  
  6. procedure TForm1.Button1Click(Sender: TObject);
  7. var
  8.   HTTP: THTTPSend;
  9. begin
  10.   HTTP:=THTTPSend.Create;
  11.   HTTP.UserAgent:='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36';
  12.   HTTP.Headers.Clear;
  13.  
  14.  
  15.   try
  16.     HTTP.HTTPMethod('GET', 'https://www.example.com');
  17.  
  18.  
  19.     if (HTTP.ResultCode >= 100) and (HTTP.ResultCode<=299) then
  20.       Memo1.Lines.LoadFromStream(HTTP.Document);
  21.   except
  22.     showMessage('Something went wrong ...');
  23.   end;
  24.  
  25.  
  26.   showMessage('HTTP.Sock.LastError :'+IntToStr(HTTP.Sock.LastError)+' ; '+HTTP.Sock.LastErrorDesc+lineEnding+
  27.               'HTTP.Sock.SSL.LastError :'+IntToStr(HTTP.Sock.SSL.LastError)+' ; '+HTTP.Sock.SSL.LastErrorDesc+lineending+
  28.               'HTTP.ResultCode :'+IntToStr(HTTP.ResultCode)+LineEnding+
  29.               'Libname :'+HTTP.Sock.SSL.LibName+LineEnding+
  30.               'Libversion :'+HTTP.Sock.SSL.LibVersion);
  31.   HTTP.Free;
  32. end;
  33.  
  34.  
  35. ...
  36.  


I noticed with this site that without setting a useragent, it will return only the first Kb ... with UserAgent set, it will return everything.
I just added the ShowMessage at the end so I can see more detail about what is going on (or not).


Hope this helps.

« Last Edit: September 23, 2018, 05:07:50 pm by Hansaplast »

 

TinyPortal © 2005-2018