Lazarus

Programming => Operating Systems => macOS / Mac OS X => Topic started by: Josh on February 08, 2020, 01:07:52 pm

Title: Synapse and Catalina SSL causing crash
Post by: Josh on February 08, 2020, 01:07:52 pm
Hi

Hope someone can help, and its not too complicated.

I have a app that gets incrypted data from company server, the app is working fine on everything up to Catalina, but when run on catalina it crashes out on start up. error below. The app is a company app and as such is not through the App Store.

I am using synapse fro the HTTP and HTTPS routines, as i also have windows clients of the app.

Is their a way around the New apple security system, or maybe does anyone know of an alternative HTTP HTTPS unit for osx (that is native) to by pass synapse for osx deployment, if so how can i configure a build option that does not add the lazsynapse requirement.

Quote
Application Specific Information:
/usr/lib/libcrypto.dylib
abort() called
Invalid dylib load. Clients should not load the unversioned libcrypto dylib as it does not have a stable ABI.

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib           0x00007fff65a3d7fa __pthread_kill + 10
1   libsystem_pthread.dylib          0x00007fff65affbc1 pthread_kill + 432
2   libsystem_c.dylib                0x00007fff659c4a1c abort + 120
3   libcrypto.dylib                  0x00007fff6333b804 __report_load + 352
4   dyld                             0x000000011602e15d ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 539
5   dyld                             0x000000011602e582 ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
6   dyld                             0x0000000116028dc7 ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 493
7   dyld                             0x0000000116026e58 ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 188
8   dyld                             0x0000000116026ef8 ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 82
9   dyld                             0x0000000116018f87 dyld::runInitializers(ImageLoader*) + 82
10  dyld                             0x0000000116022ad7 dlopen_internal + 609
11  libdyld.dylib                    0x00007fff658e1a7f dlopen + 171
12  com.company.editor               0x000000010777935a 0x10764f000 + 1221466
13  com.company.editor               0x000000010765fc1f SYSTEM_$$_LOADLIBRARY$RAWBYTESTRING$$INT64 + 15
14  com.company.editor               0x0000000107ac2246 SYNAFPC_$$_LOADLIBRARY$PCHAR$$INT64 + 86 (synafpc.pas:109)
15  com.company.editor               0x0000000107acf849 0x10764f000 + 4720713
16  com.company.editor               0x0000000107acf919 SSL_OPENSSL_LIB_$$_INITSSLINTERFACE$$BOOLEAN + 175
17  com.company.editor               0x0000000107ac8067 INIT$_$SSL_OPENSSL + 10
18  com.company.editor               0x000000010765cdcc FPC_INITIALIZEUNITS + 60
19  com.company.editor               0x000000010765058d PASCALMAIN + 10

Regards

Josh
Title: Re: Synapse and Catalina SSL causing crash
Post by: Hansaplast on February 08, 2020, 01:17:06 pm
Catalina doesn't seem to like OpenSSL anymore.
Instead (depending if it fits your needs) use the MacOS API.
See also my other post in the forum: here (https://forum.lazarus.freepascal.org/index.php/topic,43041.msg340761.html#msg340761).

Short version:

Under more recent macOS versions (at least as of Mojave), fphttpclient will not work.
Apple does not seem to like the OpenSSL library (default macOS setup!):


As an alternative for macOS users, use the unit "ns_url_request" by Phil Hess (file can be found here (https://macpgmr.github.io/ObjP/nsunits-src.zip) - Phil's Mac related page (https://macpgmr.github.io/)).
This does not require any extra libraries (beyond what comes with macOS).


A quick (and sloppy) function pulling in HTTPS content as a string:



Code: Pascal  [Select][+][-]
  1. uses ... ns_url_request ... // Note: "ns_url_request" uses also Phil's "NSHelpers" unit.
  2.  
  3. ...
  4.  
  5. function TForm1.GetURLContent(aURL:string):string;
  6. var
  7.   HTTP: TNSHTTPSendAndReceive;
  8. begin
  9.   HTTP := TNSHTTPSendAndReceive.Create;
  10.   HTTP.Method   := 'GET';
  11.   HTTP.Address  := aURL;
  12.   HTTP.SendAndReceive(Result);
  13.   HTTP.Free;
  14. end;  

Title: Re: Synapse and Catalina SSL causing crash
Post by: Jonas Maebe on February 08, 2020, 02:51:43 pm
Catalina doesn't seem to like OpenSSL anymore.

Even if that is true, that has absolutely nothing to do with the error message.

The issue is that OpenSSL does not provide ABI compatibility between different versions. That means that if you load the unversioned libcrypto.dylib, you may get a version of the library that uses a different ABI than the one you expect. That can lead to bugs like buffer overflows, reading uninitialised memory, or other undefined behaviour. These are sources of security holes and hence things you definitely don't want to happen when using a crypto library.

The solution is to explicitly load either /usr/lib/libcrypto.0.9.7.dylib or /usr/lib/libcrypto.0.9.8.dylib, depending on which version of OpenSSL your (Synapse) interface uses. At first sight, this needs to be fixed in a unit called ssl_openssl_lib.
Title: Re: Synapse and Catalina SSL causing crash
Post by: Hansaplast on February 08, 2020, 03:33:22 pm
You're right, better explained than I did ...


My only concern with that, is that on another Mac, now or in the future, that particular library version may or may not be installed on their system?
So one may have to distribute that exact version of OpenSSL (and all that comes with it) with your application to avoid that?
Where as the other approach would not only rely on Apple's API.


(please correct me if I'm wrong)
Title: Re: Synapse and Catalina SSL causing crash
Post by: Jonas Maebe on February 08, 2020, 03:42:37 pm
You're right, better explained than I did ...


My only concern with that, is that on another Mac, now or in the future, that particular library version may or may not be installed on their system?
So one may have to distribute that exact version of OpenSSL (and all that comes with it) with your application to avoid that?
Where as the other approach would not only rely on Apple's API.

Apple also merrily deprecates and removes its own APIs from time to time, so that's not really safer in general.
Title: Re: Synapse and Catalina SSL causing crash
Post by: trev on February 08, 2020, 11:33:31 pm
This is what I use - it works on macos Mojave and Catalina, FreeBSD, Ubuntu Linux and Windows.

Code: Pascal  [Select][+][-]
  1. {$IFDEF UNIX}
  2. function GetMicrochipPage(const URL: string): string;
  3. var
  4.   Client: TFPHttpClient;
  5.   {$IFDEF DARWIN}
  6.   MsgStr: String;
  7.   {$ENDIF}
  8. begin
  9.   Client := TFPHttpClient.Create(nil);
  10.  
  11.   Try
  12.     Client.AllowRedirect := true;
  13.     Client.AddHeader('User-Agent', 'Mozilla/5.0(compatible; fpweb)');
  14.     Result := Client.Get(URL);
  15.   except
  16.       on E: Exception do
  17.            {$IFDEF DARWIN}
  18.            begin
  19.                MsgStr := 'Retrieval of: ' + URL + LineEnding
  20.                        + 'Failed with error: ' + E.Message + LineEnding
  21.                        + 'HTTP code: ' + IntToSTr(Client.ResponseStatusCode);
  22.  
  23.                ShowAlertSheet(Form1_Main.Handle, 'Alert', MsgStr);
  24.            end;
  25.            {$ENDIF}
  26.            {$IFNDEF DARWIN}
  27.            ShowMessage('Retrieval of: ' + URL + LineEnding
  28.                        + 'Failed with error: ' + E.Message + LineEnding
  29.                        + 'HTTP code: ' + IntToSTr(Client.ResponseStatusCode));
  30.            {$ENDIF}
  31.   end;
  32. end;
  33. {$ENDIF}
  34.  
  35. {$IFDEF WINDOWS}
  36. // Need to use Windows WinInet to avoid issue with HTTPS
  37. // needing two OpenSSL DLLs to be provided with application
  38. // if using TFPHttpClient.
  39. // The WinINet API also gets any connection and proxy settings
  40. // set by Internet Explorer. Blessing or curse?
  41.  
  42. function GetMicrochipPage(const Url: string): string;
  43. var
  44.   NetHandle: HINTERNET;
  45.   UrlHandle: HINTERNET;
  46.   Buffer: array[0..1023] of Byte;
  47.   BytesRead: dWord;
  48.   StrBuffer: UTF8String;
  49. begin
  50.   Result := '';
  51.   NetHandle := InternetOpen('Mozilla/5.0(compatible; WinInet)', INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0);
  52.  
  53.   // NetHandle valid?
  54.   if Assigned(NetHandle) then
  55.     Try
  56.       UrlHandle := InternetOpenUrl(NetHandle, PChar(Url), nil, 0, INTERNET_FLAG_RELOAD, 0);
  57.  
  58.       // UrlHandle valid?
  59.       if Assigned(UrlHandle) then
  60.         Try
  61.           repeat
  62.             InternetReadFile(UrlHandle, @Buffer, SizeOf(Buffer), BytesRead);
  63.             SetString(StrBuffer, PAnsiChar(@Buffer[0]), BytesRead);
  64.             Result := Result + StrBuffer;
  65.           until BytesRead = 0;
  66.         Finally
  67.           InternetCloseHandle(UrlHandle);
  68.         end
  69.       // o/w UrlHandle invalid
  70.       else
  71.         ShowMessage('Cannot open URL: ' + Url);
  72.     Finally
  73.       InternetCloseHandle(NetHandle);
  74.     end
  75.   // NetHandle invalid
  76.   else
  77.     raise Exception.Create('Unable to initialize WinInet');
  78. end;
  79. {$ENDIF}
  80.  

Note: I'm using FPC 3.3.1 (trunk).
Title: Re: Synapse and Catalina SSL causing crash
Post by: MISV on February 18, 2020, 12:03:09 pm
Indy and the openssl/libressl libs included in macos generally works for me.
Title: Re: Synapse and Catalina SSL causing crash
Post by: MISV on February 18, 2020, 12:04:32 pm
Quote
The solution is to explicitly load either /usr/lib/libcrypto.0.9.7.dylib or /usr/lib/libcrypto.0.9.8.dylib, depending on which version of OpenSSL your (Synapse) interface uses. At first sight, this needs to be fixed in a unit called ssl_openssl_lib.

These are too old with regards to TLS support for many websites in my experience. But macos has newer openssl/libressl which I believe works OK (at least with Indy)
Title: Re: Synapse and Catalina SSL causing crash
Post by: Thaddy on February 18, 2020, 01:11:20 pm
It also works on OSX, any version as long as tls 1.2 is *specifically* specified.
Modern browsers ignore anything less.
You can also try fcl-web. That is more resilient than synapse. (and imho now much better)
Title: Re: Synapse and Catalina SSL causing crash
Post by: esvignolo on February 18, 2020, 04:15:12 pm
Indy and the openssl/libressl libs included in macos generally works for me.

Hi MISV do you have a link to donwload the openssl binaries?

Thanks!
Title: Re: Synapse and Catalina SSL causing crash
Post by: MISV on February 18, 2020, 06:14:37 pm
You probably already have LibreSSL installed on your Mac OS system (and those versions overall seem compatible with regards to HTTPS usage. Newer versions of LibreSSL may differ)

https://github.com/IndySockets/Indy/issues/231#issuecomment-566542370 (https://github.com/IndySockets/Indy/issues/231#issuecomment-566542370)

(0.9.7 / 0.9.8 will not work for many websites.) 
Title: Re: Synapse and Catalina SSL causing crash
Post by: Thaddy on February 18, 2020, 06:20:11 pm
The latter doesn't mean too much for OSX, as long as you specify tls 1.2. (although I just could connect with 1.1, which is also legacy, on my apple iMac)
Old code with e.g. ssl2 or ssl3 or tls 1.0 simply will not work. There is a reason for that......
If your code tries to use older encryption, you could have known that it will be refused and your code obviously does not handle that.
That is your fault, not the library. We have try except for that. And NEVER use deprecated protocols anywhere when it concerns encryption.
Title: Re: Synapse and Catalina SSL causing crash
Post by: esvignolo on February 18, 2020, 10:22:50 pm
You probably already have LibreSSL installed on your Mac OS system (and those versions overall seem compatible with regards to HTTPS usage. Newer versions of LibreSSL may differ)

https://github.com/IndySockets/Indy/issues/231#issuecomment-566542370 (https://github.com/IndySockets/Indy/issues/231#issuecomment-566542370)

The problem is de fphttp is not working with https in trunk. I think, maybe if i put de dylib in the binary directory maybe work.
Title: Re: Synapse and Catalina SSL causing crash
Post by: Thaddy on February 18, 2020, 10:34:45 pm
I only test with FPC trunk or 3.20, do you mean Lazarus trunk 2.10?
FPC trunk works (although there are some general issues as is known)
Title: Re: Synapse and Catalina SSL causing crash
Post by: MISV on February 19, 2020, 10:16:24 am
You probably already have LibreSSL installed on your Mac OS system (and those versions overall seem compatible with regards to HTTPS usage. Newer versions of LibreSSL may differ)

https://github.com/IndySockets/Indy/issues/231#issuecomment-566542370 (https://github.com/IndySockets/Indy/issues/231#issuecomment-566542370)

The problem is de fphttp is not working with https in trunk. I think, maybe if i put de dylib in the binary directory maybe work.

Just dug up some comments I made in my source code concerning Mac LibreSSL fork of OpenSSL which appears compatible:

LibreSSL 2.2.7 - in /usr/lib 0.35
LibreSSL 2.8.3 - in /usr/lib 0.44

seems to work OK (but no guarantees)

On Windows you can find .dll files at
https://indy.fulgan.com/SSL/ (https://indy.fulgan.com/SSL/)

Title: Re: Synapse and Catalina SSL causing crash
Post by: Thaddy on February 19, 2020, 10:32:03 am
While trying to put together a working example, my iMac mini's disk has just died. It is pushing up daisies, gone to meet its maker, it is an ex-mini.
Will try to get a new one (second hand, any offers?)
It was only seven years old.... :'( :'( It had already a long history of illness...
Title: Re: Synapse and Catalina SSL causing crash
Post by: trev on February 19, 2020, 11:26:22 am
While trying to put together a working example, my iMac mini's disk has just died. It is pushing up daisies, gone to meet its maker, it is an ex-mini.
Will try to get a new one (second hand, any offers?)
It was only seven years old.... :'( :'( It had already a long history of illness...

I've replaced disk drives in Mac minis from 2009, 2010, 2011 and 2012 - it's relatively easy, and even easier once you can dispense with the putty knife models. No need to buy another mini. Plenty of DIY videos online.

My 2009 Mac mini finally died after 11 years of 24x7 service earlier this year - spontaneous reboots several times a day. Not a bad innings though :)
Title: Re: Synapse and Catalina SSL causing crash
Post by: VTwin on February 20, 2020, 06:36:55 pm
This is what I use - it works on macos Mojave and Catalina, FreeBSD, Ubuntu Linux and Windows.

Many thanks. I have been using Synapse, which seemed to work fine, but was unaware of the Catalina issue, and the potential Windows issue. This gets rid of the Synapse dependence as well.

Your code is working fine for me on Linux, Mac, and Windows. I do not currently have Catalina installed.
Title: Re: Synapse and Catalina SSL causing crash
Post by: trev on February 20, 2020, 11:24:16 pm
Good to hear VTwin.
Title: Re: Synapse and Catalina SSL causing crash
Post by: Arvur on January 09, 2021, 11:56:13 am
For those who need Synapse to work with LibreSSL on Catalina and BigSur. Call this routine from initialization section of ssl_openssl_lib:
Quote
{$IFDEF DARWIN}
procedure CheckOpenSSLFixNeeded;
const
  fixDarwinRelease = 19; // Catalina
var
  KernelInfo: UtsName;
begin
  if (fpUName(KernelInfo) = 0) then
    if (StrToIntDef(Copy(KernelInfo.Release, 1, 2), 0) >= fixDarwinRelease) then begin
      DLLSSLName  := '/usr/lib/libssl.46.dylib';
      DLLUtilName := '/usr/lib/libcrypto.44.dylib';
    end; 
end;
{$ENDIF DARWIN}
Use IFDEF to call this on Darwin only.

P.S. Catalina blocked using libssl.dylib and libcrypto.dylib wrappers and BigSur stopped us from listing dylib's in /usr/lib/ to find last version. So we just stick on existing LibreSSL version.
Title: Re: Synapse and Catalina SSL causing crash
Post by: herux on March 13, 2021, 12:09:34 pm
thank @Arvur it work on bigSur
Title: Re: Synapse and Catalina SSL causing crash
Post by: Igor Kokarev on March 13, 2021, 12:35:49 pm
I'm afraid to use Synaps/libssl anymore. It may become broken with any future macOS version again.

Moved my code to Cocoa network classes.
Title: Re: Synapse and Catalina SSL causing crash
Post by: trev on March 13, 2021, 11:36:58 pm
Using the macOS third party SSL libraries is no longer, if it ever was, a good idea - please see the Wiki article OpenSSL, LibreSSL, Secure Transport, Network Framework (https://wiki.lazarus.freepascal.org/macOS_Programming_Tips#OpenSSL.2C_LibreSSL.2C_Secure_Transport.2C_Network_Framework) for why.
Title: Re: Synapse and Catalina SSL causing crash
Post by: superc on March 15, 2022, 10:12:54 am
For those who need Synapse to work with LibreSSL on Catalina and BigSur. Call this routine from initialization section of ssl_openssl_lib:
Quote
{$IFDEF DARWIN}
procedure CheckOpenSSLFixNeeded;
const
  fixDarwinRelease = 19; // Catalina
var
  KernelInfo: UtsName;
begin
  if (fpUName(KernelInfo) = 0) then
    if (StrToIntDef(Copy(KernelInfo.Release, 1, 2), 0) >= fixDarwinRelease) then begin
      DLLSSLName  := '/usr/lib/libssl.46.dylib';
      DLLUtilName := '/usr/lib/libcrypto.44.dylib';
    end; 
end;
{$ENDIF DARWIN}
Use IFDEF to call this on Darwin only.

P.S. Catalina blocked using libssl.dylib and libcrypto.dylib wrappers and BigSur stopped us from listing dylib's in /usr/lib/ to find last version. So we just stick on existing LibreSSL version.

hello, I've the same problem, but i don't understand where insert code of 'CheckOpenSSLFixNeeded', where is initialization section of ssl_openssl_lib?

Thanks in advance.
Title: Re: Synapse and Catalina SSL causing crash
Post by: trev on March 16, 2022, 09:01:09 am
In the Synapse unit ssl_openssl_lib.pas

replace:

Code: Pascal  [Select][+][-]
  1. var
  2.   {$IFNDEF MSWINDOWS}
  3.     {$IFDEF DARWIN}
  4.     DLLSSLName: string = 'libssl.dylib';
  5.     DLLUtilName: string = 'libcrypto.dylib';
  6.     {$ELSE}

with:

Code: Pascal  [Select][+][-]
  1. var
  2.   {$IFNDEF MSWINDOWS}
  3.     {$IFDEF DARWIN}
  4.     DLLSSLName: string = 'libssl.48.dylib';      // Monterey 48 - Big Sur/Catalina 46
  5.     DLLUtilName: string = 'libcrypto.46.dylib';  // Monterey 46 - Big Sur/Catalina 44
  6.     {$ELSE}
  7.      {$IFDEF OS2}

Unfortunately those LibreSSL library versions are years out of date (and LibreSSL just had a new denial of service - infinite loop - CVE yesterday).

The best option is to use your own up-to-date copy of the OpenSSL libraries in your Application Bundle's Resources/Frameworks folder which is what Apple recommends.

Also note that LibreSSL libraries are not 100% compatible with OpenSSL.
Title: Re: Synapse and Catalina SSL causing crash
Post by: ArminLinder on March 23, 2022, 08:13:19 pm
Sorry if I stomp in here, I just by coincidence ran into this problem when trying an older program of mine on BigSur. Another thing to mention is that the piece of software is for internal company use only, so the number of installations is quite limited (1), and I have full control of the OS version in use.

I resolved the issue by installing OpenSSL via Macports (which, btw, already gave me more deprecated software like gdb(ggdb), svn and now openssl). I listed the installed files and found that it puts copies of libssl.dylib and libcrypto.dylib into the /opt/local/lib directory. I have afterwards patched ssl_openssl_lib from the synapse package:

Code: Pascal  [Select][+][-]
  1.     {$IFDEF DARWIN}
  2.     DLLSSLName: string = '/opt/local/lib/libssl.dylib';
  3.     DLLUtilName: string = '/opt/local/lib/libcrypto.dylib';
  4.     {$ELSE}
  5.  

and my program compiles und runs like before. What I really like is the absence of any version number in the file name. What I see does, however, not fit with what you were writing about. Maybe because there are different SSL distributions (OpenSSL vs LibreSSL)?

Anyway, I what do you think about this approach? Can I tick "problem solved" on my todo list, or did I just set vectors for the next desaster?

Thnx, Armin.
Title: Re: Synapse and Catalina SSL causing crash
Post by: trev on March 24, 2022, 12:06:31 am
That approach would normally result in macOS throwing the error:

Code: [Select]
Invalid dylib load. Clients should not load the unversioned libcrypto dylib as it does not have a stable ABI.
I'm not sure why it doesn't in your case - that is very odd! The Forum and Net are littered with that issue. FPC itself was changed to prevent this issue - bug reports https://gitlab.com/freepascal.org/fpc/source/-/issues/36484 and https://gitlab.com/freepascal.org/fpc/source/-/issues/37977 but that does not help Synapse and hardcoding the unversioned libraries.

Anyway, for the reason stated in the usual macOS crash message, it's not a good idea.

OpenSSL 1.1.1 is on life support until 11th September 2023 (at which point all support ceases, so no bug fixes for security problems) as OpenSSL moves to version 3 (now at 3.02) which has even more significant ABI changes. There is also the issue that the LibreSSL libraries with the same names are not 100% compatible.
TinyPortal © 2005-2018