Lazarus

Programming => Operating Systems => macOS / Mac OS X => Topic started by: Grahame Grieve on November 23, 2021, 10:30:27 am

Title: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on November 23, 2021, 10:30:27 am

I am trying to link openSSL directly into my application to see if I can figure out how to distribute my application as a hardened run time. (Sure, I can put the two dylibs into my .app package and sign them, and make the whole thing a hardened run time, but libssl has a non-versioned dependency on libcrypto and OSX won't allow that in a hardened run time. I can set up a pipeline to do a custom compile of openSSL with a modified link dependency, according to the openSSL mailing list, but someone there suggested that I just link the openSSL code directly into the binary. Sounds like a good idea to me if it works, and $Linklib looks like it should do the job.

Well... not so easy. I link the .a into my code on a Mac M1 with the instruction:

{$IFDEF FPC}
  {$DEFINE STATICLOAD_OPENSSL}
  {$LINKLIB libcrypto.a}
  {$LINKLIB libssl.a}
{$ENDIF}
           
(The STATICLOAD_OPENSSL is a define in the openSSL 1.1x branch of Indy)

When I compile, I get this error:

Error: linker:   "___chkstk_darwin", referenced from:
Debug:       _BN_mod_exp_mont_consttime in libcrypto.a(bn_exp.o)
Debug:       _do_ssl3_write in libssl.a(rec_layer_s3.o)
Debug:       _tls_parse_ctos_cookie in libssl.a(extensions_srvr.o)
Debug:       _curve448_base_double_scalarmul_non_secret in libcrypto.a(curve448.o)
Debug:       _ERR_print_errors_cb in libcrypto.a(err_prn.o)
Error: ld: symbol(s) not found for architecture arm64

All the online help I can find suggests that this is OS version problem in the linker. And indeed, prior to that, I get 100s of errors like this:

Error: ld: warning: object file (/Users/grahame/work/tools/fpc/units/aarch64-darwin/rtl/sysinit.o) was built for newer macOS version (11.0) than being linked (10.8)

One of these for every pascal unit in my code, and one for every c module in the openSSL code, finishing like this:

Error: ld: warning: object file (/Users/grahame/work/openssl/libcrypto.a(p12_attr.o)) was built for newer macOS version (11.0) than being linked (10.8)

The environment make file is /usr/bin/make - is that the linker? How should I fix the macOS linking version error? and will that fix the openSSL linking error?

btw,
* if I define -Xt, then I get a different error: "Error: ld: library not found for -lc". Should I do something about this? c?
* if I define -WP11.0, which some posts here seem to suggest, I get that this is an unrecognised compiler option. (or any other number)

As you can probably tell, I'm completely ignorant about linking stuff, and I have no idea how to progress this...

btw, Lazarus + FPC = trunk yesterday. Mac = Mac M1 running Big Sur 11.6. Xcode v13.0. Are any other versions relevant?

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on November 23, 2021, 10:39:00 am

well, immediately after I posted this, I found https://wiki.freepascal.org/Mac_Installation_FAQ#ld:_library_not_found_for_-lc. Which is kind of helpful, except that it's not. Put these things on the command line - which command line? When I run Lazarus? I tried all the likely places in project options... not them...?

Title: Re: Static Linking OpenSSL and OSX version
Post by: AlexTP on November 23, 2021, 10:46:13 am

I see that 'Project options' dialog gives the "Compiler Options / Custom Options" memo-field.

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on November 23, 2021, 11:13:56 am

Yep, it does. What would I put in there that might make a difference? because I haven't figured it out

Title: Re: Static Linking OpenSSL and OSX version
Post by: AlexTP on November 23, 2021, 11:35:00 am

I am ignorant in this, but a guess: your .o object files must be recompiled on macOS 11, and now they are for 10.x.

Title: Re: Static Linking OpenSSL and OSX version
Post by: trev on November 23, 2021, 11:43:37 am

Quote from: Grahame Grieve on November 23, 2021, 10:39:00 am

well, immediately after I posted this, I found https://wiki.freepascal.org/Mac_Installation_FAQ#ld:_library_not_found_for_-lc. Which is kind of helpful, except that it's not. Put these things on the command line - which command line? When I run Lazarus? I tried all the likely places in project options... not them...?

If you read what I wrote in that FAQ, you'll see it was in the context of compiling FPC on the command line with make. So, in a different context, you'll need to adapt the solution. I suggest adding "-XR/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" to your /etc/ fpc.cfg file to solve any such issue in the future.

As to the OpenSSL issues, I would just avoid them altogether and use the Apple Network Framework which doesn't require additional libraries which is what I've been doing. See my articles in the Wiki on NSURLConnection (now deprecated) and NSURLSession (macOS 10.14 onwards).

If you wish to continue to pursue static linking OpenSSL libraries, you could refer to my article on macOS Static Libraries.

As to "object file xxx was built for newer macOS version (11.0) than being linked",  you need to set the compiler flag -WM (eg -WM10.x) to the version number of the SDK you want to build against. So, add -WM10.x (or whatever) to your /etc/fpc.cfg file.

Finally, how did you install FPC and Lazarus because these odd issues you're encountering are not normal and I've never encountered them using the official Lazarus pkg installers.

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on November 23, 2021, 11:43:52 am

well, I just compiled them all afresh right now, either using the openssl build or compiling them on the M1 Mac I'm using to link them

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on November 23, 2021, 11:57:27 am

> If you read what I wrote in that FAQ, you'll see it was in the context of compiling FPC on the command line with make. > So, in a different context, you'll need to adapt the solution. I suggest adding "-XR/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" to your /etc/ fpc.cfg file to solve any such issue in the future.

I don't have an /etc/fpc.cfg file? Should I create it?

> As to the OpenSSL issues, I would just avoid them altogether and use the Apple Network Framework which doesn't require additional libraries which is what I've been doing. See my articles in the Wiki on NSURLConnection (now deprecated) and NSURLSession (macOS 10.14 onwards).

I use openSSL extensively for way more than HTTP client - quite a bit of crypto stuff, and an HTTP server. And it's a cross platform app, so simply using NSURLConnection really doesn't seem like an option to me. (The apple people keep saying that to me too)

> If you wish to continue to pursue static linking OpenSSL libraries, you could refer to my article on macOS Static Libraries.

I'll read it again, and see if I'm ready to learn more from it.

> As to "object file xxx was built for newer macOS version (11.0) than being linked",  you need to set the compiler flag -WM (eg -WM10.8) to the version number of the SDK you want to build against. So, add -WM10.8 (or whatever) to your /etc/fpc.cfg file.

well, that fixed the linking problem with openSSL, and now I get lots of access violations to play with. Thanks.

> Finally, how did you install FPC and Lazarus because these odd issues you're encountering are not normal and I've never encountered them using the official Lazarus pkg installers.

Using fpclazup, this command: ./tools/fpclazup --noconfirm --fpcVersion=trunk.gitlab --lazVersion=trunk.gitlab --installdir=tools --include=anchordocking,lazprojectgroups,virtualtreeview,fpdebug

Title: Re: Static Linking OpenSSL and OSX version
Post by: trev on November 23, 2021, 12:18:02 pm

Quote from: Grahame Grieve on November 23, 2021, 11:57:27 am

Quote from: trev

Finally, how did you install FPC and Lazarus because these odd issues you're encountering are not normal and I've never encountered them using the official Lazarus pkg installers.

Using fpclazup, this command: ./tools/fpclazup --noconfirm --fpcVersion=trunk.gitlab --lazVersion=trunk.gitlab --installdir=tools --include=anchordocking,lazprojectgroups,virtualtreeview,fpdebug

Aha. I think that goes some way to explaining some of the odd issues you have encountered.

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on November 23, 2021, 01:17:52 pm

it's not clear how else to install Lazarus (trunk version), other than fpcLazup or FpcupDeluxe. Anyway, the access violations are because while the whole thing compiles, none of the openSSL code is actually linked in. I guess I need -Xt for that, which brings me back to ... should I have a /etc/fpc.cfg?

Title: Re: Static Linking OpenSSL and OSX version
Post by: DonAlfredo on November 23, 2021, 01:55:14 pm

We solved the -lc error together on Github. But this was limited to the compiler build itself.
This issue seems to make it clear that the same method should be used for FPC itself during use.
That would indeed mean an addition into fpc.cfg.
Will have a look.

Title: Re: Static Linking OpenSSL and OSX version
Post by: trev on November 23, 2021, 11:57:03 pm

Quote from: Grahame Grieve on November 23, 2021, 01:17:52 pm

it's not clear how else to install Lazarus (trunk version), other than fpcLazup or FpcupDeluxe.

I have instructions on the Wiki (Installing Lazarus on macOS (https://wiki.lazarus.freepascal.org/Installing_Lazarus_on_macOS)) for installing from source.

I also have daily automated snapshot builds of both aarch64 FPC and aarch64 Lazarus development (trunk in svn speak or main in git speak) versions available for download from  SourceForge (https://sourceforge.net/projects/macos-lazarus-snapshots/).

Title: Re: Static Linking OpenSSL and OSX version
Post by: Remy Lebeau on November 27, 2021, 06:50:23 am

Quote from: Grahame Grieve on November 23, 2021, 10:30:27 am

{$IFDEF FPC}
  {$DEFINE STATICLOAD_OPENSSL}
  {$LINKLIB libcrypto.a}
  {$LINKLIB libssl.a}
{$ENDIF}
           
(The STATICLOAD_OPENSSL is a define in the openSSL 1.1x branch of Indy)

First, that branch is OLD, and probably needs to be deleted. Indy's latest OpenSSL 1.1x support code is in this pull request (https://github.com/IndySockets/Indy/pull/299) instead.

Second, Indy is coded to support static linking of OpenSSL only on iOS systems (see the IdSSLOpenSSLHeaders_static.pas unit). But even so, STATICLOAD_OPENSSL is not something you can just define outside of Indy and have it magically take effect inside of Indy. That define needs to be enabled inside of Indy's own code, specifically in the IdCompilerDefines.inc file(s), and then Indy would need to be recompiled.

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on November 28, 2021, 11:17:42 am

@trev:

> I have instructions on the Wiki (Installing Lazarus on macOS) for installing from source.

I tried to follow them, but all the compiles on current source fail. Maybe I have to install older versions from the downloads first (that's not clear), before I can build the current versions, but the downloads don't support M1 Macs... I couldn't figure out from that page what I actually have to do to get things to work on my Mac M1

@Remy

I am using the openSSL 1.1 branch of Indy (well, my own custom hack of it for memory management and thread tracking reasons). So I have to figure out how to statically bind openSSL on OSX, and yes, of course, Indy will have to be recompiled for that

Title: Re: Static Linking OpenSSL and OSX version
Post by: Remy Lebeau on November 29, 2021, 08:01:02 pm

Quote from: Grahame Grieve on November 28, 2021, 11:17:42 am

@Remy

I am using the openSSL 1.1 branch of Indy

Again, that is an OLD branch, you really shouldn't be using it at all.

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on November 30, 2021, 05:39:11 am

well, what current branch is tied to openSSL 1.1? I need openSSL 1.1...

Title: Re: Static Linking OpenSSL and OSX version
Post by: Remy Lebeau on November 30, 2021, 05:23:43 pm

Quote from: Grahame Grieve on November 30, 2021, 05:39:11 am

well, what current branch is tied to openSSL 1.1? I need openSSL 1.1...

There is no branch for OpenSSL 1.1.x.  As I stated a few days ago:

Quote from: Remy Lebeau on November 27, 2021, 06:50:23 am

First, that branch is OLD, and probably needs to be deleted. Indy's latest OpenSSL 1.1x support code is in this pull request (https://github.com/IndySockets/Indy/pull/299) instead.

The OpenSSL 1.1.x code is still a WIP, use the master branch and add the pull request's code on top of it.  OpenSSL 1.1.x support is being implemented as a new set of SSLIOHandler components that are specific to 1.1.x, not as an update to the existing SSLIOHandler components that are for 1.0.2 and earlier.

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on December 02, 2021, 01:22:39 pm

Remy, I'm not explaining myself well, that is the code I'm using for Indy

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on December 04, 2021, 09:46:00 pm

well, back to this... I now have an fpc.cfg, and it has an entry in it for -XR pointing to the relevant command line SDK, which does exist. And I remove it, I get "Error: ld: framework not found Cocoa". So I think the compiler is seeing it, but I still get "Error: ld: library not found for -lc". Is there a way to get the command line passed to ld?

Title: Re: Static Linking OpenSSL and OSX version
Post by: trev on December 05, 2021, 12:09:12 am

I think these configuration issues are related to how you originally installed FPC. We can keep patching the existing configuration files for every issue that comes up or delete everything and start afresh with the official installers from SourceForge (https://sourceforge.net/projects/lazarus/files/Lazarus%20macOS%20x86-64/) which I know work.

I think your current configuration issue might be resolved by adding the line:

Code: Text  [Select][+][-]

1. -Fl/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib

to your fpc.cfg file.

If there's any more configuration issues though, I would strongly recommend deleting everything and starting again with the official installers and then building the development version of FPC (and Lazarus if necessary) from source. It will be quicker in the long run.

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on December 05, 2021, 11:53:50 am

That line already exists in the fpc.cfg file. And I have deleted everything and reinstalled following the instructions on the wiki page

Title: Re: Static Linking OpenSSL and OSX version
Post by: trev on December 05, 2021, 12:12:45 pm

1. if you run Lazarus and create a new project for an application and then compile that otherwise empty project, does it fail?

2. if yes, please cut and paste the contents of Lazarus > View > IDE Internals > About FPC from Lazarus version down to just before Defines.

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on December 06, 2021, 09:54:09 am

I create a new application and compile it - compiles ok. (doesn't run - some LLDB problem I'll get back to later). Then I give it the option -Xt and get the image attached.

View as requested:

Lazarus version: 2.3.0
Lazarus revision: main-2_3-757-g217459a76f
Lazarus build date: 2021/12/02
Lazarus was compiled for x86_64-darwin
Lazarus was compiled with FPC 3.2.2

Environment variables:
PATH=/usr/bin:/bin:/usr/sbin:/sbin
PP=
FPCDIR=
USESVN2REVISIONINC=
USER=grahame
HOME=/Users/grahame
PWD=
LANG=
LANGUAGE=

Global IDE options:
LazarusDirectory=../work/lazarus/laz_main/
Resolved LazarusDirectory=/Users/grahame/work/lazarus/laz_main/
Project's CompilerFilename=$(CompPath)
Resolved Project's CompilerFilename=/usr/local/bin/fpc
Default CompilerFilename=/usr/local/bin/fpc
Resolved default compilerFilename=/usr/local/bin/fpc
CompilerMessagesFilename=
Resolved CompilerMessagesFilename=/Users/grahame/work/lazarus/fpc_main/compiler/msg/errore.msg

Project:
lpi=
Directory=
TargetOS=
TargetCPU=
CompilerFilename=$(CompPath)
CompilerOptions=

Active target:
TargetOS=darwin
TargetCPU=aarch64

FPC executable:
Compiler=/usr/local/bin/fpc
Options=
CompilerDate=12/04/21 10:02:05 PM
RealCompiler=/usr/local/bin/ppca64
RealCompilerDate=12/04/21 10:02:05 PM
RealTargetOS=darwin
RealTargetCPU=aarch64
RealCompilerInPath=
Version=3.3.1
CfgFilename=/etc/fpc.cfg

Title: Re: Static Linking OpenSSL and OSX version
Post by: trev on December 06, 2021, 10:13:24 am

As far as I know macOS frameworks, unlike iOS frameworks, are dynamic and not static which would explain why the Cocoa framework is not being found when you try to link it statically.

[Edit]

Quote

The executable code in a framework bundle is a dynamically linked, shared library—or, simply, a dynamic shared library. This is a library whose code can be shared by multiple concurrently running programs.

Apple Source (https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPFrameworks/Concepts/FrameworkBinding.html#//apple_ref/doc/uid/20002256-BAJICBDD).

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on December 06, 2021, 11:00:59 am

umm, so what do I do differently? Or I can't do static linking on OSX because of that?

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on December 06, 2021, 11:02:19 am

but also, the Cocoa framework is found, unless I set -XR to something wrong. It's 'c' that's not being found

Title: Re: Static Linking OpenSSL and OSX version
Post by: trev on December 06, 2021, 11:13:22 am

Argh - not sure why I got hung up on Cocoa except the fact that frameworks are dynamic. Brain not keeping up with my typing.

There is no static libc library in macOS. Instead, there is the dynamic system library (/usr/lib/libSystem.dylib) which includes, among other libraries, libc.

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on December 06, 2021, 11:23:28 am

..so I'm lost. What I want do it sounded really simple: statically link libcrypto.a and libssl.a, which I have in my path, into my application, so that I can work around the fact the libssl dependencies on libcrypto do not meet OSX hardened application requirements. I added $linklib for both of them, but they don't actually seem to be linked, which is why I was trying -Xt... perhaps I shouldn't be? is what I want to do possible?

Title: Re: Static Linking OpenSSL and OSX version
Post by: trev on December 06, 2021, 11:28:52 am

Sure, you can statically link your own library into an FPC application - see the examples in my Wiki  article on macOS Static Libraries (https://wiki.lazarus.freepascal.org/macOS_Static_Libraries) where I give examples of statically linking both FPC and C libraries.

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on December 07, 2021, 11:15:27 am

Well, and it turns out it that is that easy - once the compile stuff is all sorted out, $linklib works just fine for openSSL. Now, to check it works after hardening...

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on December 07, 2021, 11:18:18 am

Anyway, thanks for all the help - it really made a difference. And as was pretty obvious, I was way out of my depth with OSX and with linking questions

Title: Re: Static Linking OpenSSL and OSX version
Post by: trev on December 07, 2021, 11:20:41 am

You're welcome!

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on December 09, 2021, 10:44:32 am

and, btw, once I have the compiler issues sorted out, statically linking openSSL is pretty straight forward. I'm using the openSSL 1.1 version of Indy (currently, just a PR) and I compiled the static headers in, and added:

{$IFDEF FPC}
  {$IFDEF DARWIN}
  { On MacOS, and only on MacOS, we statically bind to openSSL
    because the openSSL libssl.dylib doesn't bind to libcrypto.dylib
    in a way that will be approved by the OSX loader for hardened
    apps. But it's not worth the effort to statically bind on other
    platforms - On linux, openSSL is baked in. On windows, we distribute
    it with the application
  }
  {$LINKLIB libcrypto.a}
  {$LINKLIB libssl.a}
  {$ENDIF}
{$ENDIF}   

Abd bingo, it's compiled into my executable, as long as I have the openssl libraries on my path. Building them is easy (on anything but windows) - just follow the openssl instructions and then make sure those two files are on the path.

What I haven't figured out is that my practice is to use dynamic loading otherwise. So I want an Indy package that doesn't bind to either dynamic or static, and the application that uses it decides which way it's going to be bound. Lazarus packages just don't work that way. I can't even figure out how to $IFDEF the package for dynamic vs static based on OS, since all the units have the same names and just differ by path - and I don't know how to do that with the Lazarus package system. So for now, all FPC applications are statically bound to the .so / .dll. I will probably get around to statically linking to the .a

btw, can't tell you how much confusion I had because of the double use of 'static' there - statically linked to the .dll is totally different in meaning to being statically linked to the code directly. I went down several dead ends because of that

Title: Re: Static Linking OpenSSL and OSX version
Post by: Remy Lebeau on December 09, 2021, 07:24:17 pm

Quote from: Grahame Grieve on December 09, 2021, 10:44:32 am

What I haven't figured out is that my practice is to use dynamic loading otherwise. So I want an Indy package that doesn't bind to either dynamic or static, and the application that uses it decides which way it's going to be bound. Lazarus packages just don't work that way. I can't even figure out how to $IFDEF the package for dynamic vs static based on OS, since all the units have the same names and just differ by path - and I don't know how to do that with the Lazarus package system. So for now, all FPC applications are statically bound to the .so / .dll. I will probably get around to statically linking to the .a

Indy has an IdSSLOpenSSLHeaders_static.pas unit for static-linking to OpenSSL's .a files.  But that unit is currently only used for iOS.  So, you could try updating that unit to support OSX too (and then add the unit to your uses clause with an {$IFDEF}), as well as update IdCompilerDefines.inc to define STATICLOAD_OPENSSL for OSX, and then recompile Indy.

Title: Re: Static Linking OpenSSL and OSX version
Post by: Grahame Grieve on December 09, 2021, 08:41:30 pm

Hey Remy, I do have STATICLOAD_OPENSSL in my idCompilers.inc, but the IdSSLOpenSSLHeaders_static unit isn't in the openSSL 1.1 PR (not that I can see anyway). But I didn't ask about compiling, which I could figure out for sure. The problem for me is the way Lazarus Package management works

Title: Re: Static Linking OpenSSL and OSX version
Post by: Remy Lebeau on December 10, 2021, 06:46:37 pm

Quote from: Grahame Grieve on December 09, 2021, 08:41:30 pm

the IdSSLOpenSSLHeaders_static unit isn't in the openSSL 1.1 PR

Because the PR doesn't use the IdSSLOpenSSLHeaders unit.  It has its own set of units for importing OpenSSL 1.1.x.  I haven't reviewed the PR yet, so I don't know whether it includes any static-linking support or not.