Lazarus

Programming => Operating Systems => macOS / Mac OS X => Topic started by: Gizmo on March 22, 2021, 08:40:35 pm

Title: [SOLVED] How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: Gizmo on March 22, 2021, 08:40:35 pm
An application of mine that has run on OSX for a few years is unable to run properly on OSX Big Sur because my application tries to load libsqlite3.dylib like this :

Code: Pascal  [Select][+][-]
  1. LibHandle := dlopen('libsqlite3.dylib', RTLD_LAZY);
  2.     if LibHandle <> nil then          
  3. ...
  4.  

However, I gather the following issue is the cause :
         
"New in macOS Big Sur 11.0.1, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache. (62986286)"

So my question is how do I direct my TSQLDBLibraryLoader object to use the right library on OSX Big Sur?

Many thanks
Title: Re: How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: Jonas Maebe on March 22, 2021, 08:46:52 pm
You are using dlopen, so the issue is not because of that change (that one only causes issues if you e.g. try to use FileExists to check whether the library file exists before calling dlopen, since that check would fail).

However, normally you should always provide the absolute path to the library, unless it's in the current working directory. So I think it should be dlopen('/usr/lib/libsqlite3.dylib'). I don't know why it worked for you on previous OS versions.
Title: Re: How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: Gizmo on March 22, 2021, 10:40:35 pm
That doesn't work either I'm afraid.

Doing a locate on my new Big Sur system shows the following only :

Code: [Select]
/Applications/Amazon Photos.app/Contents/MonoBundle/libsqlite3.dylib
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/Library/CoreSimulator/Profiles/Runtimes/tvOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libsqlite3.dylib
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/Library/CoreSimulator/Profiles/Runtimes/watchOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libsqlite3.dylib
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/Library/CoreSimulator/Profiles/Runtimes/iOS.simruntime/Contents/Resources/RuntimeRoot/usr/lib/libsqlite3.dylib

All of which aren't guaranteed to be the same on users computers?
Title: Re: How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: Jonas Maebe on March 22, 2021, 10:50:00 pm
You won't find a file with the library's name because even if it's there, it will be part of the shared dyld cache mentioned in your original message. Similarly, you won't find any other library, including the standard C library, on your system, but that does not mean they can't be used (if the standard C library couldn't be used, not a single program could be linked or started). In other words, looking for system library files by name on a macOS 11.0 does not tell you anything about whether or not it can be used.

To debug the issue, try starting by printing the error returned by dlopen (the result of dlerror). It's possible the library is no longer included with Big Sur, but I'm not sure (I don't feel like booting my Big Sur partition right now to find out, and googling doesn't immediately turn anything up either).

Do not try to use one of the libsqlite's from random other applications, and not just because they might not be there. That's just bad practice.
Title: Re: How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: PierceNg on March 23, 2021, 02:34:10 am
So my question is how do I direct my TSQLDBLibraryLoader object to use the right library on OSX Big Sur?

How about linking your own copy of libsqlite statically into your application? This way, your application is sure to get all the SQLite functionality it requires.
Title: Re: How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: trev on March 23, 2021, 06:14:12 am
There used to be (Mojave and Catalina) /usr/lib/libsqlite3.dylib but no more.

Apple encourages developers to provide their own needed libraries in their application bundle's Frameworks directory and discourages any reliance on system provided third-party libraries which are now being thinned.

See the Wiki article macOS Dynamic Libraries (https://wiki.freepascal.org/macOS_Dynamic_Libraries) which also includes debugging strategies.
Title: Re: How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: Gizmo on March 23, 2021, 09:53:58 am
Ifound this which is interesting :

https://developer.apple.com/forums/thread/655588

Specifically :

"One problem with these SDKs is that they’re rather large. Once significant contribution to that size was the libraries. Each SDK would include a copy of the libraries necessary to link for that platfrom. However, this is nonsense when you think about it. There’s no point shipping an iOS library on macOS because it’s just a bunch of code that can’t ever execute [1]. The only thing that the linker needs from these libraries is a list of symbols. So in recent Xcode releases we’ve replaced the libraries in your SDKs with stub libraries [2]. That is, there’s no longer a libpcap.dylib library in the SDK, but rather a much smaller libpcap.tbd stub library [3]. It’s a text file (.tbd stands for text based description) so feel free to open it up and take a look.

Given the above reality, the libraries in /usr/lib are no longer needed:

 Developer tools should be looking at the stub libraries in the appropriate SDK.

 The runtime doesn’t use these libraries because they’ve been rolled into the dynamic linker shared cache [4].

And that’s why they were removed.

This is fine if you’re using Xcode, or Apple’s command-line tools, because they know how to look inside an SDK for headers and stub libraries. If you’re using third-party tools then you need to consult the support resources for those tools to find out how they’ve adjusted to this new reality. In your case the critical point is the linker. If your third-party tools use the Apple linker then you should just be able to point the tools at the usr/include directory within the appropriate SDK
."

So I guess the question here is whether\if Lazarus IDE and FPC linker will ever be able to utilise this system for us. Meanwhile, I will try to work out how to compile libsqlite3.dylib myself and include it with my application. Or whatever. The pains of developing for OSX.
Title: Re: How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: PascalDragon on March 23, 2021, 10:02:52 am
So I guess the question here is whether\if Lazarus IDE and FPC linker will ever be able to utilise this system for us. Meanwhile, I will try to work out how to compile libsqlite3.dylib myself and include it with my application. Or whatever. The pains of developing for OSX.

FPC uses the system provided linker. But that is only relevant for compile time. At runtime, when you use dlopen and such you'll have the same “problems” and restrictions as e.g. C code, because dlopen is simply a call to the C library which does all the magic.
Title: Re: How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: trev on March 23, 2021, 11:23:08 am
There used to be (Mojave and Catalina) /usr/lib/libsqlite3.dylib but no more.

I take that back:

Code: [Select]
trev@macmini8 [/Users/trev] $ sw_vers
ProductName: macOS
ProductVersion: 11.2.3
BuildVersion: 20D91

trev@macmini8 [/Users/trev] $ dyld_shared_cache_util -list | grep sqlite
/usr/lib/libsqlite3.dylib
Title: Re: How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: trev on March 23, 2021, 11:40:37 am
Code: Pascal  [Select][+][-]
  1. program libtest;
  2.  
  3. uses
  4.   Dynlibs, SysUtils;
  5.  
  6. var
  7.   // create a handle for the dynamic library
  8.   LibHandle : TLibHandle;
  9.  
  10. begin
  11.   // load and get the dynamic library handle
  12.   LibHandle := loadLibrary(PChar('libsqlite3.dylib'));
  13.  
  14.   // check whether loading was successful
  15.   if LibHandle <> 0 then
  16.     writeLn('library loads successfully')
  17.   else
  18.     writeLn('GetLastOSError = ' + SysErrorMessage(GetLastOSError));
  19.  
  20.   // unload library
  21.   if LibHandle <> NilHandle then
  22.     begin
  23.       writeLn('unloading library');
  24.       unloadLibrary(LibHandle);
  25.     end;
  26.  
  27.   LibHandle := NilHandle;
  28. end.

This works for me with FPC trunk on Big Sur aarch64 (a base M1 Mac mini).

Code: [Select]
trev@macmini8 [/Users/trev/Documents/fpc_libsqlite3] $ ./program
library loads successfully
unloading library
Title: Re: How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: Gizmo on March 23, 2021, 02:49:43 pm
Where are you getting dyld_shared_cache_util, Trev? I don't have it on my Big Sur?

I quickly tried switching to LoadLibrary instead of dlopen as you've shown with same problem. There is no trace of libsqlite3.dylib on my Big Sur right now, that I have been able to find. Though dyld_shared_cache_util looks like the ticket to searching as per my post above re the change in OSX.
Title: Re: How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: Jonas Maebe on March 23, 2021, 06:41:15 pm
Where are you getting dyld_shared_cache_util, Trev? I don't have it on my Big Sur?
You have to compile it from source: https://wiki.freepascal.org/macOS_Big_Sur_changes_for_developers#Built-in_dynamic_linker_cache

Quote
There is no trace of libsqlite3.dylib on my Big Sur right now, that I have been able to find.
As I have mentioned two times before, and as is mentioned even in text you quoted yourself in this thread [1], that is completely normal (for that library, and also for every other library from /usr included in macOS 11), and it has nothing to do with the problem you have.

[1]: "The runtime doesn’t use these libraries because they’ve been rolled into the dynamic linker shared cache"
Title: Re: How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: trev on March 24, 2021, 05:00:03 am
I quickly tried switching to LoadLibrary instead of dlopen as you've shown with same problem.

What happens if you compile and run my code above? I suspect your issue must lie elsewhere in which case refer to the debugging strategies I pointed you to earlier.

Jonas cover your other points.
Title: Re: How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: Gizmo on April 22, 2021, 12:43:43 pm
Trev, Jonas, and others.

Yes, the problem is indeed elsewhere.

The code I put above refers to the loading of the library. And yes Trev, your example does indeed work in as much as it loads the library with success.

However, the problem is later on in my procedure where I specify the PATH for SQLDBLibraryLoader. As follows :

Code: Pascal  [Select][+][-]
  1. SQLDBLibraryLoaderOSX.ConnectionType := 'SQLite3';
  2.   SQLiteLibraryPath := '';
  3.  
  4.   LibHandle := loadLibrary(PChar('libsqlite3.dylib'));
  5.  
  6.   // check whether loading was successful
  7.   if LibHandle <> 0 then
  8.     begin
  9.       ShowMessage('Library loads OK');
  10.       {Pdlinfo := LibHandle;  // This section will not compile anymore, on BigSur, so I can't set LibraryPath
  11.       PtrSQLiteLibraryPath := Pdlinfo^.dli_fbase;
  12.       SQLiteLibraryPath := String(PtrSQLiteLibraryPath);
  13.       PtrSQLiteLibraryPath := nil;}
  14.     end
  15.   else ShowMessage('Cannot load SQLite libraries for backend use.' + SysErrorMessage(GetLastOSError));
  16.  
  17.   // unload library
  18.   if LibHandle <> NilHandle then
  19.     begin
  20.       unloadLibrary(LibHandle);
  21.     end;
  22.  
  23.   LibHandle := NilHandle;  
  24.  
  25. if FileExists(SQLiteLibraryPath) then
  26.     begin
  27. ...
  28. compiler refs for Windows, Linux and OSX...    
  29. ...
  30. {$ifdef darwin}
  31.         SQLDBLibraryLoaderOSX.LibraryName := SQLiteLibraryPath;
  32.         SQLDBLibraryLoaderOSX.Enabled := true;
  33.         SQLDBLibraryLoaderOSX.LoadLibrary;
  34.       {$endif}      
  35.  

The reason that is failing is because SQLibraryPath is not (can not?) now be populated on BigSur in the same way as I was doing before on earlier OSX releases, which was as follows :

Code: Pascal  [Select][+][-]
  1.     SQLDBLibraryLoaderOSX.ConnectionType := 'SQLite3';
  2.     SQLiteLibraryPath := '';
  3.     LibHandle := dlopen('libsqlite3.dylib', RTLD_LAZY);
  4.     if LibHandle <> nil then
  5.     begin
  6.       Pdlinfo := LibHandle;
  7.       PtrSQLiteLibraryPath := Pdlinfo^.dli_fbase;
  8.       SQLiteLibraryPath := String(PtrSQLiteLibraryPath);
  9.       PtrSQLiteLibraryPath := nil;
  10.       dlclose(LibHandle);
  11.     end;
  12. ...
  13. {$ifdef darwin}
  14.         SQLDBLibraryLoaderOSX.LibraryName := SQLiteLibraryPath;
  15.         SQLDBLibraryLoaderOSX.Enabled := true;
  16.         SQLDBLibraryLoaderOSX.LoadLibrary;
  17.       {$endif}      
  18.  


And if I try just specifying the name without path it obviously compiles, but still can't launch SQLite.

Code: Pascal  [Select][+][-]
  1.  SQLDBLibraryLoaderOSX.LibraryName := 'libsqlite3.dylib';
  2.  

In other words, it seems from the replies here already that asking Big Sur for 'libsqlite3.dylib' should 'just work' and the calling application will be 'given' the library from the OSX cache, without having to provide a specific path (as such a path no longer exists). But whether I define it in code, or via the SQLDBLibraryLoder object, it does not work.
Title: Re: How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: Gizmo on April 22, 2021, 01:59:57 pm
SOLVED.

You were right. My problem was further down.

Setting SQLDBLibraryLoaderOSX.LibraryName to 'libsqlite3.dylib' does indeed work fine!

Thanks for the help
Title: Re: [SOLVED] How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: trev on April 23, 2021, 03:19:24 am
Great to hear you squashed your bug, and thanks for returning to let us know.
Title: Re: [SOLVED] How do I direct TSQLDBLibraryLoader to use SQLite on Big Sur OSX?
Post by: Gizmo on April 23, 2021, 10:12:46 am
The least I can do for sure is to do that.

For info, for anyone else, the reason it was failing despite the changes was because later on I run a "if FileExists(path to library)" which of course returns false, now, because the file does not exist on the filesystem in the same way with BigSur. So I basically created 3 ifdef compiler directives - one for Windows, one for Linux and one for OSX, and the OSX one does not do the file lookup.
TinyPortal © 2005-2018