TFPHTTPClient on MacOS

[dbannon]

Folks, I use code like below to download from an https site, on linux and windows, it works fine since FPC3.2.0 and I am currently using Laz2.2.0.

But I have just noticed it fails with an ESocketError on my (aging) MacOS box, (strange, I was sure I have tested it in the past). Anyone suggest why ?  And is there a practical alternative ?

Code: Pascal  [Select][+][-]

1. var
2.     Client: TFPHTTPClient;
3.     .....
4.     Client := TFPHttpClient.Create(nil);
5.     Client.AddHeader('User-Agent','Mozilla/5.0 (compatible; fpweb)');
6.     Client.AddHeader('Content-Type','application/json; charset=UTF-8');
7.     Client.AllowRedirect := true;
8.     SomeString := Client.Get(URL);

     

Actual code in the attached.

Davo

[sstvmaster]

Try insert InitSSLInterface befor creating the client, it is needed since fpc 3.2.0.

You must insert openssl in uses to use InitSSLInterface.

Code: Pascal  [Select][+][-]

1. var
2.     Client: TFPHTTPClient;
3. begin
4.     { SSL initialization has to be done by hand here }
5.     InitSSLInterface;
6.  
7.     Client := TFPHttpClient.Create(nil);
8. ...
9. end;
10.  

see -> https://wiki.lazarus.freepascal.org/fphttpclient

[rvk]

Try insert InitSSLInterface befor creating the client, it is needed since fpc 3.2.0.

Small correction... It's NOT needed anymore since 3.2.2.

Hence:

Folks, I use code like below to download from an https site, on linux and windows, it works fine since FPC3.2.0 and I am currently using Laz2.2.0.

[dbannon]

thanks sstvmaster, rvk, I am using FPC3.2.0 on the Mac and have tried both with and without the Init statement, it fails in both cases on the Mac but works fine on Linux (FPC3.2.2) and Windows (FPC3.2.0).

Is there some suggestion that upgrading to FPC3.2.2 on the Mac might make a difference ?

Otherwise, is the any Mac users out there who would be willing to test the simple app attached to first post of this thread ?  I do wonder if my continued use of MacOS Sierra with some Brew installed OpenSSL is the problem.

I do not have access to a more recent Mac and are very reluctant to upgrade my 2012 Powerbook . Sigh ...

Davo

[rvk]

I am using FPC3.2.0 on the Mac and have tried both with and without the Init statement, it fails in both cases on the Mac but works fine on Linux (FPC3.2.2) and Windows (FPC3.2.0).

Yeah. I don't have a Mac so I can't test it.

You could look at the suggestion given here:
https://forum.lazarus.freepascal.org/index.php/topic,53613.msg396820.html#msg396820

Especially the link to patch openssl.pas here:
https://forum.lazarus.freepascal.org/index.php/topic,34058.msg396680.html#msg396680

Not sure if that fixes it for you too.

[dbannon]

Hmm, given how I want this app to work, patching FPC is not really an option.  But following that trail, I find this -

"The issue has been fixed in FPC trunk (see Bug #37977) and is slated for the release of FPC 3.2.2"

In the wiki page https://wiki.lazarus.freepascal.org/macOS_Programming_Tips#OpenSSL.2C_LibreSSL.2C_Secure_Transport.2C_Network_Framework

So, maybe thats an indication FPC3.2.2 is indeed a good idea.  Thanks RVK, some light at the end of the tunnel.

Davo

[trev]

Personally I'd use the macOS NSURLSession API and avoid any dependence on OpenSSL libraries.

You could even use the deprecated, but not going away any time soon, NSURLConnection API.

Example code provided in my two Wiki articles above.

[dbannon]

Personally I'd use the macOS NSURLSession API and avoid any dependence on OpenSSL libraries.

You could even use the deprecated, but not going away any time soon, NSURLConnection API.

Example code provided in my two Wiki articles above.

Both appear to require hacking into the FPC code, "To compile the above code successfully you are going to have to add the missing initWithData function...". Does not seem a very appropriate solution IMHO.  Is this 'hack' (used in its positive meaning) something that will appear in FPC eventually ?

Davo

[dbannon]

"The issue has been fixed in FPC trunk (see Bug #37977) and is slated for the release of FPC 3.2.2"

In the wiki page https://wiki.lazarus.freepascal.org/macOS_Programming_Tips#OpenSSL.2C_LibreSSL.2C_Secure_Transport.2C_Network_Framework

Indeed the code mentioned in the wiki page is in FPC3.2.2 but it not help.

I have both OpenSSL 1.1 and SomethingSSL 0.9.8 and not even sure which one I am linking against. The wiki page indicates both are acceptable but maybe not mixed ....

Davo

[rvk]

Code: Pascal  [Select][+][-]

1. uses fphttpclient, opensslsockets, ssockets, fpopenssl;

Do you actually need the ssocket and fpopenssl ???

There's been a change in the way fphttpclient handles SSL.
Pre FPC 3.2.0 you would use ssockets, sslsockets, openssl and fpopenssl, now from FPC 3.2.0 onwards you only need to use opensslsockets.

So, shouldn't you just omit the ssockets and fpopenssl from the uses clause??

[sstvmaster]

There's been a change in the way fphttpclient handles SSL.
Pre FPC 3.2.0 you would use ssockets, sslsockets, openssl and fpopenssl, now from FPC 3.2.0 onwards you only need to use opensslsockets.

The wiki page should maybe be changed too?

[trev]

Personally I'd use the macOS NSURLSession API and avoid any dependence on OpenSSL libraries.

You could even use the deprecated, but not going away any time soon, NSURLConnection API.

Example code provided in my two Wiki articles above.

Both appear to require hacking into the FPC code, "To compile the above code successfully you are going to have to add the missing initWithData function...". Does not seem a very appropriate solution IMHO. 

The macOS headers are a little long in the tooth, dating from Yosemite, and are missing some of the methods - hence my adding the needed missing methods from the Objective-C header files. There are more recent modern headers produced by automated methods, but they have no testing. The current converted headers have been fixed by hand on many occasions over the years and the same or similar fixes may be necessary for the new ones (according to Ryan who converted the new ones and Jonas who fixed the old ones).

Is this 'hack' (used in its positive meaning) something that will appear in FPC eventually ?

As it is adding a method missing from an incomplete/out of date header file, I would hope so

[dbannon]

....
As it is adding a method missing from an incomplete/out of date header file, I would hope so

But is there a bug report in there with a patch ?  My guess, its not going to happen otherwise.

Davo

[dbannon]

Code: Pascal  [Select][+][-]

1. uses fphttpclient, opensslsockets, ssockets, fpopenssl;

Do you actually need the ssocket and fpopenssl ???
....

No, that lump of code has had a number of hacks applied and discarded so has a bit of history. Pretty sure they do no harm ...

@sstvmaster, yep, wiki page could do with a fixup. But I am reluctant to do so until I see something work. And right now, that does not look likely. If some Mac user with a current install would like to do a test ......

Davo

[PierceNg]

I'm on Catalina. FPC 3.2.2 and Lazarus 2.2.0, both built from source. I don't use Brew. Here are the SSL libraries on my machine:

Code: Bash  [Select][+][-]

1. % ls -l /usr/lib/*ssl*
2. -rwxr-xr-x  1 root  wheel  1489456 Sep 22  2020 /usr/lib/libboringssl.dylib*
3. -rwxr-xr-x  1 root  wheel   212288 Sep 22  2020 /usr/lib/libssl.0.9.7.dylib*
4. -rwxr-xr-x  1 root  wheel   335888 Sep 22  2020 /usr/lib/libssl.0.9.8.dylib*
5. -rwxr-xr-x  1 root  wheel   330576 Feb  5 10:08 /usr/lib/libssl.35.dylib*
6. -rwxr-xr-x  1 root  wheel   313984 Feb  5 10:08 /usr/lib/libssl.43.dylib*
7. -rwxr-xr-x  1 root  wheel   300480 Feb  5 10:08 /usr/lib/libssl.44.dylib*
8. -rwxr-xr-x  1 root  wheel   294016 Feb  5 10:08 /usr/lib/libssl.46.dylib*
9. -rwxr-xr-x  1 root  wheel    32928 Feb  5 10:08 /usr/lib/libssl.dylib*

I modified line 118 of openssl.pas by adjusting the size of the array and adding '.46' as the 2nd item:

Code: Pascal  [Select][+][-]

1.   DLLVersions: array[1..20] of string = ('', '.46', ... the rest of the original stuff

Built and installed the changed version. Then your program works. I also tried setting DLLSSLName to 'libboringssl' just to try but no dice.

I also tested with my own server, which is configured to accept TLS 1.2 or higher with "strong" cipher suites. Without the above mod, same error. With the mod, program works.

So the issue should be Gitlab's and Github's correctly rejecting known-insecure SSL/TLS versions when using outdated OpenSSL libraries.

Maybe a long term solution is for FPC to have an API to specify the particular library files to load?