Recent

Author Topic: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me  (Read 25033 times)

MISV

  • Hero Member
  • *****
  • Posts: 783
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #15 on: September 22, 2018, 01:14:05 pm »
I can confirm the demo project. That works and Synapse uses LibreSSL which appears to work.

Furthermore it is also working with another https domain running a different I believe ssl solution - and there Synapse works as well (again reporting itself as using LibreSSL)

Indy with 0.9.8 fails with both domains - probably due to the age of 0.9.8

...


Preliminary Indy explanation and fix as far as I can tell - I may be missing some things:

IdSSLOpenSSLHeaders.pas
defines
SSLDLLVers : array [0..7] of string = ('.10','.1.0.2','.1.0.1','.1.0.0','0.9.9','.0.9.8','.0.9.7','0.9.6');
Note: I think '0.9.9' should probably have been '.0.9.9' in the above, but that is a lesser... bug?

IdGlobal
On non-Windows HackLoad is used to load ssl (which iterates through SSLDLLVers list) while On Windows HackLoad is not used since the dll is not versioned in filename.

IdSSLOpenSSLHeaders.pas
routine LoadSSLCyrptography
- on windows loads the nonversioned lib file directly
- else if symbolic links supported uses HackLoad - but passes en empty versions array (This does nothing? since HackLoad only does any actions on passed array items)
- else if above fails it expands SSLDLLVers with letters (a,b,c etc.) then uses HackLoad

This means on Mac - Indy never reads the non-versioned file name - i.e. libsll.dylib - and does not read the path shown when *right clicking* - "get info" - "original"

Instead it uses the versioned numbers... Which on my system would be 0.9.8 (which is also the library loaded as mentioned in any earlier post)

Anyhow... That is why Indy never tries to load LibreSSL binaries

...

Note: This also makes fine sense if one is not sure Indy is LibreSSL compatible (Would be fatal to simply then load the library in-case a definition is different/missing. Indy also seems grab pointers to all routines in the lib - and not as needed. I saw this by checking the source code TODO comments.)

...

--What i am working on now--

It seems to work adding ".35" to the list - but I will say that the crawl seems slower than usual... Still looking into it
 
« Last Edit: September 23, 2018, 09:02:44 pm by MISV »

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1312
    • Lebeau Software
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #16 on: September 23, 2018, 08:33:10 pm »
IdSSLOpenSSLHeaders.pas
defines
SSLDLLVers : array [0..7] of string = ('.10','.1.0.2','.1.0.1','.1.0.0','0.9.9','.0.9.8','.0.9.7','0.9.6');
Note: I think '0.9.9' should probably have been '.0.9.9' in the above, but that is a lesser... bug?

Fixed.

IdGlobal
On non-Windows HackLoad is used to load ssl (which iterates through SSLDLLVers list) while On Windows HackLoad is not used since the dll is not versioned in filename.

Correct.

IdSSLOpenSSLHeaders.pas
routine LoadSSLCyrptography
- on windows loads the nonversioned lib file directly
- else if symbolic links supported uses HackLoad - but passes en empty versions array (This does nothing? since HackLoad only does any actions on passed array items)
- else if above fails it expands SSLDLLVers with letters (a,b,c etc.) then uses HackLoad

This means on Mac - Indy never reads the non-versioned file name - i.e. libsll.dylib

That is not correct.  Look again at Indy's code more carefully.  By default, Indy first loads the non-versioned files, in case symlinks are present.  If that fails, then Indy resorts to loading individual versioned files (from newer to older):

Code: [Select]
var
  GIdLoadSymLinksFirst: Boolean = True;

...

  if GIdLoadSymLinksFirst then begin
    Result := {$IFNDEF KYLIXCOMPAT}HMODULE({$ENDIF}
      HackLoad(GIdOpenSSLPath + SSLCLIB_DLL_name, [])
      {$IFNDEF KYLIXCOMPAT}){$ENDIF};
  end;
  if Result = 0 then begin
    for i := Low(SSLDLLVers) to High(SSLDLLVers) do begin
      for j := Low(SSLDLLVersChar) to High(SSLDLLVersChar) do begin
        LLibVersions[j] := SSLDLLVers[i] + SSLDLLVersChar[j];
      end;
      Result := {$IFNDEF KYLIXCOMPAT}HMODULE({$ENDIF}
        HackLoad(GIdOpenSSLPath + SSLCLIB_DLL_name, LLibVersions)
        {$IFNDEF KYLIXCOMPAT}){$ENDIF};
      if Result <> 0 then begin
        Break;
      end;
    end;
  end;
  if (Result = 0) and (not GIdLoadSymLinksFirst) then begin
    Result := {$IFNDEF KYLIXCOMPAT}HMODULE({$ENDIF}
      HackLoad(GIdOpenSSLPath + SSLCLIB_DLL_name, [])
      {$IFNDEF KYLIXCOMPAT}){$ENDIF};
  end;

As you can see, an empty array is passed to HackLoad() only when loading non-versioned files.  Whether Indy loads symlinks first, or versioned files first, is controlled by the GIdLoadSymLinksFirst variable, which is True by default (and can be changed at runtime by calling IdOpenSSLSetLoadSymLinksFirst() before OpenSSL is loaded).

Indy also seems grab pointers to all routines in the lib - and not as needed. I saw this by checking the source code TODO comments.)

Yes, I've been wanting to change that for a long time.  For other external libraries that Indy uses, it loads functions on an as-needed basis.  But for OpenSSL, because of the sheer number of functions being imported, it probably made more sense to load them all up front (I didn't write that code, I only maintain it).

I will try test the LibreSSL binaries shipped with Mac OS now by inserting their file name versions in the SSLDLLVers const.

You should not need to do that, since Indy should be loading the non-versioned symlinks first, which would map to LibreSSL.  Make sure you are using an up-to-date snapshot of Indy.
« Last Edit: September 23, 2018, 08:37:11 pm by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

MISV

  • Hero Member
  • *****
  • Posts: 783
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #17 on: September 23, 2018, 09:10:30 pm »
Quote
var
  GIdLoadSymLinksFirst: Boolean = True;

...

  if GIdLoadSymLinksFirst then begin
    Result := {$IFNDEF KYLIXCOMPAT}HMODULE({$ENDIF}
      HackLoad(GIdOpenSSLPath + SSLCLIB_DLL_name, [])
      {$IFNDEF KYLIXCOMPAT}){$ENDIF};
  end;

Quote
By default, Indy first loads the non-versioned files, in case symlinks are present.

Just to make 100% sure: You are talking about the call to HackLoad in the above code where you are passing empty array []?

If yes, this is the code of HackLoad:

Code: Pascal  [Select][+][-]
  1. function HackLoad(const ALibName : String; const ALibVersions : array of String) : HMODULE;
  2. var
  3.   i : Integer;
  4. begin
  5.   Result := NilHandle;
  6.   for i := Low(ALibVersions) to High(ALibVersions) do
  7.   begin
  8.     {$IFDEF USE_SAFELOADLIBRARY}
  9.     Result := SafeLoadLibrary(HackLoadFileName(ALibName,ALibVersions[i]));
  10.     {$ELSE}
  11.     ...
  12.  

There is no code in HackLoad outside the loop... So if it is passed empty array then nothing happens? But I am tired, so if I missing something then...
« Last Edit: September 23, 2018, 09:13:47 pm by MISV »

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1312
    • Lebeau Software
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #18 on: September 24, 2018, 02:58:07 am »
Just to make 100% sure: You are talking about the call to HackLoad in the above code where you are passing empty array []?

Yes.

If yes, this is the code of HackLoad:
...
There is no code in HackLoad outside the loop... So if it is passed empty array then nothing happens? But I am tired, so if I missing something then...

No, you didn't missed anything.  It was indeed a logic bug in Indy's code, I didn't see it before.  The *INTENT* was to specify an empty array to skip the versioned files, but it was not actually doing that correctly.  I have now fixed it.
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

MISV

  • Hero Member
  • *****
  • Posts: 783
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #19 on: September 24, 2018, 11:24:54 am »
I just downloaded and tried

1)
I am getting funky compile time error:

Quote
IdSSLOpenSSLHeaders.pas(19571,10) Error: function header doesn't match the previous declaration "GetCryptLibHandle:Int64;"
IdSSLOpenSSLHeaders.pas(18191,10) Error: Found declaration: GetCryptLibHandle:QWord;

But looking at code, both places use
Code: Pascal  [Select][+][-]
  1. function GetCryptLibHandle : HMODULE;
  2.  

Following IDE definition for HMODULE when used at line 18191:

Following IDE definition for HMODULE when used at line 19571:
- dynlibs.pas = TLibHandle
- dynlibs.pas = System.TLibHandle
- sysdlh.inc = PrInt
- systemh.inc = Int64

"Fixing" declaration at 18191 to using TLibHandle somehow works. I think this may be a Lazarus compiler bug...?

...

2)
It appears to work - been testing on two extremely troubled sites that do everything possible to make life miserably.
« Last Edit: September 24, 2018, 03:00:43 pm by MISV »

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1312
    • Lebeau Software
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #20 on: September 24, 2018, 07:44:49 pm »
I am getting funky compile time error:

Quote
IdSSLOpenSSLHeaders.pas(19571,10) Error: function header doesn't match the previous declaration "GetCryptLibHandle:Int64;"
IdSSLOpenSSLHeaders.pas(18191,10) Error: Found declaration: GetCryptLibHandle:QWord;

I didn't make any changes related to that, so if it is erroring now, it should have been erroring before, too.

This error implies that one of the units in the 'uses' clause of the 'implementation' section is likely redeclaring HMODULE.  The only non-Indy units in that clause are Classes and DynLibs.  So I'm guessing DynLibs is at fault.  But then, I would expect a similar error in the IdGlobal unit as well, as it defines HackLoad() in the 'interface' section to return an HMODULE (in which case, I guess I should remove the type-casts when IdSSLOpenSSLHeaders calls HackLoad()), and DynLibs is used in the 'implementation' section.

Update: Yup, turrns out that the DynLibs unit does define its own HModule type, as an alias for TLibHandle, which is an alias for PtrInt.  Whereas System.HMODULE is defined as an alias for PtrUInt instead.  Why isn't DynLibs using System.HMODULE instead of defining its own type?
« Last Edit: December 01, 2018, 10:03:31 pm by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1312
    • Lebeau Software
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #21 on: November 03, 2018, 01:27:33 am »
This error implies that one of the units in the 'uses' clause of the 'implementation' section is likely redeclaring HMODULE.  The only non-Indy units in that clause are Classes and DynLibs.  So I'm guessing DynLibs is at fault.  But then, I would expect a similar error in the IdGlobal unit as well, as it defines HackLoad() in the 'interface' section to return an HMODULE (in which case, I guess I should remove the type-casts when IdSSLOpenSSLHeaders calls HackLoad()), and DynLibs is used in the 'implementation' section.

I have checked in some changes to now use THandle everywhere instead of HMODULE directly.  Does the code compile now?
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

Hansaplast

  • Hero Member
  • *****
  • Posts: 674
  • Tweaking4All.com
    • Tweaking4All
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #22 on: November 05, 2018, 04:03:09 pm »

Not sure if this is helpful for your particular application, but OpenSSL (and such) appears to be on the list of things to drop for Apple, since Apple already will no longer allow to be used in sandboxed apps. Per excellent info sheet from Phil, under the section "Internet access":


Note that OpenSSL-based HTTP clients are not supported in sandboxed apps.
This includes Indy, Synapse and FPC's HTTP client. OpenSSL on Mac has been deprecated for a long time, so you should be using one of Apple's APIs instead.

If you need to make GET or POST requests, you can use the ns_url_request.pas unit from here. It uses the Foundation framework's NSURLConnection class.


I had this issue using Synapse, and to my surprise, migrating to Phil's ns_url_request.pas was super easy.
The only thing that took me a few extra seconds was the format of the headers (not Phil's fault).
Synapse uses a notation like this: "Accept: application/json",
whereas the NSURLConnection functions seem to prefer this notation: "Accept=application/json" (notice the ": " versus "=" in the string).


As far as I can tell, this function works faster than the already fast Synapse, and the unit provided by Phil is really tiny compared to Indy and Synapse. Obviously quite a lot of functionality you'll find in Synapse and Indy, are not included. But it works really well for retrieving web content with or without GET or POST.

MISV

  • Hero Member
  • *****
  • Posts: 783
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #23 on: November 24, 2018, 04:44:09 pm »
The problem for me using Phil's code is I need async https requests (since sync mode automatically follows redirects which I do not want) - and I have not been able to solve some freepascal syntax issues with delegates

https://forum.lazarus.freepascal.org/index.php/topic,37468.msg275526.html#msg275526

If you can - great - I gave up on getting the syntax working :(
« Last Edit: November 24, 2018, 11:58:18 pm by MISV »

Hansaplast

  • Hero Member
  • *****
  • Posts: 674
  • Tweaking4All.com
    • Tweaking4All
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #24 on: November 24, 2018, 05:16:08 pm »
@MISV;
Yeah that's a little over my head as well and I do feel your pain. I'm running into similar issues with other functions.
Converting remains a pain and Apple seems to be using different approaches for different frameworks.
For example; I managed to convert one of the callback functions in Disk Arbitration, and it works great. MacOS notifies my app when a disk gets ejected or inserted.
Similar, Cocoa has a function to let an application know that the theme has been changed. Whatever I tried, I can't get it to work.


My best bet would be Phil or Dmitry (skalogryz), but I'm sure they are super busy as well (I know Dmitry has been doing some amazing work on the Cocoa widgetset).
Wish I could help ...

MISV

  • Hero Member
  • *****
  • Posts: 783
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #25 on: December 01, 2018, 12:32:52 am »
IdSSLOpenSSLHeaders.pas
defines
SSLDLLVers : array [0..7] of string = ('.10','.1.0.2','.1.0.1','.1.0.0','0.9.9','.0.9.8','.0.9.7','0.9.6');
Note: I think '0.9.9' should probably have been '.0.9.9' in the above, but that is a lesser... bug?

Fixed.

I just checked the what onlinepackagemanager addin in Lazarus downloads - and it downloads an old version - at least on Lazarus Trunk on Mojave

I inspected it by looking at the above - and the .0.9.9 / 0.9.9 issues is no fixed in the version it uses. (I assume that means it uses an old version)

Not sure where to report but... Now I have posted it here...

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1312
    • Lebeau Software
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #26 on: December 01, 2018, 08:40:11 pm »
I just checked the what onlinepackagemanager addin in Lazarus downloads - and it downloads an old version - at least on Lazarus Trunk on Mojave

I inspected it by looking at the above - and the .0.9.9 / 0.9.9 issues is no fixed in the version it uses. (I assume that means it uses an old version)

Yes, that would imply an older version is being used (double-check with Indy's IdVers.inc file. The current version is 10.6.2.5486).

The current declaration of the SSLDLLVers array looks like this:

Code: Pascal  [Select][+][-]
  1. SSLDLLVers : array [0..7] of string = ('.10','.1.0.2','.1.0.1','.1.0.0','.0.9.9','.0.9.8','.0.9.7','.0.9.6');

Not sure where to report but... Now I have posted it here...

Probably better to post in the Online Package Manager discussion thread instead.  There are several messages related to OPM's version of Indy.  Last I heard, OPM was setup to resync with Indy's SVN periodically, so maybe that resync is no longer working.
« Last Edit: December 01, 2018, 08:42:00 pm by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1312
    • Lebeau Software
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #27 on: December 02, 2018, 02:54:42 am »
Update: Yup, turrns out that the DynLibs unit does define its own HModule type, as an alias for TLibHandle, which is an alias for PtrInt.  Whereas System.HMODULE is defined as an alias for PtrUInt instead.  Why isn't DynLibs using System.HMODULE instead of defining its own type?

I have checked in some further changes to now use TLibHandle on FreePascal, and THandle on Delphi.  Does the code compile now?
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

MISV

  • Hero Member
  • *****
  • Posts: 783
Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
« Reply #28 on: December 02, 2018, 11:35:06 pm »
Yes - it compiles - and also seems to resolve the issue I have reported in the other thread - thank you :)

 

TinyPortal © 2005-2018