Lazarus

Programming => Operating Systems => macOS / Mac OS X => Topic started by: MISV on September 19, 2018, 04:31:43 pm

Title: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: MISV on September 19, 2018, 04:31:43 pm
In my

/usr/lib directory I have

libcrypto.dylib, libcrypto.0.9.7.dylib, libcrypto.0.9.8.dylib, libcrypto.35.dylib, libcrypto.0.41.dylib
libssl.dylib, libssl.0.9.7.dylib, libssl.0.9.8.dylib, libssl.35.dylib, libssl.0.41.dylib

If I use right click - Ge info on libcrypto.dylib I see it points to libcrypto.35.dylib (and similar for libssl.dylib)

Online I have found out that is a LibreSSL and not OpenSSL

...

The case currently is I can not crawl SSL sites using Indy (not those I have tested anyway) ... I do have a fallback to native (but that follows redirects silently which I do not want - I want to keep track of redirects)

How do you guys use Indy on Mac with OpenSSL/LibreSSL?

I am thinking maybe pointing Indy explicitly to OpenSSL 0.9.8 dylibs (which I believe is actually newer/patched versions of OpenSSL but have kept their file names for not breaking compability)


Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: Hansaplast on September 19, 2018, 04:38:28 pm
I'm using Synapse on macOS 10.13.6 and 10.14 (latest public beta), and it works just fine with the default installed SSL libs.
Maybe switch to Synapse? (I have no experience with Indy)
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: MISV on September 19, 2018, 05:44:26 pm
What openssl/libressl libs do you have in your /usr/lib/ folder?
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: MISV on September 19, 2018, 07:37:54 pm
I also sent you a PM - but  I do not see it in my sent PMs list - but I have sen you an URL - if you can scan that using Synapse


Maybe Synapse added support for LibreSSL? But knowing what .dylib files you have in your usr/lib/ would help clarify that :)

(I rather use built-in openssl/libressl instead of shipping  those files myself)
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: Remy Lebeau on September 19, 2018, 07:47:48 pm
The case currently is I can not crawl SSL sites using Indy (not those I have tested anyway) ...

What EXACTLY happens when you try?  Are you getting an exception that Indy can't load the dylibs?  Or are you simply getting an SSL/TLS error?  Please be more specific.

I am thinking maybe pointing Indy explicitly to OpenSSL 0.9.8 dylibs (which I believe is actually newer/patched versions of OpenSSL but have kept their file names for not breaking compability)

Indy doesn't provide an option to specify the exact filenames, only the folder. It then runs through a list of pre-defined filenames until it finds a match. Changing that logic would require altering Indy's source code and recompiling.

If you want to use a specific version of OpenSSL (1.0.2 is the latest supported by Indy), you should put those dylibs in a folder of your choosing, then specify that folder path to Indy via the IdOpenSSLSetLibPath() function at app startup. Apple has deprecated support for OpenSSL on OSX, so you have to deploy your own OpenSSL dylibs with your app if you want to use newer versions of OpenSSL than what Apple still provides for backwards compatibility.

Or, maybe try using a Homebrew version of OpenSSL: Using the OpenSSL library with macOS Sierra (https://medium.com/@timmykko/using-openssl-library-with-macos-sierra-7807cfd47892)

In the meantime, I've opened a feature request to add LibreSSL support to Indy: https://github.com/IndySockets/Indy/issues/231
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: MISV on September 19, 2018, 09:30:42 pm
Here are the following exceptions I get with HEAD/GET requests to multiple https sites

SSLOptions.SSLVersions := [sslvSSLv23];
SSLOptions.Method := sslvSSLv23;
= EIdOSSLUnderlyingCryptoError

SSLOptions.SSLVersions := [sslvTLSv1_2];
SSLOptions.Method := sslvTLSv1_2;
= EIdOSSLUnderlyingCryptoError

SSLOptions.SSLVersions := [sslvTLSv1_1];
SSLOptions.Method := sslvTLSv1_1;
= EIdOSSLUnderlyingCryptoError

SSLOptions.SSLVersions := [sslvTLSv1];
SSLOptions.Method := sslvTLSv1;
= EIdOSSLUnderlyingCryptoError

Since my own website exposes the problem, I can email you example URL to test against (not sure this forum PM works, but I can PM as well)


...



I suspect it would work if I shipped OpenSSL myself. I was just hoping to be able to use system provided libraries


...


Possible solution... Searching for 0.9.8 files in /usr/lib/ and copy them out to another directory. Then point Indy to that. Not quite sure that would be completely legal though. But at least a possibility to consider.

From searching the net it seems new versions of Mac OS do not include OpenSSL anymore bu only LibreSSL
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: Hansaplast on September 20, 2018, 12:18:42 pm
I also sent you a PM - but  I do not see it in my sent PMs list - but I have sen you an URL - if you can scan that using Synapse


Maybe Synapse added support for LibreSSL? But knowing what .dylib files you have in your usr/lib/ would help clarify that :)

(I rather use built-in openssl/libressl instead of shipping  those files myself)


Stupid forum filesize limitation made me type this again  >:(

I have a test project for you (about 600 Kb - which includes the Synapse files), so if you PM me your email address, I can email it to you.


As for the lib files - I could not find a single libressl file, just the regular openssl files.

macOS 10.14 (latest public beta on a clean VM):


libssl.0.9.7.dylib
libssl.0.9.8.dylib
libssl.35.dylib
libssl.43.dylib
libssl.44.dylib
libssl.dylib -> libssl.35.dylib


libcrypto.0.9.7.dylib
libcrypto.0.9.8.dylib
libcrypto.35.dylib
libcrypto.41.dylib
libcrypto.42.dylib
libcrypto.dylib -> libcrypto.35.dylib


macOS 10.13.6:


libssl.0.9.7.dylib
libssl.0.9.8.dylib
libssl.35.dylib
libssl.43.dylib
libssl.dylib -> libssl.35.dylib


libcrypto.0.9.7.dylib
libcrypto.0.9.8.dylib
libcrypto.35.dylib
libcrypto.41.dylib
libcrypto.dylib -> libcrypto.35.dylib
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: rvk on September 20, 2018, 12:35:14 pm
Synapse just uses whatever libssl.dylib is pointing to.
So libssl.dylib should point to something that is OpenSSL "compatible" (which I assume LibreSSL/libssl.35.dylib is).

Code: Pascal  [Select][+][-]
  1.   {$IFNDEF MSWINDOWS}
  2.     {$IFDEF DARWIN}
  3.     DLLSSLName: string = 'libssl.dylib';
  4.     DLLUtilName: string = 'libcrypto.dylib';
  5.     {$ELSE}
  6.      {$IFDEF OS2}
  7.       {$IFDEF OS2GCC}
  8.     DLLSSLName: string = 'kssl.dll';

Don't point to a specific version in your code, otherwise you'll end up changing it regularly.

B.T.W. What version is that  libssl.35.dylib? I see that LibreSSL only goes up to 2.8.0.
Maybe it's a version which doesn't support the new TLS.


Edit: I see LibreSSL 2.5.5 has libssl-43.dll for Windows. So I guess libssl-35 is quite old.
libssl-35 seems to be from the 2.2.x branche which is from 2015, maybe 2016.
These might really have problems with the new TLS.
Try to find a newer LibreSSL version for your OS and install that.
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: MISV on September 20, 2018, 12:48:54 pm
So looks like Indy and Synapse does the same. However, I don't think LibreSSL/OpenSSL are fully compaible.

...

I am trying to use what Mac OS already comes with. I would rather not require my users to install LibreSSL

My mac libssl.dylib points to libssl.35.dylib file

(however, not sure you can trust the version numbers, on my Mac 0.9.8 openssl was seemingly patched by Apple many times meaning the actual version is somewhat newer - long time ago I researched those details so I may be mistaken)
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: Hansaplast on September 20, 2018, 12:58:03 pm
Just did a test, seems libssl.35.dylib is indeed LibreSSL 2.2.7 (Synapse provides this info).
Obviously libssl.43 and .44 are newer versions, but macOS by default links to libssl.35.
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: rvk on September 20, 2018, 01:23:38 pm
I'm not sure if the newer/newest versions of LibreSSL are 1:1 compatible with OpenSSL.

But I do know that even OpenSSL from 2015 has problems connecting to some newer websites where only certain TLS versions are allowed.

After installing newer LibreSSL it should point to those versions and otherwise you could adjust the symlink yourself.
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: Remy Lebeau on September 20, 2018, 06:44:28 pm
Here are the following exceptions I get with HEAD/GET requests to multiple https sites

What are the DETAILS provided by those exceptions?  What do their ErrorCode and Message properties say exactly?

The fact that you are even getting this far means Indy was able to find and load libssl and libcrypto libraries into memory and locate enough of the "OpenSSL" functions it is interested in using.  Which means LibreSSL is at least API-compatible to OpenSSL as far as Indy cares about (which makes sense, since LibreSSL is forked from OpenSSL 1.0.1g).  Though that doesn't necessarily guarantee that LibreSSL exposes everything that Indy needs to communicate with modern websites.

Synapse just uses whatever libssl.dylib is pointing to.

So does Indy, by default.  And when it can't find that, it starts looking for version-specific files next (see the code in Indy's LoadSSLCryptoLibrary() function in the IdSSLOpenSSLHeaders unit).

Just did a test, seems libssl.35.dylib is indeed LibreSSL 2.2.7 (Synapse provides this info).

What does Indy's OpenSSLVersion() function in the IdSSLOpenSSL unit report?
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: rvk on September 20, 2018, 07:17:11 pm
OP also mentions "Since my own website exposes the problem...".

If it does work on other sites using SSL with LibreSSL, my guess is that 2.2.7 is too old for the TLS used at that site. Only way to fix that is updating LibreSSL.
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: MISV on September 20, 2018, 11:59:15 pm
OpenSSLVersion reports = OpenSSL 0.9.8zh 14 jan 2016
(so seems OpenSSL lib was loaded even though the symlink points to LibreSSL? Under all circumstances I am unsure if OpenSSL is still provided in clean macOS installs)

SSLOptions.SSLVersions := [sslvSSLv23];
SSLOptions.Method := sslvSSLv23;
= EIdOSSLUnderlyingCryptoError
- ErrorCode: 336032814
- Message: Error connecting with SSL. error 1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 protocol alert version

SSLOptions.SSLVersions := [sslvTLSv1_2];
SSLOptions.Method := sslvTLSv1_2;
= EIdOSSLUnderlyingCryptoError
- ErrorCode: 336151598
- Message: Error connecting with SSL. error 1409442E:SSL routines:SSL3_READ_BYTES:tlsv1 protocol alert version

SSLOptions.SSLVersions := [sslvTLSv1_1];
SSLOptions.Method := sslvTLSv1_1;
= EIdOSSLUnderlyingCryptoError
- ErrorCode: 336151598
- Message: Error connecting with SSL. error 1409442E:SSL routines:SSL3_READ_BYTES:tlsv1 protocol alert version


...

I guess next question is why the LibreSSL is not used when I can see it is pointed to. But if LibreSSL is not fully supported then it is of course much better loading OpenSSL! Anyhow the domain I am testing on is not using CloudFlare SSL (I found out they would fail with 0.9.8zh long time ago), but I guess a similar SSL solution then.

...

HansaPlast has told me he can load the same https website with Synapse, so I will be testing that
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: Hansaplast on September 21, 2018, 10:23:20 am

To get you started with Synapse; I just toss the entire Synapse library in my project directory (usually in a separate directory, but you'd have to add that directory to the paths in your project options).


Code: Pascal  [Select][+][-]
  1. uses ... ssl_openssl, httpsend, blcksock, synautil ... ;
  2.  
  3. ...
  4.  
  5.  
  6. procedure TForm1.Button1Click(Sender: TObject);
  7. var
  8.   HTTP: THTTPSend;
  9. begin
  10.   HTTP:=THTTPSend.Create;
  11.   HTTP.UserAgent:='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36';
  12.   HTTP.Headers.Clear;
  13.  
  14.  
  15.   try
  16.     HTTP.HTTPMethod('GET', 'https://www.example.com');
  17.  
  18.  
  19.     if (HTTP.ResultCode >= 100) and (HTTP.ResultCode<=299) then
  20.       Memo1.Lines.LoadFromStream(HTTP.Document);
  21.   except
  22.     showMessage('Something went wrong ...');
  23.   end;
  24.  
  25.  
  26.   showMessage('HTTP.Sock.LastError :'+IntToStr(HTTP.Sock.LastError)+' ; '+HTTP.Sock.LastErrorDesc+lineEnding+
  27.               'HTTP.Sock.SSL.LastError :'+IntToStr(HTTP.Sock.SSL.LastError)+' ; '+HTTP.Sock.SSL.LastErrorDesc+lineending+
  28.               'HTTP.ResultCode :'+IntToStr(HTTP.ResultCode)+LineEnding+
  29.               'Libname :'+HTTP.Sock.SSL.LibName+LineEnding+
  30.               'Libversion :'+HTTP.Sock.SSL.LibVersion);
  31.   HTTP.Free;
  32. end;
  33.  
  34.  
  35. ...
  36.  


I noticed with this site that without setting a useragent, it will return only the first Kb ... with UserAgent set, it will return everything.
I just added the ShowMessage at the end so I can see more detail about what is going on (or not).


Hope this helps.

Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: MISV on September 22, 2018, 01:14:05 pm
I can confirm the demo project. That works and Synapse uses LibreSSL which appears to work.

Furthermore it is also working with another https domain running a different I believe ssl solution - and there Synapse works as well (again reporting itself as using LibreSSL)

Indy with 0.9.8 fails with both domains - probably due to the age of 0.9.8

...


Preliminary Indy explanation and fix as far as I can tell - I may be missing some things:

IdSSLOpenSSLHeaders.pas
defines
SSLDLLVers : array [0..7] of string = ('.10','.1.0.2','.1.0.1','.1.0.0','0.9.9','.0.9.8','.0.9.7','0.9.6');
Note: I think '0.9.9' should probably have been '.0.9.9' in the above, but that is a lesser... bug?

IdGlobal
On non-Windows HackLoad is used to load ssl (which iterates through SSLDLLVers list) while On Windows HackLoad is not used since the dll is not versioned in filename.

IdSSLOpenSSLHeaders.pas
routine LoadSSLCyrptography
- on windows loads the nonversioned lib file directly
- else if symbolic links supported uses HackLoad - but passes en empty versions array (This does nothing? since HackLoad only does any actions on passed array items)
- else if above fails it expands SSLDLLVers with letters (a,b,c etc.) then uses HackLoad

This means on Mac - Indy never reads the non-versioned file name - i.e. libsll.dylib - and does not read the path shown when *right clicking* - "get info" - "original"

Instead it uses the versioned numbers... Which on my system would be 0.9.8 (which is also the library loaded as mentioned in any earlier post)

Anyhow... That is why Indy never tries to load LibreSSL binaries

...

Note: This also makes fine sense if one is not sure Indy is LibreSSL compatible (Would be fatal to simply then load the library in-case a definition is different/missing. Indy also seems grab pointers to all routines in the lib - and not as needed. I saw this by checking the source code TODO comments.)

...

--What i am working on now--

It seems to work adding ".35" to the list - but I will say that the crawl seems slower than usual... Still looking into it
 
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: Remy Lebeau on September 23, 2018, 08:33:10 pm
IdSSLOpenSSLHeaders.pas
defines
SSLDLLVers : array [0..7] of string = ('.10','.1.0.2','.1.0.1','.1.0.0','0.9.9','.0.9.8','.0.9.7','0.9.6');
Note: I think '0.9.9' should probably have been '.0.9.9' in the above, but that is a lesser... bug?

Fixed.

IdGlobal
On non-Windows HackLoad is used to load ssl (which iterates through SSLDLLVers list) while On Windows HackLoad is not used since the dll is not versioned in filename.

Correct.

IdSSLOpenSSLHeaders.pas
routine LoadSSLCyrptography
- on windows loads the nonversioned lib file directly
- else if symbolic links supported uses HackLoad - but passes en empty versions array (This does nothing? since HackLoad only does any actions on passed array items)
- else if above fails it expands SSLDLLVers with letters (a,b,c etc.) then uses HackLoad

This means on Mac - Indy never reads the non-versioned file name - i.e. libsll.dylib

That is not correct.  Look again at Indy's code more carefully.  By default, Indy first loads the non-versioned files, in case symlinks are present.  If that fails, then Indy resorts to loading individual versioned files (from newer to older):

Code: [Select]
var
  GIdLoadSymLinksFirst: Boolean = True;

...

  if GIdLoadSymLinksFirst then begin
    Result := {$IFNDEF KYLIXCOMPAT}HMODULE({$ENDIF}
      HackLoad(GIdOpenSSLPath + SSLCLIB_DLL_name, [])
      {$IFNDEF KYLIXCOMPAT}){$ENDIF};
  end;
  if Result = 0 then begin
    for i := Low(SSLDLLVers) to High(SSLDLLVers) do begin
      for j := Low(SSLDLLVersChar) to High(SSLDLLVersChar) do begin
        LLibVersions[j] := SSLDLLVers[i] + SSLDLLVersChar[j];
      end;
      Result := {$IFNDEF KYLIXCOMPAT}HMODULE({$ENDIF}
        HackLoad(GIdOpenSSLPath + SSLCLIB_DLL_name, LLibVersions)
        {$IFNDEF KYLIXCOMPAT}){$ENDIF};
      if Result <> 0 then begin
        Break;
      end;
    end;
  end;
  if (Result = 0) and (not GIdLoadSymLinksFirst) then begin
    Result := {$IFNDEF KYLIXCOMPAT}HMODULE({$ENDIF}
      HackLoad(GIdOpenSSLPath + SSLCLIB_DLL_name, [])
      {$IFNDEF KYLIXCOMPAT}){$ENDIF};
  end;

As you can see, an empty array is passed to HackLoad() only when loading non-versioned files.  Whether Indy loads symlinks first, or versioned files first, is controlled by the GIdLoadSymLinksFirst variable, which is True by default (and can be changed at runtime by calling IdOpenSSLSetLoadSymLinksFirst() before OpenSSL is loaded).

Indy also seems grab pointers to all routines in the lib - and not as needed. I saw this by checking the source code TODO comments.)

Yes, I've been wanting to change that for a long time.  For other external libraries that Indy uses, it loads functions on an as-needed basis.  But for OpenSSL, because of the sheer number of functions being imported, it probably made more sense to load them all up front (I didn't write that code, I only maintain it).

I will try test the LibreSSL binaries shipped with Mac OS now by inserting their file name versions in the SSLDLLVers const.

You should not need to do that, since Indy should be loading the non-versioned symlinks first, which would map to LibreSSL.  Make sure you are using an up-to-date snapshot of Indy.
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: MISV on September 23, 2018, 09:10:30 pm
Quote
var
  GIdLoadSymLinksFirst: Boolean = True;

...

  if GIdLoadSymLinksFirst then begin
    Result := {$IFNDEF KYLIXCOMPAT}HMODULE({$ENDIF}
      HackLoad(GIdOpenSSLPath + SSLCLIB_DLL_name, [])
      {$IFNDEF KYLIXCOMPAT}){$ENDIF};
  end;

Quote
By default, Indy first loads the non-versioned files, in case symlinks are present.

Just to make 100% sure: You are talking about the call to HackLoad in the above code where you are passing empty array []?

If yes, this is the code of HackLoad:

Code: Pascal  [Select][+][-]
  1. function HackLoad(const ALibName : String; const ALibVersions : array of String) : HMODULE;
  2. var
  3.   i : Integer;
  4. begin
  5.   Result := NilHandle;
  6.   for i := Low(ALibVersions) to High(ALibVersions) do
  7.   begin
  8.     {$IFDEF USE_SAFELOADLIBRARY}
  9.     Result := SafeLoadLibrary(HackLoadFileName(ALibName,ALibVersions[i]));
  10.     {$ELSE}
  11.     ...
  12.  

There is no code in HackLoad outside the loop... So if it is passed empty array then nothing happens? But I am tired, so if I missing something then...
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: Remy Lebeau on September 24, 2018, 02:58:07 am
Just to make 100% sure: You are talking about the call to HackLoad in the above code where you are passing empty array []?

Yes.

If yes, this is the code of HackLoad:
...
There is no code in HackLoad outside the loop... So if it is passed empty array then nothing happens? But I am tired, so if I missing something then...

No, you didn't missed anything.  It was indeed a logic bug in Indy's code, I didn't see it before.  The *INTENT* was to specify an empty array to skip the versioned files, but it was not actually doing that correctly.  I have now fixed it.
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: MISV on September 24, 2018, 11:24:54 am
I just downloaded and tried

1)
I am getting funky compile time error:

Quote
IdSSLOpenSSLHeaders.pas(19571,10) Error: function header doesn't match the previous declaration "GetCryptLibHandle:Int64;"
IdSSLOpenSSLHeaders.pas(18191,10) Error: Found declaration: GetCryptLibHandle:QWord;

But looking at code, both places use
Code: Pascal  [Select][+][-]
  1. function GetCryptLibHandle : HMODULE;
  2.  

Following IDE definition for HMODULE when used at line 18191:

Following IDE definition for HMODULE when used at line 19571:
- dynlibs.pas = TLibHandle
- dynlibs.pas = System.TLibHandle
- sysdlh.inc = PrInt
- systemh.inc = Int64

"Fixing" declaration at 18191 to using TLibHandle somehow works. I think this may be a Lazarus compiler bug...?

...

2)
It appears to work - been testing on two extremely troubled sites that do everything possible to make life miserably.
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: Remy Lebeau on September 24, 2018, 07:44:49 pm
I am getting funky compile time error:

Quote
IdSSLOpenSSLHeaders.pas(19571,10) Error: function header doesn't match the previous declaration "GetCryptLibHandle:Int64;"
IdSSLOpenSSLHeaders.pas(18191,10) Error: Found declaration: GetCryptLibHandle:QWord;

I didn't make any changes related to that, so if it is erroring now, it should have been erroring before, too.

This error implies that one of the units in the 'uses' clause of the 'implementation' section is likely redeclaring HMODULE.  The only non-Indy units in that clause are Classes and DynLibs.  So I'm guessing DynLibs is at fault.  But then, I would expect a similar error in the IdGlobal unit as well, as it defines HackLoad() in the 'interface' section to return an HMODULE (in which case, I guess I should remove the type-casts when IdSSLOpenSSLHeaders calls HackLoad()), and DynLibs is used in the 'implementation' section.

Update: Yup, turrns out that the DynLibs unit does define its own HModule type, as an alias for TLibHandle, which is an alias for PtrInt.  Whereas System.HMODULE is defined as an alias for PtrUInt instead.  Why isn't DynLibs using System.HMODULE instead of defining its own type?
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: Remy Lebeau on November 03, 2018, 01:27:33 am
This error implies that one of the units in the 'uses' clause of the 'implementation' section is likely redeclaring HMODULE.  The only non-Indy units in that clause are Classes and DynLibs.  So I'm guessing DynLibs is at fault.  But then, I would expect a similar error in the IdGlobal unit as well, as it defines HackLoad() in the 'interface' section to return an HMODULE (in which case, I guess I should remove the type-casts when IdSSLOpenSSLHeaders calls HackLoad()), and DynLibs is used in the 'implementation' section.

I have checked in some changes to now use THandle everywhere instead of HMODULE directly.  Does the code compile now?
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: Hansaplast on November 05, 2018, 04:03:09 pm

Not sure if this is helpful for your particular application, but OpenSSL (and such) appears to be on the list of things to drop for Apple, since Apple already will no longer allow to be used in sandboxed apps. Per excellent info sheet from Phil (https://macpgmr.github.io/ObjP/SandboxingAppsOnMac.html), under the section "Internet access":


Note that OpenSSL-based HTTP clients are not supported in sandboxed apps.
This includes Indy, Synapse and FPC's HTTP client. OpenSSL on Mac has been deprecated (https://developer.apple.com/library/content/documentation/Security/Conceptual/cryptoservices/SecureNetworkCommunicationAPIs/SecureNetworkCommunicationAPIs.html#//apple_ref/doc/uid/TP40011172-CH13-SW3) for a long time, so you should be using one of Apple's APIs instead.

If you need to make GET or POST requests, you can use the ns_url_request.pas unit from here (https://macpgmr.github.io/ObjP/nsunits-src.zip). It uses the Foundation framework's NSURLConnection (https://developer.apple.com/documentation/foundation/nsurlconnection?language=objc) class.


I had this issue using Synapse, and to my surprise, migrating to Phil's ns_url_request.pas was super easy.
The only thing that took me a few extra seconds was the format of the headers (not Phil's fault).
Synapse uses a notation like this: "Accept: application/json",
whereas the NSURLConnection functions seem to prefer this notation: "Accept=application/json" (notice the ": " versus "=" in the string).


As far as I can tell, this function works faster than the already fast Synapse, and the unit provided by Phil is really tiny compared to Indy and Synapse. Obviously quite a lot of functionality you'll find in Synapse and Indy, are not included. But it works really well for retrieving web content with or without GET or POST.
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: MISV on November 24, 2018, 04:44:09 pm
The problem for me using Phil's code is I need async https requests (since sync mode automatically follows redirects which I do not want) - and I have not been able to solve some freepascal syntax issues with delegates

https://forum.lazarus.freepascal.org/index.php/topic,37468.msg275526.html#msg275526 (https://forum.lazarus.freepascal.org/index.php/topic,37468.msg275526.html#msg275526)

If you can - great - I gave up on getting the syntax working :(
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: Hansaplast on November 24, 2018, 05:16:08 pm
@MISV;
Yeah that's a little over my head as well and I do feel your pain. I'm running into similar issues with other functions.
Converting remains a pain and Apple seems to be using different approaches for different frameworks.
For example; I managed to convert one of the callback functions in Disk Arbitration, and it works great. MacOS notifies my app when a disk gets ejected or inserted.
Similar, Cocoa has a function to let an application know that the theme has been changed. Whatever I tried, I can't get it to work.


My best bet would be Phil or Dmitry (skalogryz), but I'm sure they are super busy as well (I know Dmitry has been doing some amazing work on the Cocoa widgetset).
Wish I could help ...
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: MISV on December 01, 2018, 12:32:52 am
IdSSLOpenSSLHeaders.pas
defines
SSLDLLVers : array [0..7] of string = ('.10','.1.0.2','.1.0.1','.1.0.0','0.9.9','.0.9.8','.0.9.7','0.9.6');
Note: I think '0.9.9' should probably have been '.0.9.9' in the above, but that is a lesser... bug?

Fixed.

I just checked the what onlinepackagemanager addin in Lazarus downloads - and it downloads an old version - at least on Lazarus Trunk on Mojave

I inspected it by looking at the above - and the .0.9.9 / 0.9.9 issues is no fixed in the version it uses. (I assume that means it uses an old version)

Not sure where to report but... Now I have posted it here...
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: Remy Lebeau on December 01, 2018, 08:40:11 pm
I just checked the what onlinepackagemanager addin in Lazarus downloads - and it downloads an old version - at least on Lazarus Trunk on Mojave

I inspected it by looking at the above - and the .0.9.9 / 0.9.9 issues is no fixed in the version it uses. (I assume that means it uses an old version)

Yes, that would imply an older version is being used (double-check with Indy's IdVers.inc file. The current version is 10.6.2.5486).

The current declaration of the SSLDLLVers array looks like this:

Code: Pascal  [Select][+][-]
  1. SSLDLLVers : array [0..7] of string = ('.10','.1.0.2','.1.0.1','.1.0.0','.0.9.9','.0.9.8','.0.9.7','.0.9.6');

Not sure where to report but... Now I have posted it here...

Probably better to post in the Online Package Manager (http://forum.lazarus.freepascal.org/index.php/topic,34297.0.html) discussion thread instead.  There are several messages related to OPM's version of Indy.  Last I heard, OPM was setup to resync with Indy's SVN periodically, so maybe that resync is no longer working.
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: Remy Lebeau on December 02, 2018, 02:54:42 am
Update: Yup, turrns out that the DynLibs unit does define its own HModule type, as an alias for TLibHandle, which is an alias for PtrInt.  Whereas System.HMODULE is defined as an alias for PtrUInt instead.  Why isn't DynLibs using System.HMODULE instead of defining its own type?

I have checked in some further changes to now use TLibHandle on FreePascal, and THandle on Delphi.  Does the code compile now?
Title: Re: Mac Os switched to LibreSSL from OpenSSL - Indy no longer working for me
Post by: MISV on December 02, 2018, 11:35:06 pm
Yes - it compiles - and also seems to resolve the issue I have reported in the other thread - thank you :)
TinyPortal © 2005-2018