Including OpenSSL in a cross-platform application

[maurobio]

Hi, @rvk!

I am getting more and more confused! 

What issues are you talking about exactly. Because I think those were never really discussed here.

  Well, what about all the problems which you have pointed out yourself??? Or my initial problem, that was not having being able to run the application on any system which has not an installation of Lazarus???

Do you mean installing OpenSSL for https access on those systems?

To begin with, I am not even sure that OpenSSL was the only problem with the application.

Or do you have other problems?

I still have the same problem I had from the start - namely, systems which do not have an installation of Lazarus seem unable to execute any search because of persistent "access violation" errors. But right now, two users complained about "access violation" errors even after they installed Lazarus on their systems (one running Windows 11, the other running Mint 21)!!!! 

I just tried Especies (compiled with trunk) on a clean Mint 21 system (default installed OpenSSL 3.0.2) and it worked the same as on Windows.

As I have told since the start of  this thread, I have the application working on Windows 10, Mint 19, and Lubuntu 18 without any problems! But all these systems have Lazarus (2.0.12) installed!

One other point: you and other members of the forum have advised me to include exception checking into my code. But although I do not claim that such checks are complete, they are mostly there! See for example the 'Snippet' method in the class TGBIFSearch:

Code: Pascal  [Select][+][-]

1. constructor TGBIFSearch.Create;
2. begin
3.   GBIF_URL := 'http://api.gbif.org/v1';
4. end;
5.  
6. procedure TGBIFSearch.Search(const searchStr: string; var key: integer;
7.   var scientificname, authorship, status, valid_name, kingdom, phylum,
8.   classe, order, family: string);
9. var
10.   JsonData: TJsonData;
11.   Client: TFPHttpClient;
12. begin
13.   try
14.     Client := TFPHttpClient.Create(nil);
15.     try
16.       JsonData := GetJson(Client.Get(GBIF_URL + '/species/?name=' +
17.         StringReplace(searchStr, ' ', '%20', [rfReplaceAll])));
18.       key := JsonData.FindPath('results[0].key').AsInteger;
19.       scientificname := JsonData.FindPath('results[0].canonicalName').AsString;
20.       authorship := JsonData.FindPath('results[0].authorship').AsString;
21.       status := JsonData.FindPath('results[0].taxonomicStatus').AsString;
22.       status := LowerCase(StringReplace(status, '_', ' ', [rfReplaceAll]));
23.       if status <> 'accepted' then
24.         valid_name := JsonData.FindPath('results[0].species').AsString;
25.       kingdom := JsonData.FindPath('results[0].kingdom').AsString;
26.       phylum := JsonData.FindPath('results[0].phylum').AsString;
27.       classe := JsonData.FindPath('results[0].class').AsString;
28.       order := JsonData.FindPath('results[0].order').AsString;
29.       family := JsonData.Findpath('results[0].family').AsString;
30.     except
31.       key := 0;
32.       scientificname := '';
33.       authorship := '';
34.       status := '';
35.       valid_name := '';
36.       kingdom := '';
37.       phylum := '';
38.       classe := '';
39.       order := '';
40.       family := '';
41.     end;
42.   finally
43.     JsonData.Free;
44.     Client.Free;
45.   end;
46. end;

       

This code may not be exactly beautiful, but it should take care of errors in the returning jason data. Or shouldn't it?

All the other search classes used by the application also have similar exception treatment (even if it can be improved here or there).

I am at a loss... 

With warmest regards,

[rvk]

Well, what about all the problems which you have pointed out yourself??? Or my initial problem, that was not having being able to run the application on any system which has not an installation of Lazarus???

I'm also puzzled by that. I tried a compiled version of your program om a fresh Mint 21 (without Lazarus or FPC) and it worked fine (with the openssl 3 fix included because I compiled with trunk and my mint 21 has OpenSSL 3).

Do you mean installing OpenSSL for https access on those systems?

To begin with, I am not even sure that OpenSSL was the only problem with the application.

Well, it was for me
I didn't have any other problems on a fresh system (other than crashing when entering something invalid).
So I don't see any other problem than OpenSSL.

Or do you have other problems?

I still have the same problem I had from the start - namely, systems which do not have an installation of Lazarus seem unable to execute any search because of persistent "access violation" errors. But right now, two users complained about "access violation" errors even after they installed Lazarus on their systems (one running Windows 11, the other running Mint 21)!!!! 

The access violation is due to the fact OpenSSL 3 is installed on most systems (at least it was for me). And your Lazarus 2.0.12/FPC 3.2.0 doesn't do OpenSSL 3.
That's really the only problem I could find.

As I have told since the start of  this thread, I have the application working on Windows 10, Mint 19, and Lubuntu 18 without any problems! But all these systems have Lazarus (2.0.12) installed!

And they also probably have OpenSSL 1.0.2 installed (or lower).

You can check that with openssl version. What is your version on those systems? And what is it on the problem systems?

One other point: you and other members of the forum have advised me to include exception checking into my code. But although I do not claim that such checks are complete, they are mostly there! See for example the 'Snippet' method in the class TGBIFSearch:

Sorry, my apologies. You are correct. I'm not sure why my version crashed with invalid data, but now it does seem to work correctly.
(I don't know how I could overlook your except structure in GBIFSearch  )

This code may not be exactly beautiful, but it should take care of errors in the returning jason data. Or shouldn't it?

If it works and is stable, there's nothing wrong with that

[maurobio]

Hi, @rvk!

Now, I breath a sign of relief!  As I (finally) do understand it, the problem is really with OpenSSL (other than comparatively minor issues with some still unhandled exceptions).

So, I ask your advice on how to best proceed regarding OpenSSL. Under Windows, should it suffice to include it in the distribution package? In this case, what should be the correct version to be included (1.x or 3.x)? Under Linux (whatever distro) should OpenSSL be added as a dependence in the Debian package used to distribute the application? The matter of the Lazarus version versus the OpenSSL version is still not clear to me.

Of course, I will also check carefully the exception handling and add it at some points where it is clearly deficient.

Thank you very much!

With warmest regards,

[rvk]

Under Windows, should it suffice to include it in the distribution package?

I assume security isn't really a big deal for communicating with that server (you don't transfer sensitive information).
So you could just use a fairly recent version which is still supported.
I don't think the 1.1 family is supported in FPC 3.2.0 so you would need the latest 1.0.2.
(I think that's 1.0.2zi. But 1.0.2u from fulgan would be fine too.)

Under Linux (whatever distro) should OpenSSL be added as a dependence in the Debian package used to distribute the application?

Yes, and your dependency would be the smallest version you support, which is 1.0.2 in that case.

But... you need to support higher versions too because a dependency won't downgrade a higher version.
(it could if you set it at exactly that level but that could just prevent your program from installing.)

For example:

Code: [Select]

Depends: openssl >= 1.0.2kI'm not an Linux expert so I'm not sure how that's done exactly.

But you will still need to support OpenSSL 3 if it's installed and that's the hard part.
You would need to recompile FPC 3.2.0 (as stated in my previous post).

There might be a possibility to use 1.0.2 on newer systems, but I haven't looked into that.
(for example search for multiple openssl linux)
As far as I could see the instructions for that are downloading openssl 1.0.2 source and compiling with install to custom directory.
After that your program would need to adjust the search path to let it find those older libraries.
I'm not sure if that's the best way to go.

Maybe someone with more Linux knowledge can give some more info about this?

[maurobio]

Hi, @rvk!

OK, I downloaded OpenSSL 1.0.2u for Windows from here: https://indy.fulgan.com/SSL/

The distribution package is a zip archive containing the following files:

Hashinfo.txt
ReadMe.txt
OpenSSLLicense.txt
openssl.exe
libeay32.dll
ssleay32.dll

Should it be enough to include just the two DDL's in my installation package for Windows?

As of Linux, if I include OpenSSL 2.0.1 in the list of dependencies, will that version be used instead of OpenSSL 3.x (even if the latter is also installed)?

Alternatively, should it be better to upgrade Lazarus (both in Windows and Linux) to the latest (but not necessarily the trunk) version if this version offers consistent support for OpenSSL 3 in all platforms?

Thanks a lot!

With warmest regards,

[rvk]

BTW. You didn't handle all exception. Yes, you have except almost everywhere but if it was everywhere you wouldn't get that access violation (because that's also an except).

If you handle the except in Snippet with a Showmessage you get image below this post. But you return Result := '' with that exception.
And further on in the program, it can't handle that Result = '' and gives an access violation. I'm not sure where but for that you would need to debug (with setting result to '' for example).

Should it be enough to include just the two DDL's in my installation package for Windows?

Yes, that should be enough.

As of Linux, if I include OpenSSL 2.0.1 in the list of dependencies, will that version be used instead of OpenSSL 3.x (even if the latter is also installed)?

(you mean 1.0.2 )
No, if you include that as >= dependencies it makes sure that's the minimum version installed.
So with >= 1.0.2 and 3 being already installed it will do nothing. But your program would need to handle the 3 version (because there is no 1.0.2).

But if you define it as = 1.0.2 then it needs to be that version exactly.
I'm not sure if it will try to downgrade to 1.0.2 but I can guarantee you that you'll end up in a dependency hell.
Because openssl has a dependency on libc6 and libcrypto. So they also need to be downgraded.
And lots of other packages are again dependend on libc6 and openssl 3 so you end up in a loop.

Alternatively, should it be better to upgrade Lazarus (both in Windows and Linux) to the latest (but not necessarily the trunk) version if this version offers consistent support for OpenSSL 3 in all platforms?

I think I saw the '.1.1' in the array for FPC 3.2.0 so OpenSSL 1.1 should be supported.
But if you want OpenSSL 3 support... I think it's only in trunk (because I have a Lazarus 2.2.6 here with FPC 3.2.2 which only has .1.1).

BTW. That was probably also the reason Synapse came up. It can use the openssl 3 without completely recompiling FPC.
But then you would need to re-program all your communications.

Does anybody know how to "hack" the openssl.pas for loading OpenSSL 3 without recompiling FPC????
For changing the constant for DLLVersions before InitSSLInterface?

I did this now and it seems to go a step further but still crashes with the access violation.
But maybe if you build in the exception check at the right places with a showmessage of what kind of exception (instead of just return), you can find the problem.

This somewhere before the first call to a https get.

Code: Pascal  [Select][+][-]

1. openssl.DLLVersions[1] := '.3';

BTW, this is a hack because it replaces the .1.1. You might want to insert it on position 2 instead of replacing it.

EDIT: That hack does work but there is some trouble/instability further on (I think in Wikimedia Commons).

[rvk]

This will probably the safest option:

Code: Pascal  [Select][+][-]

1.  openssl.DLLVersions[Length(openssl.DLLVersions) - 1] := '.3';

It will replace support for OpenSSL 0.9.1 into OpenSSL 3 

(That version should exists anymore anyway.)

Works on my Mint 21 system.
On my development system I still get some problems but there I'm running in the IDE.

[maurobio]

Hi, @rvk!

At lunch right now but I could not make you wait for my reply! That's what mobile devices are for! 

As of Wikipedia, I will entirely replace its redirection handling with that provided with Lazarus networking library, which is simpler and probably safer.

I will try the Windows version first and then Linux. Will report my results later.

Thank you very much!

With warmest regards,

[rvk]

As of Wikipedia, I will entirely replace its redirection handling with that provided with Lazarus networking library, which is simpler and probably safer.

O, I thought that WIKIPEDIA_REDIRECT_URL was for handling other kinds of redirections too.

For when you make a common typo or redirect to a similar name?
But if that's also handled though a 301 then it's not needed.

[rvk]

I forgot to include the image from when you show the exception instead of just silently returning.

You need to determine if the exception is expected or unexpected.
Expected exception can be handled silently (because you handle them correctly).

Unexpected exceptions (like no OpenSSL installed) would benefit from a more clear message.

 

[paweld]

Hi @maurobio,   
I made several changes to your application, including replacing network components with synapse, moving data retrieval to a thread, fixing memory leaks: https://github.com/paweld/especies     
It should work on Linux, you just need to install libssl-dev.     
Required components: synapse (trunk version) and bgrabitmap     
     
Tested on lazarus trunk and fpc 3.2-fixes

[maurobio]

Hi, @rvk!

O, I thought that WIKIPEDIA_REDIRECT_URL was for handling other kinds of redirections too.

As far as I know, the Wikipedia redirect mechanism is for redirecting pages; in the case of my application, it should direct the Latin scientific name Loxodonta africana to the common English name "African bush elephant". I cannot say if this redirecting mechanism also works with pages which don't exist.

Unexpected exceptions (like no OpenSSL installed) would benefit from a more clear message.

Fine, but where exactly in the code have you put the exception handler which generated that message?

With warmest regards,

[rvk]

Fine, but where exactly in the code have you put the exception handler which generated that message?

I put that in the except in the function Search (where the first openssl call tales place).
You have a result := '' and I added a showmessage.

You could also set a variable and handle it in a higher level.

You should follow the result because a result of '' will give an access violation further on in the program.

[maurobio]

Hi, @paweld!!!

I made several changes to your application, including replacing network components with synapse, moving data retrieval to a thread, fixing memory leaks:

Man, what an incredible feat!! Frankly, I am at a loss of words, in my poor pidgin English, even in my native Portuguese, to praise and tank you and @rvk enough for your invaluable help in making true Professor's E. O. Wilson dream of "one webpage for each species"!

You fully deserve a place among the true heroes of biodiversity conservation (who are not those who just speak but do nothing to save the species which share this planet with us and which have as many right to live here as ourselves). As also does Florian Klaempf and all other developers of this absolutely great tool which is Free Pascal/Lazarus!

I will try your Synapse-enhanced version ASAP.

With warmest regards, muitas felicidades!

[toby]

geez paweld that was amazing
https://github.com/paweld/especies     

you and rvk are truly amazing

maurobio better hope that boss doesn't know about this forum