Recent

Author Topic: Including OpenSSL in a cross-platform application  (Read 13581 times)

TRon

  • Hero Member
  • *****
  • Posts: 4377
Re: Including OpenSSL in a cross-platform application
« Reply #15 on: September 09, 2023, 10:46:27 pm »
My knowledge of Linux is not very good, but I know that it places great emphasis on security, so for a long time most distributions (as not all) use openssl v3.
Which is also why you should not distribute ssl libraries (or any other library for that matter) with a Linux application.  For TS: do not take my word for it. Instead go over to a Linux forum and ask the same question of how you can distribute ssl libraries with your Linux application and hopefully you are able to get out of that discussion alive and in one piece.
Today is tomorrow's yesterday.

maurobio

  • Hero Member
  • *****
  • Posts: 640
  • Ecology is everything.
    • GitHub
Re: Including OpenSSL in a cross-platform application
« Reply #16 on: September 09, 2023, 10:51:21 pm »
Hi, @paweld!

Thank you very much for your kind and comprehensive answer.

I will consider carefully the points you raised, especially the use of Synapse (as previously suggested by @toby). But I would like to know why is Synapse "better" and what makes it so, in comparison to the regular libraries provided with FPC. Does it no require OpenSSL?

Also, I should clarify that in all my systems which have FPC/Lazarus installed, the applications runs as expected - the trouble arise in other machines, which do not have an installation of FPC/Lazarus (and therefore somehow lack one or more of the required libraries - I am not even sure of what they are really, because on at least a Linux installation which does have libssl installed, the application does not work).

With best regards,
UCSD Pascal / Burroughs 6700 / Master Control Program
Delphi 7.0 Personal Edition
Lazarus 3.8 - FPC 3.2.2 on GNU/Linux Mint 19.1/20.3, Windows XP SP3, Windows 7 Professional, Windows 10 Home

maurobio

  • Hero Member
  • *****
  • Posts: 640
  • Ecology is everything.
    • GitHub
Re: Including OpenSSL in a cross-platform application
« Reply #17 on: September 09, 2023, 10:54:13 pm »
Hi, @TRon!

Thank you very much for this suggestion, I will look into Linux forums in the hope of finding a solution to that.

with warmest regards,
UCSD Pascal / Burroughs 6700 / Master Control Program
Delphi 7.0 Personal Edition
Lazarus 3.8 - FPC 3.2.2 on GNU/Linux Mint 19.1/20.3, Windows XP SP3, Windows 7 Professional, Windows 10 Home

TRon

  • Hero Member
  • *****
  • Posts: 4377
Re: Including OpenSSL in a cross-platform application
« Reply #18 on: September 09, 2023, 11:04:03 pm »
@maurobio:
Before actually doing so (I would not recommend it especially not on a Linux developers forum: they can be very harsh) try read this or do a search on/for (Linux) dependencies and try to understand how you are suppose to solve your issue on Linux.


I am not even sure of what they are really, because on at least a Linux installation which does have libssl installed, the application does not work).
To make another (probably obvious) suggestion: try and do as much error checking as possible in your code, (double) checking each and every return value for validity and let your code act accordingly (for example display a message).

Also try to run your application from a terminal (even if it is a GUI application) because especially with ssl there are some (FPC) error messages that are only displayed in the terminal when there is something wrong (and the exact error message could be very helpful in those cases).

For debugging your applications under Linux that have dependency issues you can also use something like strace.
« Last Edit: September 09, 2023, 11:17:22 pm by TRon »
Today is tomorrow's yesterday.

toby

  • Sr. Member
  • ****
  • Posts: 270
Re: Including OpenSSL in a cross-platform application
« Reply #19 on: September 11, 2023, 08:54:54 pm »


paweld : there is a 4th option :) - using curl with

tprocess.execute(
or
sysutils.executeprocess(

this would eliminate all his openssl problems and when openssl version 17.0.0.1 comes out with their quantum computer version he can wait until curl implements it - with no worries

--

maurobio :

in case you are still looking at your posting :

the trunk/web code in https://github.com/maurobio/especies
compiled on both my linux 32bit and 64bit systems with fpc-3.2.2 on them and my openssl version

openssl version
OpenSSL 3.0.5 5 Jul 2022 (Library: OpenSSL 3.0.5 5 Jul 2022)

but gave different access violation errors when run

-

trunk/web   32bit linux

<html>
<title>e-Species</title>
<body>
<title>e-Species</title>
<h3>Error filling out form</h3>
<p>Please enter a binomial specific epithet into the text box.</a>
<p>Remember: Only <a href="http://en.wikipedia.org/wiki/Species">species</a> are true natural entities!</p>
<a href="../especies/index.htm">Go back to the query form</a>
An unhandled exception occurred at $08060F31:
EAccessViolation: Access violation
  $08060F31
  $0804C0F3  main,  line 641 of especies.pas

GBIFSearch.Search(queryStr, key, scientificname, authorship, status, valid_name, kingdom, phylum, classe, order, family);

---

trunk/web   64bit linux

<html>
<title>e-Species</title>
<body>
<title>e-Species</title>
<h3>Error filling out form</h3>
<p>Please enter a binomial specific epithet into the text box.</a>
<p>Remember: Only <a href="http://en.wikipedia.org/wiki/Species">species</a> are true natural entities!</p>
<a href="../especies/index.htm">Go back to the query form</a>
<h2><i></i>  </h2>
<h3>Classification from CoL</h3>
No names found
<h3>Text tags</h3>
<span style='display:inline;border:1px solid blue; padding:1px;margin:2px;line-height:22px;background-color:rgb(181,213,255);'>You&nbsp;must&nbsp;supply&nbsp;content&nbsp;by&nbsp;using&nbsp;the&nbsp;&text&nbsp;or&nbsp;&context&nbsp;querystring&nbsp;parameters,&nbsp;or&nbsp;by&nbsp;passing&nbsp;a&nbsp;URL,&nbsp;e.g.&nbsp;&url=example.org/... </span>
<h3>Wikipedia</h3>
No article title matches
<h3>Genomics from NCBI</h3>
TaxId: <a href="http://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi?mode=Info&id=0">0</a>&nbsp;No items found for <i></i> <ul type="circle">
</ul>
<h3>Map from GBIF</h3>
No species found
<h3>Images from Wikimedia Commons</h3>
An unhandled exception occurred at $0000000000408D1C:
EAccessViolation: Access violation
  $0000000000408D1C
  $000000000040668D  main,  line 744 of especies.pas

if imgs.Count = 0 then WriteLn('No images found')
 

---

i'm surprised that you invented a fight with rvk - probably the only person on this forum who was helping you with your actual code
having him interested in helping you with your code was a wonderful thing for you - believe me

---

so :

did you try the fpc dev trunk? (make sure you don't overwrite your current installations of fpc and lazarus)

after looking at your code - it seems you want to download all of wikipedia ... and then parse what you get for your data
it looks to me that all you are changing is the url of what you are downloading

this can be easily redone using curl from an array/tstringlist of urls - redoing your code to have curl get everything would be a real positive

where are you putting the resulting data? a jsonfile ? a database? a text file? i didn't see where it was going once you get it

---

maurobio

  • Hero Member
  • *****
  • Posts: 640
  • Ecology is everything.
    • GitHub
Re: Including OpenSSL in a cross-platform application
« Reply #20 on: September 11, 2023, 09:46:07 pm »
Hi, @toby!

Thanks a lot for your detailed post and comments.

But (as there is always a damned "but"!  >:() your findings just contributed to raise my incredulity and blood pressure...

My application has been developed using Lazarus 2.0.12, where it compiles and runs quite well, under Windows 10, Mint Linux 19 (32-bit) and  Lubuntu Linux 18 (64-bit). I am appalled to learn that it raises these strange errors when run in the trunk version of the IDE. This is just wasteful and exasperating!

If you look more closely to the code of my application, you will see that it does not attempt to "download all of Wikipedia" - this would be patently absurd! As of Wikipedia, it just takes the first paragraph of text of the page referring to the searched species. As of the rest of the searched data, what the application does with them is to build a simple HTML page joining the pieces of information fetched from each source (GBIF, NCBI, Wikipedia, etc.) This is, I presume, what some used to call a "mashup". It implements the concept of "one web page for each living biological species", first presented by Professor Edward O. Wilson of Harvard in his article "The encyclopedia of life" (Trends in Ecology and Evolution, vol. 18, no. 2, Feb. 2003, pp. 77-80):

Quote
Imagine an electronic page for each species of organism on Earth, available everywhere by single access on command. The page contains the scientific name of the species, a pictorial or genomic presentation of the primary type specimen on which its name is based, and a summary of its diagnostic traits. The page opens out directly or by linking to other data bases, such as ARKive, Ecoport, GenBank and MORPHOBANK. It comprises a summary of everything known about the species’ genome, proteome, geographical distribution, phylogenetic position, habitat, ecological relationships and, not least, its practical importance for humanity.

In fact, my application has been initially developed as a simple CGI application (https://github.com/maurobio/especies/tree/master/web), running on a webserver, but because of the issues with OpenSSL which did not allow it to run on a free hosting service, I decided to try and re-develop it a desktop application.

That said, I did not "invent" a fight with mr. dvk (or anyone else for that matter). But I have sincerely tried to be polite and respectful with everyone here on this forum (of which I have been a member for more than five years now). I therefore cannot tolerate being treated with debauchery and disrespect. Anyone really willing to help should proceed just like you did: show me the errors and, objective suggestions to fix them.

Thank you very much.

With warmest regards,
UCSD Pascal / Burroughs 6700 / Master Control Program
Delphi 7.0 Personal Edition
Lazarus 3.8 - FPC 3.2.2 on GNU/Linux Mint 19.1/20.3, Windows XP SP3, Windows 7 Professional, Windows 10 Home

rvk

  • Hero Member
  • *****
  • Posts: 6799
Re: Including OpenSSL in a cross-platform application
« Reply #21 on: September 11, 2023, 10:31:10 pm »
No fight here. Also no disrespect.
But when someone wants to help, you sometimes need to explain things (like difference 32/64 bit, especially when that user suggest copying the dlls to the Windows directory which can lead to a crippled system). I can't help it if that person already know this (and could just say so). I wasn't scalding anyone. And I was not calling anyone dumb.

Only the first sentence of my post could be a bit harsh ("lovely description of an error message").
But that was more my impatiens. Sorry for that.
Sometimes a user just post "I have an error" without saying what error.

Anyway, I did take the time to look briefly at the program last week and found that there is insufficient error handling for when there is a (correct) json retrieved without the expected values. Reading (accessing) those values/array index (while they are not there) will definitely lead to a crash in TGBIFSearch.Search.

Since I couldn't find a description as to how to use the program (what to enter) I didn't look any further.

maurobio

  • Hero Member
  • *****
  • Posts: 640
  • Ecology is everything.
    • GitHub
Re: Including OpenSSL in a cross-platform application
« Reply #22 on: September 11, 2023, 11:25:48 pm »
Hi, @rvk!

Quote
Only the first sentence of my post could be a bit harsh ("lovely description of an error message").
But that was more my impatiens. Sorry for that.

Your apologies are much welcome, and fully accepted! Please also accepted mine. BTW, the reason I have not provided the full error message issued by Windows was that it is in Brazilian Portuguese (that's the language of my Windows installation), and I presume not many people can undertand that arcane idiom. I was just out if patience to translate it into my pidgin English.

Notice that we were "fighting" around a Windows issue!  :( Nobody deserves that!

Quote
Sometimes a user just post "I have an error" without saying what error.

I know what you mean, and do admit that this is a most irritating issue (like that brother OF it: "something does not work").

Thanks a lot for pointing out the problem with json in my code! I will take a full review of it, in an attempt to finding that (and hopefully other) such errors.

You can test he code using the "Vicia faba" scientific name (that is the fava bean, very popular here down under  :D).

Thank you very much!

With warmest regards,
« Last Edit: September 11, 2023, 11:56:01 pm by maurobio »
UCSD Pascal / Burroughs 6700 / Master Control Program
Delphi 7.0 Personal Edition
Lazarus 3.8 - FPC 3.2.2 on GNU/Linux Mint 19.1/20.3, Windows XP SP3, Windows 7 Professional, Windows 10 Home

rvk

  • Hero Member
  • *****
  • Posts: 6799
Re: Including OpenSSL in a cross-platform application
« Reply #23 on: September 11, 2023, 11:38:36 pm »
All good.

You can test he code using the "Vicia faba" scientific name (that is the fava bean, very popular hear down under  :D).
I'll try that tomorrow (it's late here).
I tried it with some kind of whale I found (which gave a json without some of the expected value).
But I'll try some and give you my findings.

(I did try in the IDE and I'm not sure anymore how the exception handling was set but I'll see tomorrow. I'll also try on Linux.)

maurobio

  • Hero Member
  • *****
  • Posts: 640
  • Ecology is everything.
    • GitHub
Re: Including OpenSSL in a cross-platform application
« Reply #24 on: September 11, 2023, 11:56:31 pm »
Hi, @rvk!

Thank you very much!

With best regards,
UCSD Pascal / Burroughs 6700 / Master Control Program
Delphi 7.0 Personal Edition
Lazarus 3.8 - FPC 3.2.2 on GNU/Linux Mint 19.1/20.3, Windows XP SP3, Windows 7 Professional, Windows 10 Home

rvk

  • Hero Member
  • *****
  • Posts: 6799
Re: Including OpenSSL in a cross-platform application
« Reply #25 on: September 12, 2023, 11:35:28 am »
Ok, my preliminary findings (for now on Windows).

I still have some remarks below about the error handling but it works correctly on Windows with the latest trunk and an older OpenSSL 1.0.2u.
I can enter Vicia faba and it does all the communications and presents a page (see below). Although the image is a exclamation mark.

Some remarks.
I had a hard time in the beginning because I kept getting an access violation. It seems that the wiki page kept returning a 301 redirection on the final api call in Snippet. That isn't handled and results in a crash.

I tried the URL in Curl and that worked (even with adding --http1.1 --tlsv1.2 --tls-max 1.2).
I finally noticed I used Vicia faba on Curl but types vicia faba in the Windows app.
It seems case sensitive  %)
(Even though you do the WIKIPEDIA_REDIRECT_URL before WIKIPEDIA_URL, that URL doesn't do the redirection for case issues)

And when passing vicia faba to the final api, you get a 301 redirect (which isn't handled).
Adding a Client.AllowRedirect := true; below every TFPHttpClient.Create would fix that (you didn't add it everywhere).

But... it's always wise to include some more exception handling. A website could be down and you want a graceful error then. Now it crashes in that case (even with the redirect line).

Then for the other issue... invalid input. If I just type in a b as input (yeah, I know :) ), it crashes in TGBIFSearch.Search.
That's because you expect ALL the keys to be there in the json.
Here you should also add some error handling (is the key really in the json etc).

Next... you do GetJson(Client.Get()). So you expect that Client.Get() always returns valid json. What if it doesn't?
Either an exception handler or check for valid json could be wise here (GetJson gives an exception when it's passed non-valid json).

Finally... when using Physeter macrocephalus I got some problems too but they are handled by the exception handler in TPubMedSearch.Search.
Although I don't really understand why Client.Get sometimes failed for that request in fphttpclient, the exception is handled.

Sidenote:
In TPubMedSearch.Search I also see this (not sure if that's on other places too):
Code: Pascal  [Select][+][-]
  1. MemStrm := TMemoryStream.Create;
  2. XmlData := Client.Get(PUBMED_URL
  3. // some code
  4. MemStrm.Free;
If the Client.Get gives an exception, the MemStrm isn't released, so there is a memory leak.
Always do the Create and then a try/finally to release the created item (that one probably got by you because you do at other places :) )

So on Windows it works correctly with just any older openssl dlls in the exe directory.
For Linux it should work too (I'll test that when I get some time again, not sure if that's today though).

The only issue on Linux could be the newer 1.1, 3.0 or 3.1 in the Linux distro.
But as said before... openssl should be a requirement for the package itself so it installs automatically from distro.
(How are you distributing? Via .deb? or something else?)

Trunk will probably handle the newer 1.1/3 version but did you mention you wanted to keep this on Laz 2.0?
In that case you would need to patch the openssl unit to load 1.1 or 3 version of openssl.

Switching to release Lazarus (maybe with 'fixes') would be probably easier in that case.


maurobio

  • Hero Member
  • *****
  • Posts: 640
  • Ecology is everything.
    • GitHub
Re: Including OpenSSL in a cross-platform application
« Reply #26 on: September 12, 2023, 11:41:11 pm »
Hi, @rvk!

Thank you VERY MUCH for your comprehensive analysis of the many problems with my code. As a matter of fact, I was aware that my code has many loose knots, but your analysis found several more I have overlooked.

Wikipedia uses those really dreadful redirections, but they provide means to handle them using the Wikipedia API itself - however, as your analysis have shown, it seems it is not performing as expected. The FPC/Lazarus networking units also have a mean of handle these redirections; in the end I mixed up both and the results was that messy code!

I found the concept of using Curl really interesting; is there detailed documentation on how to use it? If the work required to convert my code to using it, I would consider change to Curl. What do you think? Would this be viable.

Anyway, I will proceed with a through scanning of my code to find out and fix the loose points.

More soon...

With warmest regards,
UCSD Pascal / Burroughs 6700 / Master Control Program
Delphi 7.0 Personal Edition
Lazarus 3.8 - FPC 3.2.2 on GNU/Linux Mint 19.1/20.3, Windows XP SP3, Windows 7 Professional, Windows 10 Home

rvk

  • Hero Member
  • *****
  • Posts: 6799
Re: Including OpenSSL in a cross-platform application
« Reply #27 on: September 12, 2023, 11:57:16 pm »
I found the concept of using Curl really interesting; is there detailed documentation on how to use it? If the work required to convert my code to using it, I would consider change to Curl. What do you think? Would this be viable.
What would be the upside to the requirement of curl being installed over the requirement of openssh being installed? Only upside would be that you don't need to update your code to use openssl 3.

But curl is still a requirement because it's not standard installed on systems.

Did you want to update your code to latest Lazarus? Then I don't see an upside to using curl package. You would also need to call curl with exec calls. Not my preference.

Second downside.. you would need to use intermediate files for output (or use akward stdin/stdout stuff). With tfphttpclient you can work strictly in memory.

I'll check out Linux tomorrow.
What where your exact OS versions (and did you want to upgrade laz)?
Those questions are important for investigating what is possible.

maurobio

  • Hero Member
  • *****
  • Posts: 640
  • Ecology is everything.
    • GitHub
Re: Including OpenSSL in a cross-platform application
« Reply #28 on: September 13, 2023, 01:20:04 am »
Hi, @rvk!

Quote
What where your exact OS versions (and did you want to upgrade laz)?

For the time being, I am using Lazarus 2.0.12 (32-bits) under Windows 10 (64-bits), Mint Linux 19.1 (32-bits) and Lubuntu 18.04 (64-bits). Frankly, I would not like to be obliged to upgrade these systems now, for lack of time and fear of breaking what is already working.  I really never imagined that these issues could also be related to the FPC/Lazarus version! %)

Quote
you would need to use intermediate files for output (or use akward stdin/stdout stuff). With tfphttpclient you can work strictly in memory.

In earlier versions I have used temporary files for output, but dropped them in the current versions because memory files seemed to be more efficient. I don't understand why should temporary files be more efficient (and they may need administrative privileges).

With warmest regards,
UCSD Pascal / Burroughs 6700 / Master Control Program
Delphi 7.0 Personal Edition
Lazarus 3.8 - FPC 3.2.2 on GNU/Linux Mint 19.1/20.3, Windows XP SP3, Windows 7 Professional, Windows 10 Home

rvk

  • Hero Member
  • *****
  • Posts: 6799
Re: Including OpenSSL in a cross-platform application
« Reply #29 on: September 13, 2023, 09:39:39 am »
Frankly, I would not like to be obliged to upgrade these systems now, for lack of time and fear of breaking what is already working.
No, I didn't mean upgrading the OS, but upgrading to Lazarus 2.6.
If you have problems with OpenSSL, that would be the easiest way I think.

I really never imagined that these issues could also be related to the FPC/Lazarus version! %)
What issues are you talking about exactly. Because I think those were never really discussed here.
Do you mean installing OpenSSL for https access on those systems?

Or do you have other problems?

I just tried Especies (compiled with trunk) on a clean Mint 21 system (default installed OpenSSL 3.0.2) and it worked the same as on Windows.

In earlier versions I have used temporary files for output, but dropped them in the current versions because memory files seemed to be more efficient. I don't understand why should temporary files be more efficient (and they may need administrative privileges).
No, just the other way around. Using intermediate (temporary) files is less efficient. But if you would use a system call with Curl then you might need to use those again. Or you would need to do something with stdin (catching output from a file). Besides... if temporary files could be restricted on a system, then calling an external program like Curl could also be restricted.

Edit: BTW. I just tried it with Laz 2.0.12 (which includes FPC 3.2.0).
It indeed doesn't support OpenSSL 3 but that's easily fixed (if you can recompile FPC yourself).
You do need to recompile FPC because just recompiling Lazarus doesn't help (openssl.pas is part of FPC, not Lazarus).

I'm not sure how to do that if you have a release Lazarus, I only do this from trunk and setting the version.
fpcupdeluxe is probably the easiest way although I'm not sure if you can compile after you make changes, so you would need a version which has openssl 3. Maybe fixes_xx but perhaps someone else knows this.

You can just add the ".3" to the DLLVersions array (like delphius already hinted at).
I did this here and it works with Laz 2.0.12/FPC 3.2.0.

For FPC 3.2.0 that would be (change the 17 to 18 and add the .3 before the 1.1):
Code: [Select]
  DLLVersions: array[1..18] of string = ('', '.3', '.1.1', '.1.0.6', '.1.0.5', '.1.0.4', '.1.0.3',
                                        '.1.0.2', '.1.0.1','.1.0.0','.0.9.8',
                                        '.0.9.7', '.0.9.6', '.0.9.5', '.0.9.4',
                                        '.0.9.3', '.0.9.2', '.0.9.1');

But it's probably safe to just take the complete DLLVersions array from trunk (so other versions work too):
Code: [Select]
  DLLVersions: array[1..20] of AnsiString = ('', '.3', '.1.1', '.11', '.10', '.1.0.6', '.1.0.5', '.1.0.4', '.1.0.3',
                                        '.1.0.2', '.1.0.1','.1.0.0','.0.9.8',
                                        '.0.9.7', '.0.9.6', '.0.9.5', '.0.9.4',
                                        '.0.9.3', '.0.9.2', '.0.9.1');

For Windows you can (for now) just add the 1.0.2 yourself as dll in the exe directory.

I just checked Laz 2.2.6 (fpc 3.2.2). That one is still at 1.1 in the array so doesn't support openssl 3.0.
You would need to make the same fix there too (or go for trunk).

« Last Edit: September 13, 2023, 11:10:41 am by rvk »

 

TinyPortal © 2005-2018