Lazarus

Programming => Networking and Web Programming => Topic started by: Doyenne on March 17, 2023, 05:19:11 pm

Title: Example for HTTPS client including certificate verification
Post by: Doyenne on March 17, 2023, 05:19:11 pm
Can anyone give me an example of how to download a web resource via HTTPS, including certificate verification? Preferably using the FCL (e.g. fcl-web), but Synapse is also OK.

I saw that in FCL trunk, there is a VerifySSLCertificate property in TFPHTTPClient, but then I need to write an event handler and get stuck.

Let's say that I would like to download a web page and https://badssl.com/ should work, but https://expired.badssl.com/ and https://wrong.host.badssl.com/ should be rejected.

This is what I have, but I don't know how to add certificate verification:

Code: Pascal  [Select][+][-]
  1. program project1;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. uses SysUtils, fphttpclient, openssl, opensslsockets;
  6.  
  7. const URLS: array[0..2] of string = (
  8.   'https://badssl.com/',            // should work
  9.   'https://expired.badssl.com/',    // should be rejected
  10.   'https://wrong.host.badssl.com/'  // should be rejected
  11. );
  12.  
  13. procedure TryURLs;
  14. var
  15.   URL: string;
  16. begin
  17.   for URL in URLS do
  18.     try
  19.       with TFPHTTPClient.Create(nil) do
  20.         try
  21.           AllowRedirect := True;
  22.           // do something to enable certificate verification??
  23.           Get(URL);
  24.         finally
  25.           Free;
  26.         end;
  27.       WriteLn(URL, ' succeeded.');
  28.     except
  29.       on E: Exception do
  30.         WriteLn(Format('%s failed! (%s)', [URL, E.Message]));
  31.     end;
  32. end;
  33.  
  34. begin
  35.   TryURLs;
  36. end.

Thanks for any hint in the right direction!
Title: Re: Example for HTTPS client including certificate verification
Post by: PierceNg on March 18, 2023, 02:51:33 am
Quote from: Doyenne on March 17, 2023, 05:19:11 pm
Can anyone give me an example of how to download a web resource via HTTPS, including certificate verification? Preferably using the FCL (e.g. fcl-web), but Synapse is also OK.

I saw that in FCL trunk, there is a VerifySSLCertificate property in TFPHTTPClient, but then I need to write an event handler and get stuck.

The example fcl-web/examples/httpclient/httpget.pas shows how to set up the event handler.

Edit: Doesn't look fully functional as this line prints empty string:

Code: [Select]
TEncoding.ASCII.GetAnsiString( aHandler.CertificateData.Certificate.Value)
Title: Re: Example for HTTPS client including certificate verification
Post by: PierceNg on March 18, 2023, 04:01:49 am
libcurl has what you want. Below is libcurl/examples/testcurl.pp with my changes marked "<====":

Code: [Select]
{$mode objfpc}
{$H+}
program testcurl;

uses libcurl;

Var
  URL : Pchar = 'https://wrong.host.badssl.com'; // <==== Change URL
  hCurl : pCurl;

begin
  hCurl:= curl_easy_init;
  if Assigned(hCurl) then
    begin
    curl_easy_setopt(hCurl,CURLOPT_VERBOSE, [True]);
    curl_easy_setopt(hCurl,CURLOPT_URL,[URL]);
    curl_easy_setopt(hCurl, CURLOPT_SSL_VERIFYPEER, [True]); // <==== Add this line
    curl_easy_perform(hCurl);
    curl_easy_cleanup(hCurl);
    end;
end.
Running it:

Code: Text  [Select][+][-]
  1. % ./testcurl
  2. *   Trying 104.154.89.105:443...
  3. * TCP_NODELAY set
  4. * Connected to wrong.host.badssl.com (104.154.89.105) port 443 (#0)
  5. * ALPN, offering h2
  6. * ALPN, offering http/1.1
  7. * successfully set certificate verify locations:
  8. *   CAfile: /etc/ssl/certs/ca-certificates.crt
  9.   CApath: /etc/ssl/certs
  10. * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
  11. * ALPN, server accepted to use http/1.1
  12. * Server certificate:
  13. *  subject: CN=*.badssl.com
  14. *  start date: Jan 22 16:33:26 2023 GMT
  15. *  expire date: Apr 22 16:33:25 2023 GMT
  16. *  subjectAltName does not match wrong.host.badssl.com
  17. * SSL: no alternative certificate subject name matches target host name 'wrong.host.badssl.com'
  18. * Closing connection 0

For expired:

Code: Text  [Select][+][-]
  1. % ./testcurl
  2. *   Trying 104.154.89.105:443...
  3. * TCP_NODELAY set
  4. * Connected to expired.badssl.com (104.154.89.105) port 443 (#0)
  5. * ALPN, offering h2
  6. * ALPN, offering http/1.1
  7. * successfully set certificate verify locations:
  8. *   CAfile: /etc/ssl/certs/ca-certificates.crt
  9.   CApath: /etc/ssl/certs
  10. * SSL certificate problem: certificate has expired
  11. * Closing connection 0
Title: Re: Example for HTTPS client including certificate verification
Post by: BeniBela on March 18, 2023, 12:30:09 pm
My internet tools (https://benibela.de/sources_en.html#top) handle that.

Code: Pascal  [Select][+][-]
  1. program Project1;
  2.  
  3. uses internetaccess, synapseinternetaccess;
  4. var
  5.   i: TInternetAccess;
  6. begin
  7.   i := TSynapseInternetAccess.create();
  8.   i.get('https://badssl.com/');
  9. //  i.get('https://expired.badssl.com/');
  10. //  i.get('https://wrong.host.badssl.com/');
  11. end.
  12.  

I started with the Synapse HTTPS handling, but changed a lot (https://github.com/benibela/internettools/blob/master/internet/synapse_ssl_openssl_override.pas#L842-L1137)
Title: Re: Example for HTTPS client including certificate verification
Post by: domasz on March 18, 2023, 01:02:54 pm
Does any of these solutions work with HTTPS without DLLs?
Title: Re: Example for HTTPS client including certificate verification
Post by: PierceNg on March 18, 2023, 01:19:57 pm
Quote from: domasz on March 18, 2023, 01:02:54 pm
Does any of these solutions work with HTTPS without DLLs?

The DLLs provide the TLS functionality including crypto, certificate handling, wire protocols, etc. Not using those DLLs means using a pure Pascal implementation of TLS with equivalent functionality. AFAIK there is no such thing.
Title: Re: Example for HTTPS client including certificate verification
Post by: Thaddy on March 18, 2023, 03:45:59 pm
Quote from: PierceNg on March 18, 2023, 01:19:57 pm
AFAIK there is no such thing.
Of course there is: our forum member Xor-el (the one with the unpronouncable name) took care of that.... (about 4 years ago). Only "problem" is you need to know what you are doing.
Title: Re: Example for HTTPS client including certificate verification
Post by: Doyenne on March 18, 2023, 05:24:55 pm
Quote from: BeniBela on March 18, 2023, 12:30:09 pm
My internet tools (https://benibela.de/sources_en.html#top) handle that.

I'm highly impressed! I had to install the FLRE and POCO library as well before I could compile, but then it worked perfectly.

IMHO, this is how it should be in 2023: SSL verification easy, by default and out of the box. (IMHO using HTTPS without certificate checking does not make sense.)

However, your library is GPL (not LGPL) so unfortunately I cannot use it in my project.
Title: Re: Example for HTTPS client including certificate verification
Post by: Doyenne on March 18, 2023, 05:29:33 pm
If I try using Synapse trunk, same machine, same configuration, it doesn't work:

Code: Pascal  [Select][+][-]
  1. program project1;
  2.  
  3. uses blcksock, ssl_openssl3;
  4.  
  5. const URL = 'badssl.com';
  6.  
  7. var
  8.   Sock: TTCPBlockSocket;
  9. begin
  10.   Sock := TTCPBlockSocket.Create;
  11.   try
  12.     Sock.Connect(URL, '443');
  13.     Sock.SSL.VerifyCert := True;
  14.     Sock.SSLDoConnect;
  15.     if Sock.LastError <>  0 then
  16.     begin
  17.       WriteLn('Error: ', Sock.LastErrorDesc);
  18.       WriteLn('GetVerifyCert: ', Sock.SSL.GetVerifyCert);
  19.     end
  20.     else
  21.       WriteLn('Success!');
  22.   finally
  23.     Sock.Free;
  24.   end;
  25. end.
  26.  

Output:
Code: [Select]
Error: error:0A000086:SSL routines::certificate verify failed
GetVerifyCert: 20

The number 20 might mean that the root certificate could not be checked (I find OpenSSL messages always a bit unclear), because if I intentionally hide the root certificates to OpenSSL, I get verify error num=20:

Code: [Select]
$ openssl s_client -quiet -connect badssl.com:443 -CApath /tmp
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = *.badssl.com
verify return:1

Adding
Code: [Select]
sock.SSL.CertCAFile := '/etc/ssl/certs/ISRG_Root_X1.pem'; does not solve the problem, still same result.

Any hints are welcome...
Title: Re: Example for HTTPS client including certificate verification
Post by: PierceNg on March 18, 2023, 05:43:13 pm
Quote from: Thaddy on March 18, 2023, 03:45:59 pm
Quote from: PierceNg on March 18, 2023, 01:19:57 pm
AFAIK there is no such thing.
Of course there is: our forum member Xor-el (the one with the unpronouncable name) took care of that.... (about 4 years ago). Only "problem" is you need to know what you are doing.

That's a crypto library, not a TLS implementation.
Title: Re: Example for HTTPS client including certificate verification
Post by: Thaddy on March 18, 2023, 06:42:17 pm
I am sorry, but all the code you need is there. If you do not understand it, ask.
Title: Re: Example for HTTPS client including certificate verification
Post by: domasz on March 18, 2023, 06:46:47 pm
Quote from: Thaddy on March 18, 2023, 06:42:17 pm
I am sorry, but all the code you need is there. If you do not understand it, ask.
I don't. Care to explain?
Title: Re: Example for HTTPS client including certificate verification
Post by: Doyenne on October 23, 2023, 04:27:15 pm
Quote from: Doyenne on March 18, 2023, 05:29:33 pm
If I try using Synapse trunk, same machine, same configuration, it doesn't work:
(...)

Finally I figured out why this Synapse code didn't work... Badssl.com is serving different URLs from the same IP address. Therefore, the Server Name Indication (SNI) needs to be used, so the server knows which certificate to present to the client (e.g. the badssl.com certificate, the expired.badssl.com certificate, etc.). The THTTPSend component automatically adds SNI to the request, but the TCPBlockSocket does not. Therefore, I had to add this manually:
Code: Pascal  [Select][+][-]
  1. Sock.SSL.SNIHost := URL;

The easier solution is of course to use THTTPSend. The following example shows how to use certificate verification with Synapse:

This code uses Synapse trunk [r270]

Code: Pascal  [Select][+][-]
  1. program project1;
  2.  
  3. uses httpsend, ssl_openssl3;
  4.  
  5. const URLs: array[0..5] of string = (
  6.   'https://badssl.com',                // should work
  7.   'https://wrong.host.badssl.com',     // should be rejected
  8.   'https://expired.badssl.com',        // should be rejected
  9.   'https://self-signed.badssl.com',    // should be rejected
  10.   'https://untrusted-root.badssl.com', // should be rejected
  11.   'https://revoked.badssl.com'         // should be rejected
  12. );
  13.  
  14. var
  15.   URL: string;
  16.   HTTP: THTTPSend;
  17. begin
  18.   HTTP := THTTPSend.Create;
  19.   try
  20.     HTTP.Sock.SSL.VerifyCert := True;
  21.     HTTP.Sock.SSL.CertCAFile := 'ca-bundle.crt';
  22.     for URL in URLs do
  23.     begin
  24.       if HTTP.HTTPMethod('GET', URL) then
  25.         WriteLn('Success: ', URL)
  26.       else
  27.         WriteLn('Failure: ', URL, '; verify result: ', HTTP.Sock.SSL.GetVerifyCert);
  28.     end;
  29.   finally
  30.     HTTP.Free;
  31.   end;
  32. end.
  33.  

As show in this example, certificate verification also needs a reference to trusted root (CA) certificates. Without any trusted CA certificates, all verifications will fail. A file containing trusted certificates in PEM format can for example be created using the mk-ca-bundle script from the Curl project (https://curl.se/docs/mk-ca-bundle.html).

Warning!
Using the current Synapse code (trunk [r270]), certificate verification works, but wrong host certificates are not detected. This is a security risk.
I filed a bug including patches to fix this issue: https://sourceforge.net/p/synalist/bugs/75/
Title: Re: Example for HTTPS client including certificate verification
Post by: Doyenne on October 23, 2023, 04:41:37 pm
Quote from: Doyenne on March 17, 2023, 05:19:11 pm
Can anyone give me an example of how to download a web resource via HTTPS, including certificate verification? Preferably using the FCL (e.g. fcl-web), but Synapse is also OK.
(...)

I also found out why my code using fcl-web did not work. To enable certificate verification using TFPHTTPClient, the following was needed:

I filed a suggestion to ease certificate verification by eliminating the need for the callback procedure. This suggestion was accepted (thank you!), see https://gitlab.com/freepascal.org/fpc/source/-/issues/40480

So, using FCL trunk (version 2023-10-22 or later), certificate verification is as easy as:
Code: Pascal  [Select][+][-]
  1. program project1;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. uses
  6.   fphttpclient, opensslsockets;
  7.  
  8. const
  9.   URL = 'https://example.com/';
  10.  
  11. begin
  12.   with TFPHTTPClient.Create(nil) do
  13.   try
  14.     VerifySSlCertificate := True;
  15.     TrustedCertsDir := '/etc/ssl/certs/'; // or: CertCAFileName:='ca-bundle.crt';
  16.     WriteLn(Get(URL));
  17.   finally
  18.     Free;
  19.   end;
  20. end.
  21.  

Warning!
Using the current FCL code (trunk 2023-10-22), certificate verification works, but wrong host certificates are not detected. This is a security risk.
I filed a bug including patches to fix this issue: https://gitlab.com/freepascal.org/fpc/source/-/issues/40479
Title: Re: Example for HTTPS client including certificate verification
Post by: delphius on October 24, 2023, 08:19:02 am
Quote from: Thaddy on March 18, 2023, 06:42:17 pm
I am sorry, but all the code you need is there. If you do not understand it, ask.
I also thought it would be a simple story, but still not

Quote from: PierceNg on March 18, 2023, 05:43:13 pm
That's a crypto library, not a TLS implementation.
And the set of cryptoprimitives presented in it will not be enough to implement even half of the protocol

Quote from: domasz on March 18, 2023, 01:02:54 pm
Does any of these solutions work with HTTPS without DLLs?
I searched the whole Internet in search of such a thing on pure pascal, all I found were scattered pieces. There are two commercial libraries in pure pascal, but this is beyond my interests

Quote from: PierceNg on March 18, 2023, 01:19:57 pm
AFAIK there is no such thing.
It is, otherwise I wouldn't have to write my own training implementation of the TLS 1.3 protocol.
https://github.com/delphius/fpmtls
Title: Re: Example for HTTPS client including certificate verification
Post by: PierceNg on October 30, 2023, 03:09:48 am
Quote from: delphius on October 24, 2023, 08:19:02 am
Quote from: PierceNg on March 18, 2023, 01:19:57 pm
AFAIK there is no such thing.
It is, otherwise I wouldn't have to write my own training implementation of the TLS 1.3 protocol.
https://github.com/delphius/fpmtls

Nice work.

And there is an open source implementation of TLS in Pascal - https://github.com/fundamentalslib/fundamentals5/tree/master/Source/TLS
Title: Re: Example for HTTPS client including certificate verification
Post by: delphius on October 30, 2023, 05:05:18 am
Quote from: PierceNg on October 30, 2023, 03:09:48 am
And there is an open source implementation of TLS in Pascal - https://github.com/fundamentalslib/fundamentals5/tree/master/Source/TLS

But while I was doing it, I figured out the essence of the question :)

So there is still a benefit.

But thanks a lot for the link to the library, it will greatly help in determining further movement

P.S. No miracle happened)
Did not find the expected implementation of Chacha20 + Poly1305 + curve25519))))
No GCM/GHASH  :(
Title: Re: Example for HTTPS client including certificate verification
Post by: krolikbest on November 03, 2023, 09:19:56 am
Hi,
according to this example
Code: Pascal  [Select][+][-]
  1. unit Unit1;
  2. {$mode objfpc}{$H+}
  3.  
  4. interface
  5.  
  6. uses
  7.   Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls
  8.   ,httpsend,
  9.   ,ssl_openssl //it works
  10. //  ,ssl_openssl3  //not positive result at all
  11.   ,openssl;
  12.  
  13. type
  14.  
  15.   { TForm1 }
  16.  
  17.   TForm1 = class(TForm)
  18.     Button1: TButton;
  19.     Memo1: TMemo;
  20.     procedure Button1Click(Sender: TObject);
  21.   private
  22.  
  23.   public
  24.  
  25.   end;
  26.  
  27. const URLs: array[0..5] of string = (
  28.   'https://badssl.com',                // should work
  29.   'https://wrong.host.badssl.com',     // should be rejected
  30.   'https://expired.badssl.com',        // should be rejected
  31.   'https://self-signed.badssl.com',    // should be rejected
  32.   'https://untrusted-root.badssl.com', // should be rejected
  33.   'https://revoked.badssl.com'         // should be rejected
  34. );
  35.  
  36. var
  37.   URL: string;
  38.   HTTP: THTTPSend;
  39.   Form1: TForm1;
  40.  
  41. implementation
  42.  
  43. {$R *.lfm}
  44.  
  45. { TForm1 }
  46.  
  47. procedure TForm1.Button1Click(Sender: TObject);
  48. begin
  49.   HTTP := THTTPSend.Create;
  50.   try
  51.     HTTP.Sock.SSL.VerifyCert := True;
  52.     HTTP.Sock.SSL.CertCAFile := 'somecrt.crt';
  53.     for URL in URLs do
  54.     begin
  55.       if HTTP.HTTPMethod('GET', URL) then
  56.         WriteLn('Success: ', URL)
  57.       else
  58.         WriteLn('Failure: ', URL, '; verify result: ', HTTP.Sock.SSL.GetVerifyCert);
  59.     end;
  60.   finally
  61.     HTTP.Free;
  62.   end;
  63. end;
  64.  
  65. end.

, I wonder what difference is between ssl_openssl and ssl_openssl3? I keep in application directory two libraries: libeay32 and ssleay32 but in case of using ssl_openssl3  I get 10091 error for all urls, whereas using ssl_openssl I get for first url positive result and for others not (but not 10091 error). So the problem is in using ssl_openssl3, could someone explain this to me?
I use Lazarus 2.2.6, Win10 64bit.
Title: Re: Example for HTTPS client including certificate verification
Post by: paweld on November 03, 2023, 12:35:57 pm
To use openssl3 you need the libraries: libssl-3.dll and libcrypto-3.dll.
ssleay32.dll and libeay32.dll are used by openssl1.0
Title: Re: Example for HTTPS client including certificate verification
Post by: krolikbest on November 03, 2023, 02:36:32 pm
Somehow I didn't search enough, thanks a lot.
TinyPortal © 2005-2018