Example for HTTPS client including certificate verification

[Doyenne]

Can anyone give me an example of how to download a web resource via HTTPS, including certificate verification? Preferably using the FCL (e.g. fcl-web), but Synapse is also OK.

I saw that in FCL trunk, there is a VerifySSLCertificate property in TFPHTTPClient, but then I need to write an event handler and get stuck.

Let's say that I would like to download a web page and https://badssl.com/ should work, but https://expired.badssl.com/ and https://wrong.host.badssl.com/ should be rejected.

This is what I have, but I don't know how to add certificate verification:

Code: Pascal  [Select][+][-]

1. program project1;
2.  
3. {$mode objfpc}{$H+}
4.  
5. uses SysUtils, fphttpclient, openssl, opensslsockets;
6.  
7. const URLS: array[0..2] of string = (
8.   'https://badssl.com/',            // should work
9.   'https://expired.badssl.com/',    // should be rejected
10.   'https://wrong.host.badssl.com/'  // should be rejected
11. );
12.  
13. procedure TryURLs;
14. var
15.   URL: string;
16. begin
17.   for URL in URLS do
18.     try
19.       with TFPHTTPClient.Create(nil) do
20.         try
21.           AllowRedirect := True;
22.           // do something to enable certificate verification??
23.           Get(URL);
24.         finally
25.           Free;
26.         end;
27.       WriteLn(URL, ' succeeded.');
28.     except
29.       on E: Exception do
30.         WriteLn(Format('%s failed! (%s)', [URL, E.Message]));
31.     end;
32. end;
33.  
34. begin
35.   TryURLs;
36. end.

Thanks for any hint in the right direction!

[PierceNg]

Can anyone give me an example of how to download a web resource via HTTPS, including certificate verification? Preferably using the FCL (e.g. fcl-web), but Synapse is also OK.

I saw that in FCL trunk, there is a VerifySSLCertificate property in TFPHTTPClient, but then I need to write an event handler and get stuck.

The example fcl-web/examples/httpclient/httpget.pas shows how to set up the event handler.

Edit: Doesn't look fully functional as this line prints empty string:

Code: [Select]

TEncoding.ASCII.GetAnsiString( aHandler.CertificateData.Certificate.Value)

[PierceNg]

libcurl has what you want. Below is libcurl/examples/testcurl.pp with my changes marked "<====":

Code: [Select]

{$mode objfpc}
{$H+}
program testcurl;

uses libcurl;

Var
  URL : Pchar = 'https://wrong.host.badssl.com'; // <==== Change URL
  hCurl : pCurl;

begin
  hCurl:= curl_easy_init;
  if Assigned(hCurl) then
    begin
    curl_easy_setopt(hCurl,CURLOPT_VERBOSE, [True]);
    curl_easy_setopt(hCurl,CURLOPT_URL,[URL]);
    curl_easy_setopt(hCurl, CURLOPT_SSL_VERIFYPEER, [True]); // <==== Add this line
    curl_easy_perform(hCurl);
    curl_easy_cleanup(hCurl);
    end;
end.
Running it:

Code: Text  [Select][+][-]

1. % ./testcurl
2. *   Trying 104.154.89.105:443...
3. * TCP_NODELAY set
4. * Connected to wrong.host.badssl.com (104.154.89.105) port 443 (#0)
5. * ALPN, offering h2
6. * ALPN, offering http/1.1
7. * successfully set certificate verify locations:
8. *   CAfile: /etc/ssl/certs/ca-certificates.crt
9.   CApath: /etc/ssl/certs
10. * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
11. * ALPN, server accepted to use http/1.1
12. * Server certificate:
13. *  subject: CN=*.badssl.com
14. *  start date: Jan 22 16:33:26 2023 GMT
15. *  expire date: Apr 22 16:33:25 2023 GMT
16. *  subjectAltName does not match wrong.host.badssl.com
17. * SSL: no alternative certificate subject name matches target host name 'wrong.host.badssl.com'
18. * Closing connection 0

For expired:

Code: Text  [Select][+][-]

1. % ./testcurl
2. *   Trying 104.154.89.105:443...
3. * TCP_NODELAY set
4. * Connected to expired.badssl.com (104.154.89.105) port 443 (#0)
5. * ALPN, offering h2
6. * ALPN, offering http/1.1
7. * successfully set certificate verify locations:
8. *   CAfile: /etc/ssl/certs/ca-certificates.crt
9.   CApath: /etc/ssl/certs
10. * SSL certificate problem: certificate has expired
11. * Closing connection 0

[BeniBela]

My internet tools handle that.

Code: Pascal  [Select][+][-]

1. program Project1;
2.  
3. uses internetaccess, synapseinternetaccess;
4. var
5.   i: TInternetAccess;
6. begin
7.   i := TSynapseInternetAccess.create();
8.   i.get('https://badssl.com/');
9. //  i.get('https://expired.badssl.com/');
10. //  i.get('https://wrong.host.badssl.com/');
11. end.
12.  

I started with the Synapse HTTPS handling, but changed a lot

[domasz]

Does any of these solutions work with HTTPS without DLLs?

[PierceNg]

Does any of these solutions work with HTTPS without DLLs?

The DLLs provide the TLS functionality including crypto, certificate handling, wire protocols, etc. Not using those DLLs means using a pure Pascal implementation of TLS with equivalent functionality. AFAIK there is no such thing.

[Thaddy]

AFAIK there is no such thing.

Of course there is: our forum member Xor-el (the one with the unpronouncable name) took care of that.... (about 4 years ago). Only "problem" is you need to know what you are doing.

[Doyenne]

My internet tools handle that.

I'm highly impressed! I had to install the FLRE and POCO library as well before I could compile, but then it worked perfectly.

IMHO, this is how it should be in 2023: SSL verification easy, by default and out of the box. (IMHO using HTTPS without certificate checking does not make sense.)

However, your library is GPL (not LGPL) so unfortunately I cannot use it in my project.

[Doyenne]

If I try using Synapse trunk, same machine, same configuration, it doesn't work:

Code: Pascal  [Select][+][-]

1. program project1;
2.  
3. uses blcksock, ssl_openssl3;
4.  
5. const URL = 'badssl.com';
6.  
7. var
8.   Sock: TTCPBlockSocket;
9. begin
10.   Sock := TTCPBlockSocket.Create;
11.   try
12.     Sock.Connect(URL, '443');
13.     Sock.SSL.VerifyCert := True;
14.     Sock.SSLDoConnect;
15.     if Sock.LastError <>  0 then
16.     begin
17.       WriteLn('Error: ', Sock.LastErrorDesc);
18.       WriteLn('GetVerifyCert: ', Sock.SSL.GetVerifyCert);
19.     end
20.     else
21.       WriteLn('Success!');
22.   finally
23.     Sock.Free;
24.   end;
25. end.
26.  

Output:

Code: [Select]

Error: error:0A000086:SSL routines::certificate verify failed
GetVerifyCert: 20

The number 20 might mean that the root certificate could not be checked (I find OpenSSL messages always a bit unclear), because if I intentionally hide the root certificates to OpenSSL, I get verify error num=20:

Code: [Select]

$ openssl s_client -quiet -connect badssl.com:443 -CApath /tmp
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = *.badssl.com
verify return:1

Adding

Code: [Select]

sock.SSL.CertCAFile := '/etc/ssl/certs/ISRG_Root_X1.pem'; does not solve the problem, still same result.

Any hints are welcome...

[PierceNg]

AFAIK there is no such thing.

Of course there is: our forum member Xor-el (the one with the unpronouncable name) took care of that.... (about 4 years ago). Only "problem" is you need to know what you are doing.

That's a crypto library, not a TLS implementation.

[Thaddy]

I am sorry, but all the code you need is there. If you do not understand it, ask.

[domasz]

I am sorry, but all the code you need is there. If you do not understand it, ask.

I don't. Care to explain?

[Doyenne]

If I try using Synapse trunk, same machine, same configuration, it doesn't work:
(...)

Finally I figured out why this Synapse code didn't work... Badssl.com is serving different URLs from the same IP address. Therefore, the Server Name Indication (SNI) needs to be used, so the server knows which certificate to present to the client (e.g. the badssl.com certificate, the expired.badssl.com certificate, etc.). The THTTPSend component automatically adds SNI to the request, but the TCPBlockSocket does not. Therefore, I had to add this manually:

Code: Pascal  [Select][+][-]

1. Sock.SSL.SNIHost := URL;

The easier solution is of course to use THTTPSend. The following example shows how to use certificate verification with Synapse:

This code uses Synapse trunk [r270]

Code: Pascal  [Select][+][-]

1. program project1;
2.  
3. uses httpsend, ssl_openssl3;
4.  
5. const URLs: array[0..5] of string = (
6.   'https://badssl.com',                // should work
7.   'https://wrong.host.badssl.com',     // should be rejected
8.   'https://expired.badssl.com',        // should be rejected
9.   'https://self-signed.badssl.com',    // should be rejected
10.   'https://untrusted-root.badssl.com', // should be rejected
11.   'https://revoked.badssl.com'         // should be rejected
12. );
13.  
14. var
15.   URL: string;
16.   HTTP: THTTPSend;
17. begin
18.   HTTP := THTTPSend.Create;
19.   try
20.     HTTP.Sock.SSL.VerifyCert := True;
21.     HTTP.Sock.SSL.CertCAFile := 'ca-bundle.crt';
22.     for URL in URLs do
23.     begin
24.       if HTTP.HTTPMethod('GET', URL) then
25.         WriteLn('Success: ', URL)
26.       else
27.         WriteLn('Failure: ', URL, '; verify result: ', HTTP.Sock.SSL.GetVerifyCert);
28.     end;
29.   finally
30.     HTTP.Free;
31.   end;
32. end.
33.  

As show in this example, certificate verification also needs a reference to trusted root (CA) certificates. Without any trusted CA certificates, all verifications will fail. A file containing trusted certificates in PEM format can for example be created using the mk-ca-bundle script from the Curl project (https://curl.se/docs/mk-ca-bundle.html).

Warning!
Using the current Synapse code (trunk [r270]), certificate verification works, but wrong host certificates are not detected. This is a security risk.
I filed a bug including patches to fix this issue: https://sourceforge.net/p/synalist/bugs/75/

[Doyenne]

Can anyone give me an example of how to download a web resource via HTTPS, including certificate verification? Preferably using the FCL (e.g. fcl-web), but Synapse is also OK.
(...)

I also found out why my code using fcl-web did not work. To enable certificate verification using TFPHTTPClient, the following was needed:

• VerifySSlCertificate := True;
• Add a callback function and link this to the AfterSocketHandlerCreate event, e.g. AfterSocketHandlerCreate := @DoHaveSocketHandler;
• In the callback function, provide a reference to a directory containing trusted root (CA) certificates, or a reference to a file containing trusted root (CA) certificates (e.g. in PEM format).

I filed a suggestion to ease certificate verification by eliminating the need for the callback procedure. This suggestion was accepted (thank you!), see https://gitlab.com/freepascal.org/fpc/source/-/issues/40480

So, using FCL trunk (version 2023-10-22 or later), certificate verification is as easy as:

Code: Pascal  [Select][+][-]

1. program project1;
2.  
3. {$mode objfpc}{$H+}
4.  
5. uses
6.   fphttpclient, opensslsockets;
7.  
8. const
9.   URL = 'https://example.com/';
10.  
11. begin
12.   with TFPHTTPClient.Create(nil) do
13.   try
14.     VerifySSlCertificate := True;
15.     TrustedCertsDir := '/etc/ssl/certs/'; // or: CertCAFileName:='ca-bundle.crt';
16.     WriteLn(Get(URL));
17.   finally
18.     Free;
19.   end;
20. end.
21.  

Warning!
Using the current FCL code (trunk 2023-10-22), certificate verification works, but wrong host certificates are not detected. This is a security risk.
I filed a bug including patches to fix this issue: https://gitlab.com/freepascal.org/fpc/source/-/issues/40479

[delphius]

I am sorry, but all the code you need is there. If you do not understand it, ask.

I also thought it would be a simple story, but still not

That's a crypto library, not a TLS implementation.

And the set of cryptoprimitives presented in it will not be enough to implement even half of the protocol

Does any of these solutions work with HTTPS without DLLs?

I searched the whole Internet in search of such a thing on pure pascal, all I found were scattered pieces. There are two commercial libraries in pure pascal, but this is beyond my interests

AFAIK there is no such thing.

It is, otherwise I wouldn't have to write my own training implementation of the TLS 1.3 protocol.
https://github.com/delphius/fpmtls