This article describes how to test your HTTPS client or browser using openssl. To test your HTTPS client, you need an HTTPS server, or a web server, such as IIS, apache, nginx, or openssl. You also need some test cases. There are three common failure modes in SSL/TLS:
- The client makes the connection when it should not,
- The connection fails when it should succeed, and
- The connection is properly made, but the data is corrupted in transmission.
- There is a 4th failure mode: the data might not be securely transmitted. That failure mode is out of scope of this article.
To make sure that any problems uncovered in testing are due to issues in your HTTPS client, we want to use a “known good” HTTPS server. We also want a server that is “pedantic” or “unforgiving”. openssl fits these requirements precisely.
In this article, I am going to describe how use the openssl s_server command to be an HTTPS server. There are many configuration items that have to be just right, so I am not only going to show you how to do it right, but I am also going to share with you what went wrong, and how I went about diagnosing them and fixing them.
A “client” is a computer or a computer program that initiates a connection to a “server”. A “server” is a computer program that waits for a connection to arrive from a “client”. For HTTP and HTTPS, there are “browsers” and “clients”. Browsers are designed for interaction with humans and usually have graphical user interfaces. All browsers are HTTP/HTTPS clients.
However, there are HTTP/HTTPS clients which are not browsers. These clients are designed for use as automated systems. The wise server designer will ensure that their system can be used effectively with HTTPS clients that are browsers and HTTPS clients that are not browsers.
In this tutorial you will learn:
- How to select a good HTTPS client or browser
- How to use openssl as an HTTPS server
- How to use an HTTPS server to test an HTTPS client
Software Requirements and Conventions Used
| Category | Requirements, Conventions or Software Version Used |
|---|---|
| System | Any Linux system |
| Software | OpenSSL or any HTTPS server such IIS, Apache Nginx |
| Other | Privileged access to your Linux system as root or via the sudo command. |
| Conventions | # – requires given linux commands to be executed with root privileges either directly as a root user or by use of sudo command$ – requires given linux commands to be executed as a regular non-privileged user |
How to test your HTTPS client step by step instructions
I will use the adjectives “righteous” to indicate that a test did something properly, and “errorneous” to indicate that a test did something wrong. If a test fails when it should, then that’s a righteous failure. If a test passes when it should not, then that’s a erroneous pass.
I wanted to use an HTTPS client that I could break and repair at will, and I found it: the http command (it’s in github as httpie). If I use the -verify=no option, then the client is broken: it will erroneously pass tests. I was unable to create an erroneous fail, and that’s A Good Thing as it means that if the client fails, then something is wrong.
At the heart of the SSL/TLS protocol (they changed the name and little else) are two files, a “certificate” (or “cert” for short) and a secret “key”. All over the protocol, one end of the connection will ask the other end for a certificate. The first end will use some of the information in the cert to create a mathematical puzzle that only something that has the secret key can answer. The secret key never leaves its machine: solving the problem means that the near end knows that the far end has the key, but not what the key is.
The openssl command is essentially a command line interface to libssl. It contains a crude server invoked with the s_server subcommand. openssl will need a public cert/private key pair. In my case, I already had them for my production web server. I got them from let’s encrypt, for free.
As a proof-of-concept that the server is working properly, I copied the cert and key to my development machine and started the openssl HTTPS server.
On the server side:
$ openssl s_server -status_verbose -HTTP -cert fullchain.pem -key privkey.pem Using default temp DH parameters ACCEPT
My first attempt FAILED!
$ http --verify=yes jeffs-desktop:4433/index.html http: error: ConnectionError: (Connection aborted., RemoteDisconnected(Remote end closed connection without response)) while doing GET request to URL: http://jeffs-desktop:4433/index.html
First hypothesis: the key and the cert don’t match. I checked that:
$ openssl x509 -noout -modulus -in fullchain.pemls | openssl md5 (stdin)= b9dbd040d9a0c3b5d3d50af46bc87784 $ openssl rsa -noout -modulus -in privkey.pem | openssl md5 (stdin)= b9dbd040d9a0c3b5d3d50af46bc87784
They match. So why is this failing? Because my certificate is for linuxconfig.dns.net but I am using jeffs-desktop as my host name.
jeffs@jeffs-desktop:~/documents$ openssl x509 -text -noout -in fullchain.pem | fgrep CN
Issuer: C = US, O = Let’s Encrypt, CN = R3
Subject: CN = linuxconfig.ddns.net
This is a righteous failure: the server was misconfigured and my client detected it. Had I used the
-verify=no option, then I would have a broken client and it would have not detected the problem. Note that any data transmitted would still be secure against an eavesdropper. I can fix this problem by modifying my /etc/hosts file with my own IPv4 and IPv6 addresses.
192.168.1.149 linuxconfig.ddns.net 2601:602:8500:b65:155a:7b81:65c:21fa linuxconfig.ddns.net
(incidentally, the ease by which you can fake an IP address is one of the motivations of SSL/TLS in the first place).
Try again. On the server side:
$ openssl s_server -status_verbose -HTTP -cert fullchain.pem -key privkey.pem Using default temp DH parameters ACCEPT
On the client side:
http --verify=yes https://linuxconfig.ddns.net:4433/index.html On the server side, I get the error message: 140101997737280:error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:../ssl/record/rec_layer_s3.c:1543:SSL alert number 48 On the client side, I get the error message: http: error: SSLError: HTTPSConnectionPool(host='linuxconfig.ddns.net', port=4433): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)'))) while doing GET request to URL: https://linuxconfig.ddns.net:4433/
That error message, CERTIFICATE_VERIFY_FAILED, is an important clue: it means that the certificate’s Certificate Authority (CA) could not be verified. Since the client could not verify the certificate, if failed to make the connection. This is another righteous failure.
The certificate itself could be forged – and the client has no way to know. However, the certificate references a certificate authority (CA), and the CA either knows that the certificate is valid or else it rejects verification. How do we know that the CA is trustworthy?
The CA itself has a certificate, an intermediate certificate, and that certificate references another CA. Eventually, this chain of certificates reaches a root certificate. A root certificate signs itself, and therefore is, by definition, trustworthy. In this case, something has gone wrong with this chain of certificates, this chain of trust.
$ openssl s_client -showcerts -connect linuxconfig.ddns.net:4433 CONNECTED(00000003) depth=0 CN = linuxconfigan.ddns.net verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 CN = linuxconfigan.ddns.net verify error:num=21:unable to verify the first certificate verify return:1 --- Certificate chain 0 s:CN = linuxconfigan.ddns.net i:C = US, O = Let's Encrypt, CN = R3 -----BEGIN CERTIFICATE-----
I know my production server is working properly. This is what the chain is supposed to look like (note the port number 443, not 4433):
$ openssl s_client -showcerts -connect linuxconfig.ddns.net:443 CONNECTED(00000003) depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = R3 verify return:1 depth=0 CN = linuxconfig.ddns.net verify return:1 --- Certificate chain 0 s:CN = linuxconfig.ddns.net i:C = US, O = Let's Encrypt, CN = R3 -----BEGIN CERTIFICATE----- MIIFYjCCBEqgAwIBAgISA0MTOSmISSsIyRls8O/2XpAaMA0GCSqGSIb3DQEBCwUA ... -----END CERTIFICATE----- 1 s:C = US, O = Let's Encrypt, CN = R3 i:C = US, O = Internet Security Research Group, CN = ISRG Root X1 -----BEGIN CERTIFICATE---- ... -----END CERTIFICATE----- 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1 i:O = Digital Signature Trust Co., CN = DST Root CA X3 -----BEGIN CERTIFICATE----- …
There are two ways to proceed from here: I can turn off certificate verification or I can add Let’s Encrypt’s certificate to the list of known CAs. Turning off verification is fast and safe. Adding the CA to the list of known CAs is more arcane. Let’s do both. On the server side, I haven’t touched a thing. On the client side, I turn off verification and I get:
$ http –verify=no https://linuxconfig.ddns.net:4433/index.html
http: error: ConnectionError: ('Connection aborted.', BadStatusLine('\n')) while doing GET request to URL: https://linuxconfig.ddns.net:4433/index.html
$ echo $?
1
This error message tells me that there’s been a violation of the HTTP (not HTTPS) protocol. The server served the first line of the file, index.html, when it should have returned an HTTP return header block. This is a server side flaw, and it would break all HTTP clients. A careful look at the documentation tells me to use the -WWW (not -www) option with openssl, instead of the -HTTP option. I do that:
openssl s_server -status_verbose -WWW -cert fullchain.pem -key privkey.pem and it works properly, with the caveat that I haven’t gotten certificate validation to work yet.
$ http -verify=no https://linuxconfig.ddns.net:4433/helloworld.c
HTTP/1.0 200 ok
Content-type: text/plain
#include
int main (int argc, char *argv[]) {
printf("Hello, world\n\n");
}
Since I used -verify=no, this is actually an erroneous pass.
To verify that my certificate chain is valid, I can use the openssl verify command:
$ openssl verify -purpose sslserver fullchain.pem CN = linuxconfig.ddns.net error 20 at 0 depth lookup: unable to get local issuer certificate error cert.pem: verification failed
The quick solution was to try the openssl s_server command on my production web server, using production configuration files. This is (reasonably) safe to do because the openssl server will run on port 4433 while my production server is running on port 443.
# openssl s_server -status_verbose -WWW \ -cert /etc/letsencrypt/live/linuxconfig.ddns.net/fullchain.pem \ -key /etc/letsencrypt/live/linuxconfig.ddns.net/privkey.pem -accept 4433
Hmm. Nginx is working like a champ. openssl is not. This is why openssl makes for a better test bed than nginx: if nginx’s configuration is wrong, it will try to muddle through. If openssl’s configuration is wrong, it will call you on it. openssl’s configuration is stored in /etc/ssl/openssl.cnf.
It says that the CA certs are in /etc/ssl/certs . The Internet Services Research Group (ISRG)’s root certificate is there. But let’s encrypt’s intermediate cert is not. That makes sense in a way: Let’s encrypt has a wonderful certbot that knew all about nginx when I ran it, but I didn’t run certbot with openssl, so the let’s encrypt’s cert wasn’t in /etc/ssl/certs/ . I got let’s encrypt’s certificate with the:
$ wget https://letsencrypt.org/certs/lets-encrypt-r3.pem
The above command, copied the file lets_encrypt_r3.pem into /etc/ssl/certs/ , ran the the c_rehash program, and voila:
# openssl verify -CApath /etc/ssl/certs/ \ /etc/letsencrypt/live/linuxconfig.ddns.net/fullchain.pem /etc/letsencrypt/live/linuxconfig.ddns.net/fullchain.pem: OK
That’s nice, but the test is, can I see helloworld.c?
$ http --verify=yes https://linuxconfig.ddns.net:4433/helloworld.c
HTTP/1.0 200 ok
Content-type: text/plain
#include
int main (int argc, char *argv[]) {
printf("Hello, world\n\n");
}
Yes. I have now verified that my working HTTPS client will righteously pass and righteously fail, at least for the test cases I worked with. There are some other things that go wrong with SSL/TLS such as Certificate Revocation Lists (CRLs), but I hope you get a good idea.
Next, I want to verify that files sent between the openssl HTTPS server and my HTTPS client will not be corrupted, not even one bit. I can’t verify that every file will be transmitted without error, but what I can do is transmit a large binary file, verify that it was transmitted correctly, and then infer that large files won’t be corrupted.
I used the ls -lorS command to find a large file, calculated its SHA256 sum, transmitted it using openssl as the server, saved the received file, and calculate the SHA256 sum on that file. The SHA 256 sums should match.
On the server side:
$ ls -lorS | tail -1 -rw-rw-r-- 1 jeffs 121329853 May 23 2020 CybersecurityEssentials.pdf $ sha256sum CybersecurityEssentials.pdf 49a49c8e525a3d6830fce1c1ee0bfce2d3dd4b000eeff5925b074802e62024e0 CybersecurityEssentials.pdf
On the client side:
$ http --verify=no https://linuxconfig.ddns.net:4433/CybersecurityEssentials.pdf -o /tmp/CybersecurityEssentials.pdf $ sha256sum /tmp/CybersecurityEssentials.pdf 49a49c8e525a3d6830fce1c1ee0bfce2d3dd4b000eeff5925b074802e62024e0 /tmp/CybersecurityEssentials.pdf
That PDF file is 121MB, big enough for my purposes. The SHA256 sums match, so the file was transmitted properly.
Conclusion
In this article, I described the common failure modes the HTTPS protocol. I used some criteria for selecting an HTTPS server to use for testing an HTTPS client, and selected openssl. I selected an easy to use HTTPS client. I showed some common failure modes, and observed that the client detected those failures.
The hard part was configuring openssl properly, so I showed what can go wrong and how to fix it. Finally, I demonstrated that, using openssl as a server and my HTTPS client, I could transmit a file without data corruption.
