Using Domino as central source for certificates for Nginx/Apache and IBM HTTP Server

In the last 2 articles, I showed how you can use HCL Domino combined with DNS Configurations to request wildcard certificates. Wildcard certificates are extremely handy as you can use them for every single spot in your organisation where you need a TLS certificate. For this reason, many companies buy a wildcard certificate each year and use this for all their web servers.

If you have to do this once a year, the amount of effort needed, is still limited. If you have to do this every 2 months, however, as would be the case if you use LetsEncrypt certificates, the cost of updating all servers would exceed the cost of a commercial wildcard certificate by a large margin.

So to make the free LetsEncrypt wildcard certificates worthwhile, we need to be able to automate this process. Not just for Domino, which the Certificate Manager does for you, but also for all other webservers. And that’s exactly what I’ll focus on in this post. How to automate the process of how to update those other webservers, where I’ll focus in particular on Nginx/Apache (using pem certificates) and IBM HTTP Server (using a java keystore).

The 2 parts of a certificate

To understand the process to automate replacing the certificates, it’s important to realise that every certificate consists of 2 parts: The private key and the public certificate.

The private key

The private key is created before you can create a Certificate Signing Request (CSR). In the Certificate Store, it’s created automatically when you submit or save a new TLS credential document or when you create an exportable key. The private key doesn’t expire. It’s not a certificate in itself, after all. So you only need to put this private key on your web servers once.

To be able to get the private key from the Certificate Store to put it on your web servers, it’s essential that you create an exportable key from the start. So after you added TLS Credentials and before you pressed Submit, use the “Create Exportable Key” button.

After clicking the button, you have to enter a password for your private key, and it has to be a strong password. “Welcome01” and the likes won’t do. It’s nice that Domino is security conscious, but seeing that we’ll probably put the private key unencrypted on our Nginx/Apache servers, I think it’s a bit much.

When you set your password, you can enter the host names, servers with access etc and submit.

The public certificate

The public certificate is what we get back from LetsEncrypt after it validated our CSR, and it consists of multiple parts:

• The server or wildcard certificate
• One or more intermediary certificates
• The root certificate (optional)

For LetsEncrypt certificates, this server/wildcard part expires after 3 months and is usually renewed every 2 months, so this is the part that we need to distribute in an automated way in the right format to all web servers. The intermediary’s and optional root certificate don’t expire for many years, but usually come in one package with the server/wildcard certificate.

OpenSSL – the essential tool

To extract the private key, grab the certificates and convert everything to the format we need, we’ll use OpenSSL. OpenSSL is installed by default on every Linux server, so need to install anything there, but if you want to do these actions on Windows, you’ll have to install OpenSSL there. In the rest of this post, I’m going to assume that you’ll use a Linux server, as most web servers are running on Linux anyway.

Extracting the private key from Domino

If you created an exportable key as described earlier, you can export the private key using the button on the document.

While you’re here, also take note of your key type. It’s either ECDSA or RSA. I recommend ECDSA, as it’s better in basically every way, with the only “downside” that IE6 and IE7 can’t handle it. The type will have an effect on the exact OpenSSL commands to use, so know what type of certificate you’ve created.

I always export the certificate as PKCS12, but you can also export it as Base64 encoded X.509. All the commands and scripts below are based on PKCS12. The “friendly name” is useful to know at a later point in time what certificate that file on your file system holds. You can find it back in the certificate itself. The current password is the password that you set when you created the exportable key. The new password has to be strong as well.

Converting your exported key to pem

Now that you have exported your Domino key, you can convert it to the pem-format that Nginx and Apache expect, but which we’ll also use IBM HTTP Server. We’ll convert both the key and the certificate, though you really only need the private key, as we’ll get the certificate in a different way later. The commands to extract the keys:

openssl pkcs12 -in <your exported p12/pfx file> -nodes -nocerts -out tempkey.pem
openssl <ec|rsa> -in tempkey.pem -out privkey.pem

You will be asked for the password you entered during the export from Domino. In the second line you have to either enter “ec” or “rsa” depending on whether your key is an ECDSA key or an RSA key. This should give a file privkey.pem. When you open the file, you will see something like:

-----BEGIN EC PRIVATE KEY-----
MHcCAQEEILRovPVUGt9/FUFfuJLupxzKH2EJOGzHq4HTBhM13uoNoAoGCCqGSM49
AwEHoUQDQgAE2ILI7yaqJ8QN49mIOcruuq4AXRvnJVJA9k+3D/QXPFiUSw3PTLX9
12EnU123kYoNv8thhNA5W1IjK2dBS1WLPw==
-----END EC PRIVATE KEY-----

At this stage, this file is no longer password protected, so make sure you prevent unauthorised access to it.

You can extract the certificate from the pfx using this command:

openssl pkcs12 -info -in <your exported p12/pfx file> -nokeys -out cert.pem

This will give you a file cert.pem with the certificate. In your nginx configuration files, you can use these file by adding these lines:

ssl_certificate /etc/nginx/ssl/cert.pem;
ssl_certificate_key /etc/nginx/ssl/privkey.pem;

Getting the certificate from Domino

It’s nice to export the certificate from the Domino Certificate store and extract the certificate from there, but this needs manual steps, which you don’t want to do every 2 months when the certificate gets automatically renewed. Luckily, we can get this certificate straight from the CertMgr and HTTP task by using openssl to request the certificate. The command to do this is:

openssl s_client -servername $url -showcerts -connect $serverip:$port /dev/null | sed -ne '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' > $certfile

Let’s have a look at the variables in this command.

Variable	Goal
$url	The hostname for which you want the certificate. If you want to get the wildcard certificate, you have to use a URL that does not have a certificate of its own assigned in the certificate store
$serverip	The IP address of a Domino web server in your domain
$port	Usually port 443 (TLS), but this can be a custom port
$certfile	The output file with your certificate

Automating the process

Using the openssl command above, we can automate the process of regularly renewing the server certificate. Below is an example of the script I created for my own environment:

#!/bin/bash

# let's define some variables
file1=/etc/nginx/ssl/cert.pem
file2=/etc/nginx/ssl/temp.pem
url=blog.martdj.nl
serverip=192.168.1.100

# and some functions we'll use
replace_certificate()
{
        rm -f $file1
        mv $file2 $file1
}

restart_nginx()
{
        systemctl reload nginx.service
}

#start with getting the certificate
openssl s_client -servername $url -showcerts -connect $serverip:7443  </dev/null 2>/dev/null | sed -ne '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' > $file2

# test if the command went well. If not, the file would be empty
if [ -s $file2 ]; then
        # The file is not-empty. We double check by checking if the file contains some common text for a certificate
        if grep -Fq "CERTIFICATE-----" $file2; then
           # We continue. Let's see if the certificate is the same as the current one
           if ! cmp --silent "$file1" "$file2"; then
                # The file is not-empty.
                replace_certificate
                restart_nginx
                echo "all done"
           else
                # The file is empty.
                rm -f $file2
                echo "Nothing to do"
           fi

        else
           # code if not found. abort
           rm -f $file2
           echo "Nothing to do"
        fi
else
        # The file is empty. remove the file and abort
        rm -f $file2
        echo "Nothing to do"
fi

It’s written for nginx, but the basics would be the same for Apache. I’ve scheduled this script weekly using a cron job. It’s a rather basic script. You could expand it with many more checks. A useful check could, for example, be whether the certificate you download corresponds to the private key which you already have. You would have to compare the output of these 2 lines for that:

openssl x509 -in cert.pem -pubkey -noout
openssl ec -in privkey.pem -pubout

You could also mail or configure other notifications if things are not working as expected. Basically, you’re only limited by your imagination.

Creating the key store for IBM HTTP Server

IBM HTTP Server uses a key database (kdb) for its certificates. The script that I created for renewing this keystore, takes quite a detour. First I grab the pem certificate from Domino. I then put this pem certificate and the private key in a p12 certificate. This certificate can be imported in an IHS Keystore using the gskcapicmd command. I then add the Intermediary and root certificates for LetsEncrypt to this keystore. These certificates can be downloaded from LetsEncrypt. If all went well, I put the new keystore in place and restart IHS.

#!/bin/bash
version=1.2
# let's define some variables
file1=/appl/tools/certs/cert.pem
file2=/appl/tools/certs/temp.pem
url=blog.martdj.nl
serverip=192.168.7.100
keystorepw=A-very-secret-password
############## Functions ##################
remove_keystore()
{
        rm -f /opt/IBM/HTTPServer/ssl/ihskey.*
}

replace_certificate()
{
        rm -f $file1
        mv $file2 $file1
}

create_p12()
{
        openssl pkcs12 -export -out /appl/tools/certs/martdj.nl.cert.p12 -in /appl/tools/certs/cert.pem -inkey /appl/tools/certs/martdj.nl.key -password pass:$keystorepw
}

create_keystore()
{
        # Create a new keystore
        /opt/IBM/HTTPServer/bin/gskcapicmd -keydb -create -db /opt/IBM/HTTPServer/ssl/ihskey.kdb -pw $keystorepw -stash
        /opt/IBM/HTTPServer/bin/gskcapicmd -cert -add -db /opt/IBM/HTTPServer/ssl/ihskey.kdb -pw $keystorepw -file /appl/tools/certs/lets-encrypt-r3.pem -label lets-encrypt-r3
        /opt/IBM/HTTPServer/bin/gskcapicmd -cert -add -db /opt/IBM/HTTPServer/ssl/ihskey.kdb -pw $keystorepw -file /appl/tools/certs/isrgrootx1.pem -label isrgrootx1
        /opt/IBM/HTTPServer/bin/gskcapicmd -cert -import -db /appl/tools/certs/martdj.nl.cert.p12 -pw $keystorepw -target /opt/IBM/HTTPServer/ssl/ihskey.kdb -target_pw $keystorepw
        /opt/IBM/HTTPServer/bin/gskcapicmd -cert -setdefault -label CN=*.martdj.nl -db /opt/IBM/HTTPServer/ssl/ihskey.kdb -stashed
}

show_certs()
{
        # and print the output
        /opt/IBM/HTTPServer/bin/gskcapicmd -cert -list -db /opt/IBM/HTTPServer/ssl/ihskey.kdb -stashed
}

##################### start script #####################
openssl s_client -servername $url:7443 -showcerts -connect $serverip:7443  </dev/null 2>/dev/null | sed -ne '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' > $file2
# test if the command went well. If not, the file would be empty
if [ -s $file2 ]; then
        # The file is not-empty. We double check by checking if the file contains some common text for a certificate
        if grep -Fq "CERTIFICATE-----" $file2; then
           # We continue. Let's see if the certificate is the same as the current one
           if ! cmp --silent "$file1" "$file2"; then
                # The file is not-empty.
                replace_certificate
                remove_keystore
                create_p12
                create_keystore
                show_certs
                echo "all done"
           else
                # The file is empty.
                rm -f $file2
                echo "Nothing to do"
           fi

        else
           # code if not found. abort
           rm -f $file2
           echo "Nothing to do"
        fi
else
        # The file is empty. remove the file and abort
        rm -f $file2
        echo "Nothing to do"
fi

You can also find these files on my Github.

Conclusion

A certificate consists of several parts, of which only the server certificate part (or wildcard in case of a wildcard certificate) regularly changes. We can grab this part of the certificate from the Domino server by using an openssl command. This allows us to script and automate the renewal of the certificates.

This makes the whole certificate renewal process a “set once and forget”-process. Domino automatically takes care of the timely renewal of the certificates and your scripts will pick up the renewed certificates. Domino can warn you if anything goes wrong on that end. If you build in checks and notifications in your scripts, you can also get notified in a timely manner of problems on that end. For me, my scripts did their thing without any problems since I implemented them.