Domino V12 – The Certificate Manager

HCL Domino V12 is in beta, and we currently have beta 2 to work with. One of the interesting new features of Domino V12 is the Certificate Manager task (certmgr). I’ve been playing around with this task and in this post I’ll tell about my experiences.

Let me first say that some work in this area was long overdue. Up to Domino V11, all we had in Domino was the Server Certificate Admin database which dated back to, I think Domino R5. It was completely outdated and I doubt anyone used it anymore. That left the task to request and implement certificates to tools outside of Domino. When we had a certificate from another source that we wanted to use in Domino, we had a command line tool (kyrtool) to convert certificates into a Domino keyring file. Quite a tedious process compared to implementing certificates in web servers like nginx. Also, Domino wasn’t entirely up to date with the latest ciphers. So there was a lot of room for improvement.

In Domino V12, HCL set to fix these problems and design a kick ass certificate process for Domino and I can reveal that they succeeded. The new CertMgr server task combined with the Certificate Store database creates an extremely easy process to fully automate requesting and updating certificates and change Domino from the ugly duckling of certificate management into a best-of-breed product. HCL Lifetime ambassador Daniel Nashed was involved in the design and he has helped me a lot with my configuration (thanks Daniel!).

The CertMgr task and Certificate Store database are described in the HCL Documentation, so I’m not going to duplicate that, but I would like to show my journey on requesting a LetsEncrypt wildcard entry through the Certificate Store.

Configuring the CertMgr

You can run the CertMgr as a program document with specific command line options or add it to the servertasks line in the notes.ini combined with specific notes.ini lines. I chose the latter option. There are my notes.ini settings:

CertMgr_ACCEPT_TOU=1
CertMgr_AutoConfigHttp=1
CertMgr_AutoConfig=1
CertMgr_AutoRequestCert=1
CertMgr_RestartHttp=1

You should probably consider whether you want to automatically restart your web server the moment your certificate was renewed (you should have about a month for that if you work with LetsEncrypt certificates, so there’s no need to hurry), but otherwise these are pretty standard.

The Certificate Store

The Certificate store is the administration database for all certificate related stuff. On the first time opening after creation, you need to configure the Global Configuratiofn. Just choose ECDSA as Key Algorithm and NIST P-256 as Curve Name. After that you can start.

Although you can use the database to create certificate requests for other certificate authorities, the nice part for me is the built-in ACME tooling that allows you to work with LetsEncrypt. LetsEncrypt can use 2 methods to validate that you’re authorised to request certificates for a specific url:

HTTP-01 challenge

This is the default challenge. If you want to use this verification you need to make sure that your websites will respond to the challenges. You do this by adding a dsapi filter the Internet Site document belonging to that website (configuration tab – DSAPI filters as explained here). You also need to set a notes.ini parameter and restart your HTTP task.

 

When you’re done, you can create a TLS Credentials document.

The Host names field should contain the Common name of your certificate and any Subject Alternative Names (SANs). If you want to create a LetsEncrypt certificate then ACME is your Certificate provider. When you’re still experimenting with the database, it’s good to use LetsEncryptStaging as your ACME account as you’re only allowed to do 50 certificate requests per week on production. The Key type and Curve name are taken from your Global Settings. The value in the Common name is actually not used, but might be necessary anyway. I haven’t tested that to be honest.

This is all you have to enter. When done, just press ‘Submit Request’ to change the status to “pending” and save the document. The CertMgr task should pick up your documents pretty quickly. You can check in your Domino console for errors from this task to see if anything went wrong. If everything went right, you’ll get a green light (or a yellow light if you used the Staging provider) and the expiry date for your new certificate.
This method worked for me for some websites, but not for all, and I couldn’t really figure out why, though I also didn’t really try to. My reverse proxy in front of Domino might have been a complicating factor, but the other reason why I didn’t push this, was that I tend to use a wildcard certificate for my domains, and you can’t request a wildcard certificate through an HTTP-01 challenge.

DNS-01 challenge

To request a wildcard certificate, you need to use a DNS-01 challenge. For this type of challenge, LetsEncrypt will provide a challenge string which has to be made available as a DNS TXT record in the DNS of your domain (or in a forwarded DNS, but that’s out of scope for this post). If the domain is mail.martdj.nl, the TXT record should have the name _acme-challenge.mail.martdj.nl.

Of course, if you want to automate the process of requesting and renewing certificates through the method, you need to be able to add and remove these DNS records automatically and this is the tricky part. Not all DNS providers have an API for adding/editing/deleting DNS records and the ones that do (the large majority) all seem to have created their own API. This means there’s no standard way to create these records. A well-known script that tries to automate the requesting of LetsEncrypt certificates through DNS-01 challenge is the acme.sh script. They tried to create scripts for most DNS providers they could find, which lead to 129(!) API scripts. These however didn’t cover the API of my DNS provider. Writing my own API to talk to the API of my DNS provider has been high on my To-Do list for a very long time, but I never got to it. The availability of a programming interface as part of the new certificate store was a great motivation to finally pick up on this.

I found the process to choose between HTTP-01 and DNS-01 challenges not that clear, but once it’s explained it’s easy enough. Whether the CertMgr task uses an HTTP-01 or a DNS-01 challenge depends on whether a DNS Provider was configured and enabled for the domain of the host name for which you request a certificate. If you have a DNS Provider document for the registered domain materials.acme.org and your request a certificate for www.materials.acme.org, CertMgr will use a DNS-01 challenge, but if you request a certificate for www.acme.org, CertMgr will try to use a HTTP-01 challenge. To request a certificate through an HTTP-01 challenge for a hostname with a domain for which you created a DNS Provider, you need to disable this provider first, before you submit the request. I’m not sure what will happen in this case with the automatic renewals if you re-enable the DNS provider afterwards.

Creating the DNS Configuration document

Before you create a DNS Provider document, you should create or import a DNS Configuration document first. The import option is an interesting one. Daniel envisioned a place where users will share their DNS configuration documents, so there would be a central place where configurations for as many DNS Providers as possible can be found, much like the provider API scripts for acme.sh. He created an easy way for this to export and import documents in the tool. Under Actions in your menu you’ll find the possibility to export selected documents as DXL or import DXL documents from your file system.

If a DNS configuration for your provider doesn’t exist yet, you’ll have to create one yourself. Currently in beta 2, you have 3 methods for this:

  1. HTTP Requests
  2. LotusScript agent
  3. Command

Of these, the last one will probably be removed in the final product as it adds extra complexity and you won’t be able to do more with it than with the other two.

Happy to dust off my LotusScript skills, I immediately started to write a LotusScript class that contained all methods of the API of my DNS provider, combined with an agent that used this class to create the DNS TXT record. Daniel convinced me however that if there was a way to get it done with HTTP Requests, this method was the preferred method as it provides the best integration and easiest debug options. I therefore took another look at the way my DNS provider implemented their API to see if I could use that API through HTTP requests. I figured out I could.

This is my provider configuration. Many DNS providers provide a REST API that returns JSON. My provider doesn’t. They return XML. However, if you regard the XML simply as one big string and use @formula to get the values you need, the XML is not a problem.

It’s easiest to create a script if you think in the same conceptual way that the tool was designed. As I always renewed my wildcard certificates manually, I had a _acme-challenge TXT record in my DNS which I updated with the string I received while renewing my certificate. In script that would entail finding the right index for the _acme-challenge entry and updating it. My provider’s API contains a getrecord and editrecord command, so this is a possible way. It’s however not the concept that Daniel had in mind. From his perspective, you add a DNS TXT record for the challenge when the tool gets the challenge string from LetsEncrypt and you delete it after the validation was done. This does make sense. For me that meant deleting the record I had manually to start with a clean slate and then creating the right formulas to add a request and delete a request.

The flexibility that @Formula provides in combination with the availability of all the fields you could need, make this easy as soon as you grasp the concept. An overview of the available fields:

Field nameExplanation
cfg_AuthKeyValue of DNS Provider document – Authorization key. Useful when you authenticate with your DNS Provider through an authorization key/token
cfg_AuthTokenValue of DNS Provider document – Authorization token
cfg_CustomValueValue of DNS Provider document – Custom value. Can be anything
cfg_DnsZoneValue of DNS Provider document – DNS zone
cfg_InternetAddressValue of DNS Provider document – Email address
cfg_PasswordValue of DNS Provider document – Password
cfg_URLValue of DNS Provider Configuration document – Request URL
cfg_UserNameValue of DNS Provider document – User Name. Useful when you authenticate to your DNS Provider’s API using username/password
param_DnsTxtNameValue for DNS TXT record name. e.g. _acme-challenge.martdj.nl
param_DnsTxtValueToken of the ACME challenge which needs to be added to the DNS TXT record
param_Hostname1st value of TLS Credentials document – Host names field
param_RegisteredDomainValue of DNS Provider document – Registered domain
ret_AddResultText result from the HTTP Add Request
ret_DelResultText result from the HTTP Delete Request
ret_AddStatusHTTP Status from the HTTP Add Request
ret_DelStatusHTTP Status from the HTTP Delete Request
retJSON_Add.resultJSON result from the HTTP Add Request (if your DNS Provider’s API returns JSON)
retJSON_Lookup.resultJSON result from the HTTP Lookup Request
Available fields for creating HTTP Request Formulas

Above list is not conclusive. Depending on your configuration there are far more fields available. The way to find these fields is by setting HTTP request tracing in the DNS Provider Configuration document to ‘Enabled’ and checking the DNS trace logs that result from requesting certificates with an enabled DNS-01 challenge provider. For example url_add, url_del, post_add etc. are also available.

Creating the DNS Provider document

The DNS Provider document will link your DNS Provider Configuration to a domain. Where the configuration document contains all the code for your DNS Provider’s API, the Provider document contains your personal settings, like username/passwords or authorization token. The value in the Registered Domain field will decide whether CertMgr will use a DNS-01 challenge based on this configuration for a requested domain or not. Below document is used for all domain names ending with .martdj.nl. Through the status field you can enable or disable a DNS Provider, while in DNS Configuration field you select which DNS Configuration document should be used for this domain. The other fields correspond to values that are used in your DNS Provider Configuration formulas.

Testing your configuration

When you’ve created a DNS Provider Configuration document and DNS Provider document, it’s time to test. Make sure your HTTP Request tracing is set to Enabled in your DNS Provider Configuration document and create a TLS Credential document for a hostname that ends with the Registered domain of your DNS Provider document. Check the DNS trace logs for the result and adjust your configuration where needed. If everything works, you can change your HTTP Request tracing to only produce trace logs on errors.

Conclusion

The CertMgr task combined with Certificate Store provides a very flexible way to request or import certificates. Especially the process to request LetsEncrypt certificates through either HTTP-01 challenges or DNS-01 challenges has been well worked out and is very flexible, but it also has become a lot easier to add existing certificates to Domino. Domino V12 switched to the far more standard PEM format for certificates, though the old Domino keyring files can also still be used for backwards compatibility reasons.

There’s one element still missing that prevents Domino from becoming the central store for all your company’s certificates: An automated export mechanism for the certificates. Although you can export your certificates by copy/pasting the PEM keys from your TLS certificate documents, I want an automatic way, so I can also update my Nginx servers, Apache servers and IBM HTTP Servers certificates in an automated way. Hopefully this will be incorporated in the final Domino V12.

References

Automating TLS Certificate Management with Domino V12

Domino V12 Let’s Enrypt DNS-01 Challenges delegating a sub-domain to Digital Ocean

My bhosted.dxl DNS Provider Configuration file