Domino V12 – The Certificate Manager
HCL Domino V12 is in beta, and we currently have beta 2 to work with. One of the interesting new features of Domino V12 is the Certificate Manager task (certmgr). I’ve been playing around with this task and in this post I’ll tell about my experiences.
Let me first say that some work in this area was long overdue. Up to Domino V11, all we had in Domino was the Server Certificate Admin database which dated back to, I think Domino R5. It was completely outdated and I doubt anyone used it anymore. That left the task to request and implement certificates to tools outside of Domino. When we had a certificate from another source that we wanted to use in Domino, we had a command line tool (kyrtool) to convert certificates into a Domino keyring file. Quite a tedious process compared to implementing certificates in web servers like nginx. Also, Domino wasn’t entirely up to date with the latest ciphers. So there was a lot of room for improvement.
In Domino V12, HCL set to fix these problems and design a kick ass certificate process for Domino and I can reveal that they succeeded. The new CertMgr server task combined with the Certificate Store database creates an extremely easy process to fully automate requesting and updating certificates and change Domino from the ugly duckling of certificate management into a best-of-breed product. HCL Lifetime ambassador Daniel Nashed was involved in the design and he has helped me a lot with my configuration (thanks Daniel!).
The CertMgr task and Certificate Store database are described in the HCL Documentation, so I’m not going to duplicate that, but I would like to show my journey on requesting a LetsEncrypt wildcard entry through the Certificate Store.
Configuring the CertMgr
You can run the CertMgr as a program document with specific command line options or add it to the servertasks line in the notes.ini combined with specific notes.ini lines. I chose the latter option. There are my notes.ini settings:
CertMgr_ACCEPT_TOU=1
CertMgr_AutoConfigHttp=1
CertMgr_AutoConfig=1
CertMgr_AutoRequestCert=1
CertMgr_RestartHttp=1You should probably consider whether you want to automatically restart your web server the moment your certificate was renewed (you should have about a month for that if you work with LetsEncrypt certificates, so there’s no need to hurry), but otherwise these are pretty standard (update: in Domino V12.0 it’s no longer necessary to restart the HTTP task to make a new certificate active, so the RestartHttp notes.ini parameter is now deprecated).
The Certificate Store
The Certificate store is the administration database for all certificate related stuff. On the first time opening after creation, you need to configure the Global Configuratiofn. Just choose ECDSA as Key Algorithm and NIST P-256 as Curve Name. After that you can start.
Although you can use the database to create certificate requests for other certificate authorities, the nice part for me is the built-in ACME tooling that allows you to work with LetsEncrypt. LetsEncrypt can use 2 methods to validate that you’re authorised to request certificates for a specific url:
HTTP-01 challenge
This is the default challenge. If you want to use this verification you need to make sure that your websites will respond to the challenges. You do this by adding a dsapi filter the Internet Site document belonging to that website (configuration tab – DSAPI filters as explained here). You also need to set a notes.ini parameter and restart your HTTP task.
When you’re done, you can create a TLS Credentials document.
The Host names field should contain the Common name of your certificate and any Subject Alternative Names (SANs). If you want to create a LetsEncrypt certificate then ACME is your Certificate provider. When you’re still experimenting with the database, it’s good to use LetsEncryptStaging as your ACME account as you’re only allowed to do 50 certificate requests per week on production. The Key type and Curve name are taken from your Global Settings. The value in the Common name is actually not used, but might be necessary anyway. I haven’t tested that to be honest.
This is all you have to enter. When done, just press ‘Submit Request’ to change the status to “pending” and save the document. The CertMgr task should pick up your documents pretty quickly. You can check in your Domino console for errors from this task to see if anything went wrong. If everything went right, you’ll get a green light (or a yellow light if you used the Staging provider) and the expiry date for your new certificate.
This method worked for me for some websites, but not for all, and I couldn’t really figure out why, though I also didn’t really try to. My reverse proxy in front of Domino might have been a complicating factor, but the other reason why I didn’t push this, was that I tend to use a wildcard certificate for my domains, and you can’t request a wildcard certificate through an HTTP-01 challenge.
DNS-01 challenge
To request a wildcard certificate, you need to use a DNS-01 challenge. For this type of challenge, LetsEncrypt will provide a challenge string which has to be made available as a DNS TXT record in the DNS of your domain (or in a forwarded DNS, but that’s out of scope for this post). If the domain is mail.martdj.nl, the TXT record should have the name _acme-challenge.mail.martdj.nl.
Of course, if you want to automate the process of requesting and renewing certificates through the method, you need to be able to add and remove these DNS records automatically and this is the tricky part. Not all DNS providers have an API for adding/editing/deleting DNS records and the ones that do (the large majority) all seem to have created their own API. This means there’s no standard way to create these records. A well-known script that tries to automate the requesting of LetsEncrypt certificates through DNS-01 challenge is the acme.sh script. They tried to create scripts for most DNS providers they could find, which lead to 129(!) API scripts. These however didn’t cover the API of my DNS provider. Writing my own API to talk to the API of my DNS provider has been high on my To-Do list for a very long time, but I never got to it. The availability of a programming interface as part of the new certificate store was a great motivation to finally pick up on this.
I found the process to choose between HTTP-01 and DNS-01 challenges not that clear, but once it’s explained it’s easy enough. Whether the CertMgr task uses an HTTP-01 or a DNS-01 challenge depends on whether a DNS Provider was configured and enabled for the domain of the host name for which you request a certificate. If you have a DNS Provider document for the registered domain materials.acme.org and your request a certificate for www.materials.acme.org, CertMgr will use a DNS-01 challenge, but if you request a certificate for www.acme.org, CertMgr will try to use a HTTP-01 challenge. To request a certificate through an HTTP-01 challenge for a hostname with a domain for which you created a DNS Provider, you need to disable this provider first, before you submit the request. I’m not sure what will happen in this case with the automatic renewals if you re-enable the DNS provider afterwards.
Creating the DNS Configuration document
Before you create a DNS Provider document, you should create or import a DNS Configuration document first. The import option is an interesting one. Daniel envisioned a place where users will share their DNS configuration documents, so there would be a central place where configurations for as many DNS Providers as possible can be found, much like the provider API scripts for acme.sh. He created an easy way for this to export and import documents in the tool. Under Actions in your menu you’ll find the possibility to export selected documents as DXL or import DXL documents from your file system.
If a DNS configuration for your provider doesn’t exist yet, you’ll have to create one yourself. Currently in beta 2, you have 3 methods for this:
- HTTP Requests
- LotusScript agent
- Command
Of these, the last one will probably be removed in the final product as it adds extra complexity and you won’t be able to do more with it than with the other two.
Happy to dust off my LotusScript skills, I immediately started to write a LotusScript class that contained all methods of the API of my DNS provider, combined with an agent that used this class to create the DNS TXT record. Daniel convinced me however that if there was a way to get it done with HTTP Requests, this method was the preferred method as it provides the best integration and easiest debug options. I therefore took another look at the way my DNS provider implemented their API to see if I could use that API through HTTP requests. I figured out I could.
This is my provider configuration. Many DNS providers provide a REST API that returns JSON. My provider doesn’t. They return XML. However, if you regard the XML simply as one big string and use @formula to get the values you need, the XML is not a problem.
It’s easiest to create a script if you think in the same conceptual way that the tool was designed. As I always renewed my wildcard certificates manually, I had a _acme-challenge TXT record in my DNS which I updated with the string I received while renewing my certificate. In script that would entail finding the right index for the _acme-challenge entry and updating it. My provider’s API contains a getrecord and editrecord command, so this is a possible way. It’s however not the concept that Daniel had in mind. From his perspective, you add a DNS TXT record for the challenge when the tool gets the challenge string from LetsEncrypt and you delete it after the validation was done. This does make sense. For me that meant deleting the record I had manually to start with a clean slate and then creating the right formulas to add a request and delete a request.
The flexibility that @Formula provides in combination with the availability of all the fields you could need, make this easy as soon as you grasp the concept. An overview of the available fields:
| Field name | Explanation |
| cfg_AuthKey | Value of DNS Provider document – Authorization key. Useful when you authenticate with your DNS Provider through an authorization key/token |
| cfg_AuthToken | Value of DNS Provider document – Authorization token |
| cfg_CustomValue | Value of DNS Provider document – Custom value. Can be anything |
| cfg_DnsZone | Value of DNS Provider document – DNS zone |
| cfg_InternetAddress | Value of DNS Provider document – Email address |
| cfg_Password | Value of DNS Provider document – Password |
| cfg_URL | Value of DNS Provider Configuration document – Request URL |
| cfg_UserName | Value of DNS Provider document – User Name. Useful when you authenticate to your DNS Provider’s API using username/password |
| param_DnsTxtName | Value for DNS TXT record name. e.g. _acme-challenge.martdj.nl |
| param_DnsTxtValue | Token of the ACME challenge which needs to be added to the DNS TXT record |
| param_Hostname | 1st value of TLS Credentials document – Host names field |
| param_RegisteredDomain | Value of DNS Provider document – Registered domain |
| ret_AddResult | Text result from the HTTP Add Request |
| ret_DelResult | Text result from the HTTP Delete Request |
| ret_AddStatus | HTTP Status from the HTTP Add Request |
| ret_DelStatus | HTTP Status from the HTTP Delete Request |
| retJSON_Add.result | JSON result from the HTTP Add Request (if your DNS Provider’s API returns JSON) |
| retJSON_Lookup.result | JSON result from the HTTP Lookup Request |
Above list is not conclusive. Depending on your configuration there are far more fields available. The way to find these fields is by setting HTTP request tracing in the DNS Provider Configuration document to ‘Enabled’ and checking the DNS trace logs that result from requesting certificates with an enabled DNS-01 challenge provider. For example url_add, url_del, post_add etc. are also available.
Creating the DNS Provider document
The DNS Provider document will link your DNS Provider Configuration to a domain. Where the configuration document contains all the code for your DNS Provider’s API, the Provider document contains your personal settings, like username/passwords or authorization token. The value in the Registered Domain field will decide whether CertMgr will use a DNS-01 challenge based on this configuration for a requested domain or not. Below document is used for all domain names ending with .martdj.nl. Through the status field you can enable or disable a DNS Provider, while in DNS Configuration field you select which DNS Configuration document should be used for this domain. The other fields correspond to values that are used in your DNS Provider Configuration formulas.
Testing your configuration
When you’ve created a DNS Provider Configuration document and DNS Provider document, it’s time to test. Make sure your HTTP Request tracing is set to Enabled in your DNS Provider Configuration document and create a TLS Credential document for a hostname that ends with the Registered domain of your DNS Provider document. Check the DNS trace logs for the result and adjust your configuration where needed. If everything works, you can change your HTTP Request tracing to only produce trace logs on errors.
Conclusion
The CertMgr task combined with Certificate Store provides a very flexible way to request or import certificates. Especially the process to request LetsEncrypt certificates through either HTTP-01 challenges or DNS-01 challenges has been well worked out and is very flexible, but it also has become a lot easier to add existing certificates to Domino. Domino V12 switched to the far more standard PEM format for certificates, though the old Domino keyring files can also still be used for backwards compatibility reasons.
There’s one element still missing that prevents Domino from becoming the central store for all your company’s certificates: An automated export mechanism for the certificates. Although you can export your certificates by copy/pasting the PEM keys from your TLS certificate documents, I want an automatic way, so I can also update my Nginx servers, Apache servers and IBM HTTP Servers certificates in an automated way. Hopefully this will be incorporated in the final Domino V12.
Update
Above article was written for Domino V12 beta 2. In Domino V12 beta 3, HCL implemented a number of new features for the Certificate Manager among which an updated TLS Credential cache. This cache has the advantage that “you don’t need to restart any internet tasks like HTTP or SMTP to add or update a certificate”.
References
Automating TLS Certificate Management with Domino V12
Domino V12 Let’s Enrypt DNS-01 Challenges delegating a sub-domain to Digital Ocean
Great blog post! 🙂
I agree on the exporting issue. Many are using a reverse proxy in front because Domino was not modernized.
I use Nginx with Domino. LE4D is used to create certificates and I then have a application on Domino which exports the certificates to the Nginx servers. This work very well, but an export feature is badly needed in V12.
Thanks, Jesper, for your kind comment. I have the same situation where we installed Nginx or Apache servers as reverse proxy’s with SSL offloading in front of Domino at quite a few customers. Regarding the automated export, please have a look at, and, if you have the time, please participate in this discussion in the V12 beta forum: https://registration.hclpartnerconnect.com/d12betaforum.nsf/topicThread.xsp?documentId=4CD6. Current outlook is that automated export will arrive in 12.0.1.
Hi Martin — thanks for this detailed post. I sell a product (CertMatica) that can renew Let’s Encrypt certificates over DNS challenges but the process does contain manual steps due to the absence of any uniform APIs (if any) among hosting providers. I took a look at the acme.sh script you mention — 129 APIs and counting — whew! While it certainly seems useful, supporting it (commercially) could be interesting!
I’m curious about the customers that front-end Domino with Nginx — are they using one or more centralized Nginx servers or is the Nginx service typically installed locally on the Domino machine? I don’t currently have this feature in the product but if folks find it useful and HCL won’t be offering it for another six months or more, I’d certainly consider adding it to the product.
Best,
Richard
Domino customers with Nginx or Apache frontend typically have these installed on the system itself and use HTTP-01 challenges to get their certificates renewed. They typically do this using certbot or acme.sh which, of course, both are free. For Domino as webserver there’s currently another free product: LE4D – https://www.midpoints.de/de-solutions-LE4D.
For me the interesting part of the Domino certificate manager is the ease with which it allows one to manage your certificates, both through HTTP-01 challenges and DNS-01 challenges and its possibility to become a central store for companywide certificates (using DNS-01 challenges). For the latter, automated export mechanisms will be key though.
Thanks for your response Martijn. I’m not familiar with the workings of the LE4D product and I haven’t had a chance to test CertMgr on Domino v12 Beta yet but in addition to importing LE certificates into .kyr files, CertMatica also keeps a copy of the certificates on the server’s file system. As a result, CertMatica can already be used with Nginx as long as you update the proper file name directives in the Nginx configuration file (I just verified this with Nginx 1.18.0 on Windows).
Nonetheless, I see the value in the ability to export/copy certificates to alternate locations (and perhaps other formats) and am considering implementing that in an upcoming release. This way, users who can’t immediately migrate to Domino 12 will have another option.
Thanks again for your thoughts,
Richard