A few weeks ago I wrote about the new Certificate Manager in Domino 12, which enabled Domino 12 to request and automatically update LetsEncrypt certificates and implemented a better way of Server Name Indication (previously introduced in Domino 11.0.1), so you can use different SSL certificates for different websites without needing multiple IP addresses. The Certificate Manager also allows you to use the most recent (ECDSA) ciphers. The lack of this functionality in previous versions of Domino was an important reason why, in many Domino installations, an Nginx, Apache or IHS server is placed in front of the Domino HTTP task as a reverse proxy. There was however another reason: Domino used a lot of cpu power for and was rather slow to decrypt and encrypt SSL traffic. Letting Nginx/Apache/IHS offload the SSL de-/encryption task, reduced total load on the server and sped up performance. I therefore wondered if HCL also managed to solve this problem.
To test this performance I created the following setup:
To remove as many variables as possible, I perform my tests from a Linux server that contains both Domino (running in a container through Podman) and an Nginx installation. In my DNS server I configured 2 url’s. Url 1 has the IP address of the host server and will therefore be handled by the Nginx server, which will do the SSL offloading and proxy the request to Domino. Url 2 has the IP address of the Domino server and will therefore be handled straight by the Domino server. Both urls end up at the default homepage.nsf that comes with Domino 12. The tests are performed using the Apache Benchmarking tool which hits Domino with 25.000 requests. This tool is run on the same server. I used version 2.3 Revision 1843412 of this tool.
I used this command for my tests:
ab -n 25000 -c 1000 https://url1.martdj.nl:443/homepage.nsf
followed by the same command for url2. The results were:
Server Software: nginx Server Hostname: url1.martdj.nl Server Port: 443 SSL/TLS Protocol: TLSv1.2,ECDHE-ECDSA-AES256-GCM-SHA384,256,256 Server Temp Key: X25519 253 bits TLS Server Name: url1.martdj.nl Document Path: /homepage.nsf Document Length: 4788 bytes Concurrency Level: 1000 Time taken for tests: 28.953 seconds Complete requests: 25000 Failed requests: 138 (Connect: 0, Receive: 0, Length: 138, Exceptions: 0) Non-2xx responses: 138 Total transferred: 127735392 bytes HTML transferred: 119064924 bytes Requests per second: 863.48 [#/sec] (mean) Time per request: 1158.103 [ms] (mean) Time per request: 1.158 [ms] (mean, across all concurrent requests) Transfer rate: 4308.48 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 1 26 82.8 4 510 Processing: 7 1119 339.8 1021 2362 Waiting: 2 1118 341.1 1021 2361 Total: 24 1145 322.6 1025 2505
The same test for url2 gave this result:
Server Software: Lotus-Domino Server Hostname: url2.martdj.nl Server Port: 443 SSL/TLS Protocol: TLSv1.2,ECDHE-ECDSA-AES256-GCM-SHA384,256,256 Server Temp Key: X25519 253 bits TLS Server Name: url2.martdj.nl Document Path: /homepage.nsf Document Length: 4788 bytes Concurrency Level: 1000 Time taken for tests: 29.823 seconds Complete requests: 25000 Failed requests: 0 Total transferred: 127600000 bytes HTML transferred: 119700000 bytes Requests per second: 838.27 [#/sec] (mean) Time per request: 1192.938 [ms] (mean) Time per request: 1.193 [ms] (mean, across all concurrent requests) Transfer rate: 4178.23 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 2 376 56.9 367 764 Processing: 4 800 112.0 792 1505 Waiting: 1 787 109.6 779 1505 Total: 220 1175 128.6 1163 1874
As you can see from these result, both times it took about 30 seconds to process 25.000 requests. Which one was the fastest would differ between runs, but they would always end up within a few seconds of each other. It’s safe to say that there’s no notable difference in performance between going straight to Domino or using Nginx as an SSL offloader. During the runs, I would monitor the cpu load of the server. Here are the graphs:
As you can see there’s a difference between cpu cycles in system space and cpu cycles in user space between both solutions, but the overall cpu load is about the same. The tiny advantage for the nginx solution can be explained by the fact that it also took about 4% longer in these explicit runs, so also here Domino’s SSL decryption works as well as using Nginx as an SSL offloader.
The Real Performance Killer
When I started my tests I initially got very different results. The SSL offloading solution through SSL would be a lot slower than Domino. Check this result:
Server Software: nginx Server Hostname: url1.martdj.nl Server Port: 443 SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256 Server Temp Key: ECDH P-384 384 bits TLS Server Name: url1.martdj.nl Document Path: /homepage.nsf Document Length: 4788 bytes Concurrency Level: 1000 Time taken for tests: 62.673 seconds
CPU load was between 64 and 89%, so a lot lower than in the previous results, but check the time it took for 25.000 requests. Just using a different cipher and different Server temp key, it takes more than twice as long for the server to process the same amounts of requests!
The biggest performance killer turned out to be the Server Temp Key. Once I added X25519 in this line
in my Nginx configuration, the test only took between 29-32 seconds. That’s still 5-10% slower than with the ECDSA key. Once I configured Nginx to use the X25519 curve and created a ECDSA SSL certificate, I got the same results as Domino. If I wouldn’t have done these performance tests between Domino with and without Nginx reverse proxy, I would never have realised that I could easily improve my Nginx performance by a staggering 55%!
Domino 12 removes the need to put a reverse proxy in front of your Domino server on the same machine for SSL offloading. There’s no difference in throughput or cpu usage between letting Domino handle the SSL de-/encryption or letting Nginx do this. My tests also showed that the throughput is highly dependent on using the right cipher and curve. A suboptimal configuration can easily halve the amount of requests your server can handle per second. Seeing that Domino, before Domino 12, couldn’t handle the most optimal ciphers and curves, this could be the explanation why Domino’s SSL performance was so poor in previous versions of Domino.