Domino 12 – SSL Performance

Posted on by

A few weeks ago I wrote about the new Certificate Manager in Domino 12, which enabled Domino 12 to request and automatically update LetsEncrypt certificates and implemented a better way of Server Name Indication (previously introduced in Domino 11.0.1), so you can use different SSL certificates for different websites without needing multiple IP addresses. The Certificate Manager also allows you to use the most recent (ECDSA) ciphers. The lack of this functionality in previous versions of Domino was an important reason why, in many Domino installations, an Nginx, Apache or IHS server is placed in front of the Domino HTTP task as a reverse proxy. There was however another reason: Domino used a lot of cpu power for and was rather slow to decrypt and encrypt SSL traffic. Letting Nginx/Apache/IHS offload the SSL de-/encryption task, reduced total load on the server and sped up performance. I therefore wondered if HCL also managed to solve this problem.

Test Setup

To test this performance I created the following setup:

To remove as many variables as possible, I perform my tests from a Linux server that contains both Domino (running in a container through Podman) and an Nginx installation. In my DNS server I configured 2 url’s. Url 1 has the IP address of the host server and will therefore be handled by the Nginx server, which will do the SSL offloading and proxy the request to Domino. Url 2 has the IP address of the Domino server and will therefore be handled straight by the Domino server. Both urls end up at the default homepage.nsf that comes with Domino 12. The tests are performed using the Apache Benchmarking tool which hits Domino with 25.000 requests. This tool is run on the same server. I used version 2.3 Revision 1843412 of this tool.

The Test

I used this command for my tests:

ab -n 25000 -c 1000 https://url1.martdj.nl:443/homepage.nsf

followed by the same command for url2. The results were:

Server Software:        nginx
 Server Hostname:        url1.martdj.nl
 Server Port:            443
 SSL/TLS Protocol:       TLSv1.2,ECDHE-ECDSA-AES256-GCM-SHA384,256,256
 Server Temp Key:        X25519 253 bits
 TLS Server Name:        url1.martdj.nl
 Document Path:          /homepage.nsf
 Document Length:        4788 bytes
 Concurrency Level:      1000
 Time taken for tests:   28.953 seconds
 Complete requests:      25000
 Failed requests:        138
    (Connect: 0, Receive: 0, Length: 138, Exceptions: 0)
 Non-2xx responses:      138
 Total transferred:      127735392 bytes
 HTML transferred:       119064924 bytes
 Requests per second:    863.48 [#/sec] (mean)
 Time per request:       1158.103 [ms] (mean)
 Time per request:       1.158 [ms] (mean, across all concurrent requests)
 Transfer rate:          4308.48 [Kbytes/sec] received
 Connection Times (ms)
               min  mean[+/-sd] median   max
 Connect:        1   26  82.8      4     510
 Processing:     7 1119 339.8   1021    2362
 Waiting:        2 1118 341.1   1021    2361
 Total:         24 1145 322.6   1025    2505

The same test for url2 gave this result:

 Server Software:        Lotus-Domino
 Server Hostname:        url2.martdj.nl
 Server Port:            443
 SSL/TLS Protocol:       TLSv1.2,ECDHE-ECDSA-AES256-GCM-SHA384,256,256
 Server Temp Key:        X25519 253 bits
 TLS Server Name:        url2.martdj.nl
 Document Path:          /homepage.nsf
 Document Length:        4788 bytes
 Concurrency Level:      1000
 Time taken for tests:   29.823 seconds
 Complete requests:      25000
 Failed requests:        0
 Total transferred:      127600000 bytes
 HTML transferred:       119700000 bytes
 Requests per second:    838.27 [#/sec] (mean)
 Time per request:       1192.938 [ms] (mean)
 Time per request:       1.193 [ms] (mean, across all concurrent requests)
 Transfer rate:          4178.23 [Kbytes/sec] received
 Connection Times (ms)
               min  mean[+/-sd] median   max
 Connect:        2  376  56.9    367     764
 Processing:     4  800 112.0    792    1505
 Waiting:        1  787 109.6    779    1505
 Total:        220 1175 128.6   1163    1874

As you can see from these result, both times it took about 30 seconds to process 25.000 requests. Which one was the fastest would differ between runs, but they would always end up within a few seconds of each other. It’s safe to say that there’s no notable difference in performance between going straight to Domino or using Nginx as an SSL offloader. During the runs, I would monitor the cpu load of the server. Here are the graphs:

As you can see there’s a difference between cpu cycles in system space and cpu cycles in user space between both solutions, but the overall cpu load is about the same. The tiny advantage for the nginx solution can be explained by the fact that it also took about 4% longer in these explicit runs, so also here Domino’s SSL decryption works as well as using Nginx as an SSL offloader.

The Real Performance Killer

When I started my tests I initially got very different results. The SSL offloading solution through SSL would be a lot slower than Domino. Check this result:

Server Software:        nginx
 Server Hostname:        url1.martdj.nl
 Server Port:            443
 SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
 Server Temp Key:        ECDH P-384 384 bits
 TLS Server Name:        url1.martdj.nl
 Document Path:          /homepage.nsf
 Document Length:        4788 bytes
 Concurrency Level:      1000
 Time taken for tests:   62.673 seconds

CPU load was between 64 and 89%, so a lot lower than in the previous results, but check the time it took for 25.000 requests. Just using a different cipher and different Server temp key, it takes more than twice as long for the server to process the same amounts of requests!

The biggest performance killer turned out to be the Server Temp Key. Once I added X25519 in this line

ssl_ecdh_curve X25519:secp384r1;

in my Nginx configuration, the test only took between 29-32 seconds. That’s still 5-10% slower than with the ECDSA key. Once I configured Nginx to use the X25519 curve and created a ECDSA SSL certificate, I got the same results as Domino. If I wouldn’t have done these performance tests between Domino with and without Nginx reverse proxy, I would never have realised that I could easily improve my Nginx performance by a staggering 55%!

Conclusion

Domino 12 removes the need to put a reverse proxy in front of your Domino server on the same machine for SSL offloading. There’s no difference in throughput or cpu usage between letting Domino handle the SSL de-/encryption or letting Nginx do this. My tests also showed that the throughput is highly dependent on using the right cipher and curve. A suboptimal configuration can easily halve the amount of requests your server can handle per second. Seeing that Domino, before Domino 12, couldn’t handle the most optimal ciphers and curves, this could be the explanation why Domino’s SSL performance was so poor in previous versions of Domino.