Protecting your Domino container with fail2ban
If your Domino server is connected to the Internet, you’ll find that bots (hacked systems running a script) will throw a brute force attack on your Domino server. For me, especially, my SMTP server was under heavy attack. I actually couldn’t even see normal log entries between all these entries:
[000738:000012-00007F1164B21700] 05-11-2022 14:18:50 SMTP Server: Authentication failed for user honors ; connecting host 5.34.207.55
[000738:000016-00007F1164B21700] 05-11-2022 14:18:51 SMTP Server: 5.34.207.55 connected
[000738:000013-00007F1164C23700] 05-11-2022 14:18:51 SMTP Server: 5.34.207.52 disconnected. 0 message[s] received
[000738:000013-00007F1164B21700] 05-11-2022 14:18:51 SMTP Server: 5.34.207.52 connected
[000738:000012-00007F1164C23700] 05-11-2022 14:18:52 SMTP Server: 5.34.207.55 disconnected. 0 message[s] received
[000738:000010-00007F1164B21700] 05-11-2022 14:18:52 SMTP Server: Authentication failed for user protocollo@martdj.nl ; connecting host 2.57.122.207
[000738:000010-00007F1164C23700] 05-11-2022 14:18:53 SMTP Server: 2.57.122.207 disconnected. 0 message[s] received
[000738:000014-00007F1164B21700] 05-11-2022 14:18:53 SMTP Server: Authentication failed for user lhy ; connecting host 5.34.207.55
[000738:000010-00007F1164C23700] 05-11-2022 14:18:53 SMTP Server: 5.34.207.55 connected
[000738:000014-00007F1164C23700] 05-11-2022 14:18:55 SMTP Server: 5.34.207.55 disconnected. 0 message[s] received
[000738:000012-00007F1164B21700] 05-11-2022 14:18:55 SMTP Server: command-combo.halffail.com (141.98.11.75) connected
[000738:000017-00007F1164C23700] 05-11-2022 14:18:55 SMTP Server: Authentication failed for user slice ; connecting host 5.34.207.55
The reason why it’s interesting for hackers to find a valid login on an SMTP server, is that this will probably allow them to send spam through your mail server. Most mail servers allow sending mail through their servers for other domains for authenticated users only. The chances of them guessing any of the users in my Domino directory right and then also guessing the password correctly are basically zero, but the pollution of my log file is reason enough to stop them.
Fail2ban to the rescue
Fail2ban is a very elegant program for Linux to do just that. You can configure it to scan log files for certain patterns (it uses RegEx to recognise them) and add hosts that match those patterns too often within a defined period of time, to the block list of iptables. By default, fail2ban has a filter for Domino SMTP already (at least, it does in CentOS). However, this filter assumes that you have the notes.ini parameter DEBUG_THREADID set to 0. By default it’s set to 1, which gives the thread id as shown in the log line examples above. Daniel Nashed therefore created a new filter, along with a nice little script, which takes these thread IDs along. I found, though, that Daniels filter didn’t include the SMTP server entries that show in my logs. I’ve notified him, so I’m sure he’ll soon fix that, but in the mean time, you can simply edit the /etc/fail2ban/filter.d/domino.conf file and add the last line of the block below to the existing lines.
failregex = ^.? (http|smtp): .? [] authentication failure using internet password.?
^.? TLS/SSL connection .? failed with rejected.?
^.? SMTP Server(?::|\s+[[^]]+]) Authentication failed for user .? \; connecting hostOnce setup, fail2ban is configured correctly for a standard native Domino server.
Special configuration for Domino containers
Daniel’s configuration consists of 2 parts, The domino.conf filter and a jail.local file which blocks both attempts on ssh and on Domino services (HTTP/HTTPS/LDAP./SMTP/IMAP/POP3). Looking at the Domino part of the config, there are a couple of parameters to take into consideration.
[domino]
enabled = true
filter = domino
logpath = /local/notesdata/notes.log
backend = polling
port = 0:65535
findtime = 300
bantime = 300
maxretry = 8
ignoreip = 127.0.0.1/8filter
logpath
findtime
bantime
maxretry
ignoreip
This needs to correspond to the name of the filter in the filters.d folder
This refers to the path to the notes.log file on your host system. This might therefore differ from this path
Within what time period should fail2ban look for attempts. 300 seconds is the default. I find that a bit short and have increased it to 5m (5 minutes)
For how long is the perpetrator banned? I find 300 seconds far too short. I’ve set it to a day (24h)
How often would someone be allowed to try. If you don’t allow authenticated SMTP, you can set this to 1. Remember that this can also be configured for HTTP traffic though, so be careful here
I’ve included my own lan network here
If you run your Domino container straight on the host network (the default if you use Daniels script), you should be done here. However, if you are forwarding ports to your Domino container (like I do), read on!
You can see if everything’s working correctly by running ‘domban’ (if you installed Daniels script). This will show the fail2ban jail. If you haven’t installed Daniels’ script, ‘fail2ban-client status domino’ will do the same.
Status for the jail: domino |- Filter | |- Currently failed: 1 | |- Total failed: 140 | `- File list: /<path-to>/notesdata/notes.log `- Actions |- Currently banned: 4 |- Total banned: 4 `- Banned IP list: 5.34.207.52 5.34.207.55 2.57.122.207 87.246.7.77
If you have banned IP addresses listed or the failed counter > 0 then you know that fail2ban is working correctly for your Domino server. You can check with ‘iptables -S | grep f2b’, if the IP addresses are also listed in the ip tables. Docker/Podman containers, however, have their own iptables rules and you if you don’t use the host network, you will find that the IPs are not blocked inside your container. To get this working, you need to define a special action. What action you need depends on whether you’re using Docker or Podman. I’ve added a file for both to my /etc/fail2ban/action.d folder. The docker-action.conf has the following content:
[Definition] actioncheck = iptables -n -L DOCKER-USER | grep -q 'DOCKER-USER[ \t]' actionban = iptables -I DOCKER-USER -s <ip> -j DROP actionunban = iptables -D DOCKER-USER -s <ip> -j DROP
My podman-action.conf file looks like this:
[Definition] actioncheck = iptables -n -L NETAVARK_FORWARD | grep -q 'NETAVARK_FORWARD[ \t]' actionban = iptables -I NETAVARK_FORWARD -s <ip> -j DROP actionunban = iptables -D NETAVARK_FORWARD -s <ip> -j DROP
These are a generic files, so I didn’t mention the word domino. To make the domino configuration for fail2ban use one of these actions, you need to add an extra line in to domino part in your jail.local file:
banaction= docker-action
or
banaction= podman-action
depending on what you’re using. That’s it. Now, your banned IP addresses are blocked in iptables in a way that includes your docker/podman containers.
Thanks for this article, Martijn!
Question- is it possible (or even desirable) to run fail2ban within the domino container?
It’s doing so much already. Also, when I try to run the domban install command, I encounter problems because of the lack of the EPEL repository, then because there is no firewalld nor systemd. It seems like it may be possible with enough effort but it is trying to do it the hard way.
Do you recommend running a separate fail2ban container that has access to the domino container’s network and notes.log instead? Or am is there a streamlined process to get it streamlined into the domino docker image at build time?
Thanks Ben. Regarding your question, I think you regard containers too much as VMs. They’re not! They share parts with the host OS and the network stack is one of them. Limiting the incoming connections is therefore something you should do on the host. Don’t try to combine this with the Domino container as that’s the wrong spot to implement it.
Indeed, that was the concern that brought up the question in my mind so I appreciate your response. I know containers do things differently than VMs, but I’m still learning the implementation details of that. Already I can see that the domino container doesn’t really yet do things the “container way” as there are so many different services and processes running in that single instance. If designed fresh with container implementation in mind, it would probably be implemented much like Sametime on docker is with a compose file and many containers doing their one (and only one) thing well.
I had not considered running fail2ban on the host, so thanks- that makes a lot of sense!
Ah, yes. I could have made that clearer in my article. Thanks for the feedback!