Domino 12 and Borg backup
I must admit that I’m quite excited about Domino 12. I was thinking this morning why actually. The new features in Domino 12 aren’t necessarily groundbreaking. They’re more about fixing things which should have been in the platform already, but were neglected by IBM in the years in which it would have been logical to implement them. Features like proper handling and support for TLS certificates, Time-based one-time password authentication (aka 2 factor authentication, though the Notes client had 2-factor authentication before that even was a thing) and the subject of this post: proper backup solution support, aren’t the most sexy features. They are however essential features for an enterprise product. And I guess that’s why I’m so excited about Domino 12. Domino has always been a fantastic platform to create the type of business applications that are the oil of any organisation, but the lack of some basic features was seriously hindering Domino. Many competitors took their chance to call Domino an outdated platform while pointing at these missing features, and we had a hard time countering them. HCL taking the time to fixing these issues (and fixing them well), makes Domino fully competitive again.
So back to the subject of this post. Backup for Domino. To put expectations right, there are many good backup solutions today and HCL doesn’t plan to compete with them. A problem we faced more and more in recent years though, is that there were very few backup solutions that support Domino. The thing is that you can’t just copy the .nsf-files on a running Domino server to a backup and expect to have a consistent database if you ever need to restore this file. Also very few solutions supported Domino’s transactional logs. If HCL would provide a solution which would bridge this gap and provide an interface for backup solutions to get consistent databases from a running Domino server and support forwarding the state of this database to a later timestamp using the transactional logging, you would be able to use any backup solution to backup your Domino server. So this is what they did.
This new backup interface also opens the door for open source backup solutions like Borg Backup. Borg backup is a highly efficient Linux/BSD based backup solution that supports deduplication and compression.
The setup
The subject of my test is my own Domino environment. A 19 year old Domino environment which includes my 5,5 GB mailfile. My main server is currently running version 12 beta 3 inside a Podman container, which was built using the scripts from the Domino Docker project. Daniel added a -borg option to the build parameters to include all software needed for Borg backup. He also wrote a very nice blog post on what else to set to enable the Domino server for backup with Borg.
Backup results
So what is the result of backing up my mailfile with Borg backup and the backup tool? To see the info of your backups, you should go to your borg directory (by default /local/borg and type:
borg info .
This is my output:
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
All archives: 24.00 GB 13.80 GB 8.49 GB
Unique chunks Total chunks
Chunk index: 5396 8908
This is the result of 2 backups of which the last one is about 2 weeks ago. Let’s make a new one. I could do this from the Domino console, through a program document or directly from the command line, Obviously a program document makes the most sense to make regular backups, but for this test I just make the backup from the command line:
[notes@poseidon local]$ cd /local/notesdata [notes@poseidon notesdata]$ /opt/hcl/domino/bin/backup mail/ [102338:000002-00007F8C759690C0] 18-04-2021 12:29:15 Backup: Domino Database Backup [102338:000002-00007F8C759690C0] 18-04-2021 12:29:15 Backup: Started [102338:000002-00007F8C759690C0] 18-04-2021 12:29:15 Backup: BackupNode: [Poseidon], BackupName: [default], Translog Mode: [CIRCULAR], Backup Mode: [SELECT] [102338:000002-00007F8C759690C0] 18-04-2021 12:29:15 Backup: LastBackupTime: [102338:000002-00007F8C759690C0] 18-04-2021 12:29:15 Backup: Starting backup for 15 database(s) [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: --- Backup Summary --- [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: Previous Backup : [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: Start Time : 18-04-2021 12:29:15 [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: End Time : 18-04-2021 12:30:15 [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: Runtime : 00:00:59.481 [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: All : 15 [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: Processed : 15 [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: Excluded : 0 [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: Pending Compact : 0 [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: Compact Retries : 0 [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: Backup Errors : 0 [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: Not Modified : 0 [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: Delta Files : 0 [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: Delta applied : 0 [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: Total DB Size : 7.0 GB [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: Total DeltaSize : 0.0 Bytes [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: Data Rate : 120.2 MB/sec [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: --- Backup Summary --- [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: [102338:000002-00007F8C759690C0] 18-04-2021 12:30:15 Backup: Finished
So what’s the result of this new backup? A ‘borg info /local/borg’ will tell:
Original size Compressed size Deduplicated sizeAll archives: 31.50 GB 17.98 GB 11.28 GBUnique chunks Total chunksChunk index: 7226 11637
My backup size grew by 2.8 GB to a total of 11.28 GB. Not bad for 3 backups of an 8.7 GB mail directory.
Restore
Obviously the whole idea behind a backup is that you can restore it at some moment in time. So how does this work with the new backup tool? Imagine this scenario. An employee comes to you with the request to help him restore an extremely important mail which came in during the day, but which he deleted and on exiting his mailfile he also cleared his trash.
The backups were made before this mail came in, so with most backup tools that mail would be unrecoverable. The new Domino backup tool however, is able to deal with the transactional logs, so if you can replay the log till a time before the mail was removed, you could help the employee.
The first step is to open the Domino backup database. In this database, you can see the inventory of all databases on the server. Select the database you want to restore and press ‘Restore’. Next you have to fill in a form. I know I’ve deleted the mail after 19:55, so if I select to recover the database till that time, I should find the mail.
Submitting this document, gives me a pending restore job. The way to process it, is to start the restore task, which I would normally do from the Domino console, but to not get other Domino console logging between the restore output, now do from the command line:
[notes@poseidon notesdata]$ /opt/hcl/domino/bin/restore [006336:000002-00007FA4C96DD0C0] 18-04-2021 20:16:09 Restore: Translog Mode: CIRCULAR [006336:000002-00007FA4C96DD0C0] 18-04-2021 20:16:09 Restore: Domino Database Restore [006336:000002-00007FA4C96DD0C0] 18-04-2021 20:16:09 Restore: Started [006336:000002-00007FA4C96DD0C0] 18-04-2021 20:16:09 Restore: Restoring to [restore/mdejong.nsf] [/local/notesdata/restore/mdejong.nsf] [006336:000002-00007FA4C96DD0C0] 18-04-2021 20:16:36 Restore: Recovering Database [/local/notesdata/restore/mdejong.nsf.DAD] PointInTime: [18-04-2021 19:55:00] [006336:000002-00007FA4C96DD0C0] 18-04-2021 20:16:36 Restore: Applying 134.0 Bytes to [/local/notesdata/restore/mdejong.nsf.DAD] [006336:000002-00007FA4C96DD0C0] Media Recovery Replay (10 MB): 10% 100% [006336:000002-00007FA4C96DD0C0] 18-04-2021 20:16:36 Recovery Manager: Media Recovery complete for /local/notesdata/restore/mdejong.nsf.DAD, last update applied 18-04-2021 19:52:20. [006336:000002-00007FA4C96DD0C0] 18-04-2021 20:16:38 Restore: Last Transaction restored : [18-04-2021 19:52:20] [006336:000002-00007FA4C96DD0C0] 18-04-2021 20:16:38 Restore: Database restored: [mail/mdejong.nsf] -> [restore/mdejong.nsf] [006336:000002-00007FA4C96DD0C0] 18-04-2021 20:16:39 Restore: Finished
You can see that the transactional log was played and sure enough, the restored mailfile contained the missing mail!
Conclusion
The new Domino backup solution provides an easy way to use a plethora of backup tools together with Domino. Just like the new Certificate Manager, this tool was really written from a user/administrator perspective: What options would I want to have in a tool if I were an administrator responsible for backups and restores? Currently, the form to select what you want to restore contains so many options that you might get overwhelmed if restoring databases from the backup is not a daily activity for you. We’re still in beta though, so hopefully the form will get a bit of a usability overhaul still before GA with on-hover popups of what option does what and categorizing the options more clearly together, but the fact that I can’t think of missing options is really great. I’m also very happy that there’s finally an out-of-the-box backup solution with transactional log support, as afaik Tivoli Storage Manager was the only solution which supported this, which was severely limiting. The new backup solution fixed, what was starting to become a problem for Domino and they fixed it well!
