Modern email protocols: DANE, MTA-STS and TLS-RPT

Posted on by

In my recent OpenNTF webinar on modern E-mail Server operations, I covered several SMTP-related protocols like DKIM, SPF, and DMARC. However, with ongoing efforts to enhance the security of SMTP, new protocols have emerged, and these are the focus of this article.

My DANE status has become green in the mean time

Two weeks after my OpenNTF presentation, my former colleague Erwin Stamer, contacted me regarding the DANE status of my domain as it was yellow instead of green. He was looking at the status of my domain as they were implementing it at his employer (a large Dutch bank) and was looking for an example. I must admit that I initially had no idea what DANE was, but as it was in line with my presentation, I dived into it. DANE, MTA-STS and TLS-RPT all work together, but let’s look at them separately.

DNS-based Authentication of Named Entities (DANE)

DANE, as described in RFC 6698, works atop Transport Layer Security (TLS) and Domain Name System Security Extensions (DNSSEC) protocols. The primary aim of DANE is to secure SMTP by verifying TLS certificates directly through DNS records. This capability helps prevent man-in-the-middle (MITM) attacks, which can intercept and decrypt SMTP traffic, especially when self-signed certificates are in use.

DANE requires a DNS provider that supports DNSSEC, which ensures the integrity and authenticity of DNS responses. By using DNSSEC, it becomes significantly harder for attackers to intercept traffic. To enable DANE, you’ll create a TLSA resource record in your domain’s DNS. This record contains a hash of your TLS certificate, specifying how the hash should be interpreted.

My TLSA record

The structure of a TLSA record is:
Host/A-name: _port._protocol.<hostname or wildcard> e.g.: _25._tcp*
Value: <usage> <selector> <matching-type> <hash>

You can check here what the possible values are for the usage, selector, matching-type and hash. Personally, I used this to calculate the value:

openssl s_client -connect mail.martdj.nl:25 -starttls smtp -showcerts </dev/null 2>/dev/null | openssl x509 -pubkey -noout | openssl pkey -pubin -outform DER | openssl dgst -sha256

This line grabs the certificate from the server from my MX record (mail.martdj.nl in my case), takes the public key, converts the public key to DER format (required for hashing), and calculates the SHA-256 hash of the public key.

SHA2-256(stdin)= c6cb042a075ee7f902475d31dc97bd10e95b313ebb387113253eedd002d88ada

For this hash, the correct usage, selector and matching-type are 3 (DANE-EE), 1 (Public Key) and 1 (SHA-256). TLSA records can also be used to use a self-signed certificate for a website that is accepted by browsers (in which case.the record would start with _443_tcp).

Of couse, for DANE to be of any value, the sending server needs to support it, so that it checks the certificate against the values in DNS, and doesn’t send mail when the MX host advertises that it doesn’t support secure SMTP connections.

Mail Transfer Agent Strict Transport Security (MTA-STS)

This is where Mail Transfer Agent Strict Transport Security (MTA-STS) comes in view. MTA-STS, defined in RFC 8461, allows SMTP servers to declare their support for secure connections and instruct sending servers to reject messages if secure SMTP cannot be established.

MTA-STS requires:

  1. A DNS TXT record indicating the presence of an MTA-STS policy.
  2. A policy file hosted at: https://mta-sts.<your domain>/.well-known/mta-sts.txt

DNS TXT Record

Your DNS TXT record for your domain lists the MTA-STS version (currently, there’s just one version) and a date of your MTA-STS policy.

My MTA-STS DNS record
My MTA-STS DNS record
  • Type: TXT
  • Host: _mta-sts.yourdomain.com
  • Value: v=STSv1; id=YYYYMMDD
    • Replace YYYYMMDD with the date or version identifier for your policy (e.g., 20230923). This allows you to update the policy, and other servers will check if they need to reload it based on the ID.

MTA-STS Policy File

The policy file is a text file that consists of at least 4 lines. Here’s an example:

version: STSv1
mode: enforce
mx: mail.martdj.nl
mx: mx.bhosted.nl
max_age: 86400
  • version: Always “STSv1”
  • mode: Indicates whether to enforce TLS secured connections. Options:
    • enforce: This instructs external mail servers to strictly enforce TLS
    • testing: This allows you to test your setup by informing you of policy failures without enforcing them
    • none: no enforcement
  • mx: Your domain’s mail servers. This can include multiple entries if you have several mail servers (e.g., mx: mail1.yourdomain.com)
  • max_age: The time (in seconds) that the policy should be cached by other servers. 86400 seconds is equivalent to 1 day

It’s a good idea to start with the testing mode. Mail providers that support DANE and MTA-STS (like Google and Microsoft), will send you reports that will show any policy failures. At least, they will if you tell them where to send those reports. That’s where TLS-RPT comes along.

SMTP TLS Reporting (TLS-RPT)

SMTP TLS Reporting (TLS-RPT) as outlined in RFC 8460, is used to gather feedback on failed TLS connections. Similar to DMARC, it uses a DNS TXT record to specify where reports should be sent. Like for DMARC, online TLS-RPT record generators are available to help you with the contents of your TLS-RPT record, like this one from easy DMARC. The record syntax is as follows:

TLS-RPT record
  • Type: TXT
  • Host: _smtp._tls.yourdomain.tld
  • Value: v=TLSRPTv1; rua=mailto:postmaster@yourdomain.tld
    • The version, TLSRPTv1 is fixed. rua= defines where to send reports, which next to an email address can also be a web-api.

To support TLS-RPT, the mail server needs to:

  • Generate and Send Reports:
    • The server must collect metrics about successful and failed TLS connections (e.g., failed certificate validation, unable to establish TLS, etc.).
    • It must then generate aggregated reports on these issues based on the domain’s reporting preferences.
  • Query for TLS-RPT Record:
    • The server must query the _smtp._tls DNS TXT record to find the reporting email address or web endpoint where reports should be sent.
  • Support Report Formats:
    • The server must support creating and sending reports in JSON format, as specified by the standard.
    • Reports should include details such as the sending domain, recipient domain, certificate issues, time ranges, and more.
  • Send Periodic Reports:
    • The server should periodically send reports (daily or as configured) to the designated report recipients

As you can see above, DANE and MTA-STS records are no requirements for TLS-RPT, but a TLS-RPT record is a requirement to have a MTA-STS record using the testing mode make sense.

Testing your configuration

Once you have set your DNS records and made an mta-sts.txt file available at the required location, you can test if everything is correct.

  • For DANE, I found this a good validator, as it not just checks your records, but also checks if the hash in your records corresponds to your mail servers.
  • For MTA-STS, MX Toolbox provides a great overview.
  • You can check your TLS-RPT record through here, but you really have to wait for mails to come in, to really know if everything works as expected. These reports look like this:

TLS-RPT report

{
    "organization-name": "Google Inc.",
    "date-range": {
        "start-datetime": "2024-11-06T00:00:00Z",
        "end-datetime": "2024-11-06T23:59:59Z"
    },
    "contact-info": "smtp-tls-reporting@google.com",
    "report-id": "2024-11-06T00:00:00Z_martdj.nl",
    "policies": [
        {
            "policy": {
                "policy-type": "sts",
                "policy-string": [
                    "version: STSv1",
                    "mode: testing",
                    "mx: mail.martdj.nl",
                    "max_age: 86400"
                ],
                "policy-domain": "martdj.nl",
                "mx-host": [
                    "mail.martdj.nl"
                ]
            },
            "summary": {
                "total-successful-session-count": 15,
                "total-failure-session-count": 0
            }
        }
    ]
}

HCL Domino and DANE, MTA-STS and TLS-RPT

Currently, HCL Domino (up to version 14) does not support DANE, MTA-STS, or TLS-RPT, and there is no indication that support will be added soon. If you feel that this should change, please consider voting for this AHA idea to encourage HCL to implement these protocols.

As stated in this idea, to fully support DANE, MTA-STS, and TLS-RPT, a mail server must:

To fully support DANE, MTA-STS, and TLS-RPT, a mail server must:

  • DANE: Validate DNSSEC and enforce TLSA records.
  • MTA-STS: Check for and enforce MTA-STS policies.
  • TLS-RPT: Collect and report TLS connection issues in JSON format.

Dividing these between incoming and outgoing mail, here’s what each scenario requires:

  1. For Incoming Mail:
    • Validate DNSSEC for sending domains.
    • Verify the sender’s TLSA records.
    • Log any connection issues and be ready to generate TLS-RPT reports.
  2. For Outgoing Mail:
    • Check for MTA-STS policies or TLSA records before sending mail.
    • Enforce TLS as required by the recipient’s MTA-STS policy.
    • Validate the recipient’s certificate against any TLSA records.
    • Log TLS connection issues and generate reports for domains with TLS-RPT policies.

In practical terms, HCL Domino is more often used as an outgoing mail server, so I urge HCL to focus on implementing these protocols for outgoing email first, helping make SMTP mail more secure.

Resources