After successful DNSSEC signing of go6.si, go6lab.si, zorz.si and other domains in Go6lab we decided that it was time to start experimenting with DANE, firstly for email server TLS certificate verification. First reality-check and disappointment was after we checked the Postfix versions that we are running on mail servers, we figured out that we have pre-DANE-supported postfix versions and currently the latest one that we could find in any repository is 2.10 (for CentOS7). This meant that we can’t verify TLS certificates of remote SMTP servers, because DANE verification for smtp client part of postfix is only supported in versions >=2.11.
After some contemplation we decided to leave the production servers alone and just setup the certificates for SSL/TLS encryption on them to be ready to accept emails over encrypted channel and setup a completely new MX SMTP frontend in separate VPS with Postfix 3.0.1 compiled from the source. Setting up new MX server was fairly easy with CentOS and Amavisd-new (with ClamAV and SpamAssasin) and after some initial tests and re-routing of SMTP traffic for zorz.si domain (where we can test things as it’s not high volume email domain) we decided that our new postfix server is ready for some DANE and TLS surgery.
There is a very good step-by-step guide how to setup and create certificate for your posfix server, how to create a hash of our certificate and how to setup our TLSA record in our DNSSEC signed domain.
Ultimately the TLS part of postfix configuration on mx.go6lab.si looks like this:
smtpd_use_tls = yes smtpd_tls_security_level = may smtpd_tls_key_file = /etc/postfix/ssl/server.pem smtpd_tls_cert_file = /etc/postfix/ssl/server.pem smtpd_tls_auth_only = no smtpd_tls_loglevel = 1 smtpd_tls_received_header = yes smtpd_tls_session_cache_timeout = 3600s smtp_tls_security_level = dane smtp_use_tls = yes smtp_tls_note_starttls_offer = yes smtp_tls_loglevel = 1 tls_random_exchange_name = /var/run/prng_exch tls_random_source = dev:/dev/urandom smtp_dns_support_level = dnssec
In our case the MX record for zorz.si became mx.go6lab.si (new MX frontend) and after following up that instructions the TLSA record looks like:
_25._tcp.mx.go6lab.si IN TLSA 3 0 1 b4b7a46f9f0dfea0151c2e07a5ad7908f4c8b00 50e7cc25908da05e2a84748ed
Our “backend” mail server where mail is stored is still mail.go6.si, so we need also a TLSA record for this server.
_25._tcp.mail.go6.si IN TLSA 3 0 1 d928e2e06a8c126b3238fb2802c71310eb89eb 90bc21e4d852edbd902f3689ba
After inserting this in our go6lab.si and go6.si zone and after signing them, it was time for further testing! We investigated a bit and figured out that our friends from NLnet Labs also implemented DANE, so it was time to send and email to Benno Overeinder and ask him to check the mail server log files to see if his server verified the TLS when he responded back to my email. And it did! In his second email I got back the copy/paste from his log saying:
dicht postfix/smtp[29626]: Verified TLS connection established to mail.go6.si[2001:67c:27e4::61]:25: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)
dicht postfix/smtp[29540]: Verified TLS connection established to mx.go6lab.si[2001:67c:27e4::23]:25: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
Very cool, this means that NLnetLabs server actually verified the TLS certificate to mail.go6.si (where email response for [email protected]) went and also it verified the TLS connection to mx.go6lab.si where mail for [email protected] went.
mail.go6.si server postfix version is much lower than 2.11, so this means, that you can setup a certificate and TLSA also for older postfix servers and if sending server is DANE capable it will verify the TLS certificate – but you will not be able to verify your outgoing SMTP sessions, as smtp sender part is not ready yet.
Of course, we checked if our new MX server verified NLnetLabs TLS sertificate – and it did 🙂
mx postfix/smtp[643]: Verified TLS connection established to open.nlnetlabs.nl[2a04:b900::1:0:0:10]:25: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
Time to test things with somebody else! We knew that our second largest ISP in Slovenia (T-2) signed their domains with DNSSEC and as we know pretty well the person (Rok Potočnik) who is responsible for that (and also mail system), we contacted him and put him in front of a small challenge… 🙂
He immediately responded back and as he was already familiar with DANE concept he actually just added the TLSA record for their mail servers and viola, my mail server started to verify his certificate with TLSA information:
mx postfix/smtp[31332]: Verified TLS connection established to smtp-good-in-2.t-2.si[2a01:260:1:4::24]:25: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)
So, we have an ISP in the country that actually enabled DANE for email and everybody that sends an email to their server can actually verify that he’s sending to the right server with verified certificate for TLS session. Rok needed whooping 15 minutes to make it work, so I see no excuses for other operators to not do it also. Next challenge for Rok is to upgrade/update the sending part of his email system to be DANE capable, so he can also verify where he’s sending email and what certificates are being presented to his server for a SMTP session.
Further we tested some different DANE enforcement levels, as at Postfix TLS site it does not say very clearly what each level of DANE enforcement does for smtp_tls_security_level setting. After careful reading we understood that if you set this setting to dane-only the behaviour would be: “is there TLSA record? If yes, it must verify, otherwise drop. If not – send in clear text.”
So we tested the dane-only setting and the result was not what we thought it would be, but somehow predicted:
warning: TLS policy lookup for gmail.com/alt2.gmail-smtp-in.l.google.com: non DNSSEC destination to=<(******@gmail.com>, relay=none, delay=0.55, delays=0.02/0.03/0.5/0, dsn=4.7.5, status=deferred (non DNSSEC destination)
We put back “dane” setting and now the behaviour is more real-life acceptable and it sends emails also to non-verified TLS servers if there is no TLSA record. The only thing left to test was “malformed” TLSA record – what if hash in TLSA does not verify with the offered certificate?
Let’s try. We created an AAAA record for mail-bad.go6lab.si pointing to our old email server and TLSA record for that server with 1 character changed in hash data. The result was predictable, our server could not succesfully verify the certificate from existing TLSA record and dropped the connection. In case of “man-in-the-middle” attack, when somebody would not have the right certificate – we would not trust him.
May 8 15:02:38 mx postfix/smtp[1765]: Untrusted TLS connection established to mail-bad.go6lab.si[2001:67c:27e4::beee]:25: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) May 8 15:02:38 mx postfix/smtp[1765]: 3A4BE8EE5C: Server certificate not trusted
One of the frustrations with TLSA records was the fact, that we are using PowerDNS with PowerAdmin as front-end for DNS server where unsigned domain zones are and are then signed with OpenDNSsec platform and served from our front-end DNS servers. PowerDNS as such supports TLSA records, but PowerAdmin does not. So the trick was to enter this record as TXT record, connect directly to mysql database and change that field in the database directly, before triggering the DNSSEC signing.
Please, PowerAdmin, can you fix this? We hope that DANE gets up from the floor, it’s not magic or nuclear science – it’s a very good tool to verify who’s on the other end of SMTP session and it’s quite easy to implement and if done correctly – harmless.