Some would think that having a trusted certificate for your services such as web and email servers is enough, but some might wish to add an extra layer of security by publishing the hash of that in the DNS through a TLSA record and signed with DNSSEC.
We already enabled Let’s Encrypt certificates for various web servers in the Go6lab, but this time we wanted to use them to secure a mail server (postfix) and also add a DANE TLSA record to it – which is where the fun begins.
The validity of Let’s Encrypt certificates is only 90 days for various reasons, and our policy is simply to renew them using an automated script every month. So far so good until you want to add DANE TLSA records to the DNS as the question arises as to which certificate usage and selector combination to use?
Let’s Encrypt recommends using ‘2 1 1’ and ‘3 1 1’ records – the first of which publishes the hash of the issuing CA certificate (DST Root CA X3), whilst the second one ‘pins’ the public key fingerprint of the server certificate and does not require a valid chain up to the root certificate.
On the one hand, ‘3 1 1’ seems a good choice as it publishes the public key fingerprint of the certificate itself, but you’ll need to add new TLSA records with a new hash each time the public key is changed (which happens each time your certificate is renewed, triggered by the default Let’s Encrypt client configuration). This becomes a bit annoying if you have to do it every month, especially if you have many servers to manage.
It is possible to use the --csr option in the Let’s Encrypt client so the CSR is derived from the same underlying key upon renewal, but pinning the same key for long periods of time is seen as a potential security risk by some people.
We are NOT suggesting using the same key forever, but we are suggesting automated renewal with whatever key is currently in use, changing keys periodically with human invention. We’ll look at this mechanism in the coming days, but we still need to do some additional testing and scripting so we’ll leave it for one of the next blog posts.
So using the ‘2 1 1’ method leads to another issue – namely lack of an DST Root CA X3 certificate in the fullchain.pem file provided by the Let’s Encrypt client. Let’s see what happens if we use the fullchain.pem file without the requisite DST Root CA X3 CA certificate, and adding it to the end of different TLSA selectors with different hashes.
For extracting the hashes for TLSA we used a very handy script, provided by Viktor Dukhovni:
printf '_25._tcp.%s. IN TLSA 3 1 1 %sn'
$(uname -n)
$(openssl x509 -in cert.pem -noout -pubkey |
openssl pkey -pubin -outform DER |
openssl dgst -sha256 -binary |
hexdump -ve '/1 "%02x"')
You can save that in a script (named tlsagen) and use it as follows:
$ ./tlsagen cert.pem $(uname -n) 3 1 1
There is an even better tool to create TLSA records from the fullchain.pem file called chaingen (TAR download), again provided by Viktor Dukhovni. This produces all types of TLSA records for our chain, and we recommend using it in a production environment.
For experimentation purposes we used the first script, created a single 3 1 1 TLSA record and published it for the hostname mx.go6lab.si. Checking it with the tool at SYS4 gives us a valid result:
3 1 1 TLSA published
Please note that following text includes some possible failures in the process which are provided in order that people understand what not to do and can avoid doing mistakes.
So now let’s try with 2 1 1, as the Let’s Encrypt certificate should have a valid chain. We used the tlsagen script again to extract the hash and it’s the same as for 3 1 1, except that the SYS4 testing tool now returns that the TLSA record is not valid:
2 1 1 TLSA hash published
What to do then? Let’s try and add the DST Root CA X3 certificate to fullchain.pem file that the Let’s Encrypt client generates after renewal. To automate that process we put together a simple shell script that fetches that certificate from the DST web site and adds it to the fullchain.pem file:
lynx --source https://www.identrust.com/certificates/trustid/root-download-x3.html | grep -v "/textarea" | awk '/textarea/{x=NR+18;next}(NR<=x){print}' | sed -e '1i-----BEGIN CERTIFICATE-----' | sed -e '$a-----END CERTIFICATE-----' >> /etc/letsencrypt/live/mx.go6lab.si/fullchain.pem
This adds the DST Root CA X3 cert to the end of the fullchain.pem file just after the renewal process, and before we automatically restart postfix to use the new certificate.
To understand better why we need to add the issuing CA certificate to our chain file, please read the blog post about avoiding using ‘3 0 1’ and ‘3 0 2’ DANE TLSA records with LE certificates.
Unfortunately, we still have no luck with the current 2 1 1 TLSA record as the tlsagen script still outputs the same hash as before. We’ll therefore have to do it differently by writing the DST Root CA X3 certificate to a separate file and trying to obtain the 2 1 1 hash using the tlsagen script:
Both 2 1 1 hashes in TLSA
As you can see above, the bottom 2 1 1 TLSA record is now valid as it publishes the public key fingerprint of the DST Root CA X3 certificate. We should now be able to renew our Let’s Encrypt certificate and so long as we keep adding the DST Root CA X3 certificate to the fullchain.pem file, we should be fine until the root certificate changes. In case of using the “chaingen” tool we would have there all covering TLSA records.
The problem is that we don’t have the slightest idea when the issuing CA certificate will change, so it might be a good idea to write a script that compares the old and newly fetched DST Root CA X3 certificate every time we renew our Let’s Encrypt certificate, and set-up an email alert if it changes.
If we also add a 3 1 1 TLSA record then both records now become valid:
Both valid – 2 1 1 and 3 1 1
We know the 3 1 1 TLSA record will become invalid after we renew our certificate, but the 2 1 1 TLSA record provides a backup if we forget to change the 3 1 1 record. We can also add a simple line to the renewal script that generates the 3 1 1 TLSA hash in order to send an email reminder to add a new TLSA 3 1 1 record.
Useful reading: (both by Viktor Dukhovni)
We still think keeping the same underlying key and corresponding certificate request (CSR) throughout the renewals, and therefore keeping the same hash in 3 1 1 TLSA records is the best way to go, and we’re already testing the Let’s Encrypt client with the --csr option. After we do some thorough tests we’ll report our findings in our next blog post on the subject. Stay tuned.
We grateful to Viktor Dukhovni for his thorough review of this blog post, all the scripts and pieces of text provided to make this writing even better, and especially for his excellent work in this area! Cheers!
Jan Žorž