In the first part of our blog on testing the Let’s Encrypt certificates for mail servers and adding extra security with DANE TLSA records, we focused on how to deal with the default behaviour of the Let’s Encrypt client on Linux. This generates a new underlying key and certificate request every time we renew the certificate, which therefore means the TLSA hash value changes and renders old TLSA records invalid.
This introduces some hassle as we need to add a new TLSA each time we renew a certificate, and as they only have a 90 days validity and good practice is to renew them every month, this creates some additional operational work. This is particularly problematic when operating a large number of servers and domains.
One of the options available in the Let’s Encrypt client is to instruct it to use the same CSR each time the certificate is renewed. After reading some recommendations in a post to the LE Community forum, we tests this using the --csr option.
Our approach for this type of renewal was to run a script called ‘generate-csr.sh’ that’s found in the sub-directory ‘examples’ to generate a new private key for mx.go6lab.si:
./generate-csr.sh mx.go6lab.si
Generating a 2048 bit RSA private key
................+++
..+++
writing new private key to 'key.pem'
-----
You can now run: letsencrypt auth --csr csr.der
The script generated the private key “key.pem” and certificate signing request “csr.der”.
We copied that key to a secure place and ran the suggested command. After obtaining the new signed certificate based on the newly generated CSR, we extracted the TLSA record value with the ‘tlsagen’ script that was described in the first part of this blog post and added it to the previous TLSA record that will become invalid as soon as we will start using the new certificate (because for this experiment we changed the underlying key from ‘dynamic’ to ‘temporarily static’ and therefore the hash of the certificate changed). After some time (depending on your zone TTLs) we reconfigured Postfix to use new certificate and reloaded it.
How long we should wait before switching to a new certificate on Postfix? According to Viktor’s Dukhovni, this should typically be much longer than just for a zone TTL value as all secondary name servers first need to receive the updated records, and then it takes 1-2 TTLs for stale records to expire. Only then should Postfix be reconfigured to use the new key and certificate chain, that includes all the issuer certificates needed to match any DANE-TA(2) TLSA records.
The newest stable version of Postfix (3.1.0) comes with built-in support for a ‘3 1 1’ (only) equivalent ‘tlsagen’ script as documented at http://www.postfix.org/postfix-tls.1.html:
# postfix tls output-server-tlsa
The argument can point to your not-yet-deployed key and you create the new TLSA from that output.
After following the above mentioned steps, testing the DANE validity with https://dane.sys4.de/smtp/go6.si gives us the expected result:
The old 3 1 1 TLSA record is of course now invalid and can be safely removed. This allows us to choose the 3 1 1 TLSA method of validation, or use the 2 1 1 TLSA method with the same technique of adding the DST Root CA X3 certificate to our certificate chain. To understand why in this case we need to add the issuing CA certificate to our chain file, please read the blog post about avoiding using ‘3 0 1’ and ‘3 0 2’ DANE TLSA records with LE certificates.
Again utilising the same script used previously to add the DST Root CA X3 certificate to our certificate chain, and removing the old 3 1 1 TLSA record and reloading postfix, has a positive result at https://dane.sys4.de/smtp/go6.si:
Our renewal script now uses the command:
./letsencrypt-auto certonly -t --debug --renew -a standalone --csr ./mx.go6lab.si.der --keep
and also adds the DST Root CA X3 certificate to our certificate chain after successful renewal. We’ve run the renewal process several times now and https://dane.sys4.de/smtp/go6.si shows both valid TLSA records without changing the TLSA values.
As Viktor Dukhovni clearly points out, we’re not suggesting using the same key forever. Instead we are suggesting that automated renewals use whatever key is currently in place, with key rotation happening separately and being driven by human intervention.
With key rotation, you prepare a new key file, generate a CSR and obtain a certificate with it, before publishing updated TLSA records with both current and new digests. Once enough TTLs expire after all secondary DNS servers have the new data, deploy the new key pair and certificate. This step is difficult to automate reliably, so key rotation can happen asynchronously from certificate replacement.
So after trying several methods of renewing certificates and validating them with DANE, we decided to settle on this last method for the Go6lab production environment. It’s simple, it works, and you don’t have to change TLSA records every month or whenever you renew the certificate. The process can be automated until you decide to change the underlying key for your certificate, and even then it’s just a matter of changing the values in your renewal script.
We are grateful to Viktor Dukhovni for his thorough review of this blog post, all the scripts and text he provided to annotate the content, but particularly for his excellent work in this area! Cheers!
Jan Žorž