In a previous series of articles (part 1, part 2) we described how to install and use DANE for verifying your email and web server certificates through the DNS. In this article, I discuss how to monitor whether your TLSA records still match the certificates used for your services.
The aim is help people doing something similar, as this monitoring system saved the DANE deployment in the Go6lab from being broken many times, especially at the very beginning of deployment when the automated systems we built didn’t work completely correctly all the time 🙂
If we’re using self-signed certificates for our services and we manually change the underlying key, then we’ll probably also change the values of the TLSA records. If we forget to change them or are using, for example, Let’s Encrypt that automatically renews the certificate every 90 days, then we need to have some automatic DANE monitoring system that informs us of misalignment between the certificate being used and the TLSA record.
For this reason there are more and more online tools to check individual services by entering the URL and port, and with a press of a button tell if things are still working fine. A few examples of such tools include:
These tools allow you to quickly check your services, but you need to manually enter their details and check them one-by-one. In the Go6lab we use TLSA records to secure many services, and since we’re using Let’e Encrypt certificates that are automatically renewed, we had to build a tool that checks for DANE validity and even dynamically change the TLSA records in our DNS to match the new certificate.
Using the default mode of certbot, one of the Let’s Encrypt applications to fetch and renew certificates, the underlying key of certificate is also changed whenever we renew. In this case, the proper process is to renew the key, use a tool to generate a TLSA record from our certificate using the Command Line Interface (e.g. tlsagen), use the dynamic DNS updater (nsupdate) to add the new TLSA record, wait for the DNSSEC process to finish, sign and distribute the zone, then restart the service to commence using the new certificate and then remove the old TLSA record from our zone.
Let’s focus here on the DANE monitoring tools that we are using for remote DANE monitoring in the Go6lab:
- For checking HTTPS we are using ldns-dane, part of LDNS toolkit from NLNETLabs.
- For checking SMTP we are using a tool named “danecheck“, built by Viktor Dukhovni.
LDNS toolkit from NLNETLabs is pretty straightforward, and you can install it with the apt or yum package installers if you happen to use a Linux system (just search for “ldns”). However, if you have an SNI-based https domain service (multiple virtual hosts using the same IP address), then please note that Server Name Indication (SNI) was introduced in ldns-dane in version 1.6.17. At the time of writing, the majority of apt and yum repositories offer version 1.6.16 and with that version checking of the SNI domains will fail despite the fact that your TLSA might be correct.
The tool for testing DANE is called ldns-dane and usage is pretty simple:
[root@mx dane-monitor]# /usr/local/bin/ldns-dane verify www.go6lab.si 443 91.239.96.52 dane-validated successfully 2001:67c:27e4::52 dane-validated successfully
The tool will send an DNS query for the TLSA record for, in our case, ‘_443._tcp.www.go6lab.si.’, and will also query for A and AAAA records associated with ‘www.go6lab.si’. It will then communicate with the IP addresses, ask for the certificate and compare the two of them.
If your output does not look similar to the above, something is wrong with your TLSA records or DNSSEC signatures.
If you decide to download and build LDNS toolkit from the NLNETLabs website, please note that version 1.17 requires OpenSSL 1.1. If you use version 1.0, then stick with LDNS version 1.16.17.
Viktor Dukhovni’s danecheck tool is built a little bit differently as it requires a working Haskell GHC plus Stack toolchain. These can be downloaded from the Haskell platform website and are also available as packages for various operating systems.
Using the tool is quite simple, you specify just a domain name that you wish to test (in most cases):
root@openNMS:/# danecheck go6lab.si go6lab.si. IN DS 15361 8 1 104e2a57cd968ec0bb283872c7f546d9b174 1839 ; AD=1 NoError go6lab.si. IN DS 15361 8 2 f2cf0d4b46d3be6ccd25516cafc33fb5105f 28c3a9a82d782afe2ff1017bb208 ; AD=1 NoError go6lab.si. IN DNSKEY 256 3 8 AwEAAc48Qs/Ew8nipd9vYGlk+NzXqE/llT FU0IePzbEnH8sGbCGziDyo86aNIZQ3RuVjc30TUCR3cLZ6wrrOynPBeNGc20WpV IvidQeWOWGieXkVYdQi6U+Cgk5LInq5aXg6IHriRC80mv7K1cofLVLRTTPlKXPf tUbTStVzBX85k28cw7KfA92CbSbnAYl023eRaIE6lzKyzpuuel1OLUWxwmAGHH5 bTc6aa6A+jfcelkzUahCFZrQh+cU303LU8jVhNZGr6AgqiE6m2AE3iFdjzm6AMk lqWNmYJeuQ0WwbSwu04Kuco9H5E0tdx26jbzaRffsxRPB/Kr0ydVqTU84fJnc= ; AD=1 NoError go6lab.si. IN DNSKEY 257 3 8 AwEAAcgqIy9CjcyqWY+sJpNk9hC7CAUozO xSpJddgwjIdvn6uZaEL7mHeAEGzHtubaBQBjVfiT7FwWL3tF8T/pirKuQ3tgVYS 4CS9T9uNFyYaue0P8lGX1s8vxtLNcnVfmoDRHJp/ne/6w4CqD4olOoExECPkXk5 LYAFG1fP+dPJKblWHE5jIlG6sxRp2dDxr6o7iEoL8VYEqWKe1sXZciF5P2N9/jo NKe1QY4WNYoV/Hdr5ZqEeleArDa9UjWZ8sKQ6pXbJNdr1vDj23ANo4zmw/TS3Oz O2VBU2Y16bHKOQsck4WyPovc0J7zdUX7QqIQod0DzB2qbdf1sXo4+T24yJzvU= ; AD=1 NoError go6lab.si. IN DNSKEY 257 3 8 AwEAAcmW5udATR8Gmpeqyv4WYVGvr3gE5s nQZUSKgAU0mua39pbihh9EA8mxHNybbczYPlzS8JTTUZxcTNhuQzCkD1ELTA51K WtZpnMqQWv3Rbvse8Eh0T7mvTGU6z6WVW0bh2uauKKR8obIyHTo3ol6Em7Jp4tJ vbvdkTAi+3XFxalZlK0b94P2xzNmlfnttgJXPfNrKMukZqUljoZ5D/2ewEhY2K9 K++QkLLYO6kuwD0dPVKrJ2f3HUdbv+JGM0SOnXSiskdziOMJU2QrOd6g5Wku4VP juQgFotDTaxX3yR/y7hPCOq02EtQD2kMqgsOHK4nyqx79l1sME1CZ8aDSFBgc= ; AD=1 NoError go6lab.si. IN MX 10 mail.go6lab.si. ; AD=1 NoError go6lab.si. IN MX 20 mail.go6.si. ; AD=1 NoError mail.go6lab.si. IN A 91.239.96.17 ; AD=1 NoError mail.go6lab.si. IN AAAA 2001:67c:27e4::beee ; AD=1 NoError _25._tcp.mail.go6lab.si. IN TLSA 3 1 1 39726a2fe2bb052cf00e6b95 a8385f7446c3ea76d015fd0644085d6e9f3e8853 ; AD=1 NoError mail.go6lab.si[91.239.96.17]: pass: TLSA match: depth = 0, name = mail.go6lab.si TLS = TLS12 with ECDHE-RSA-AES128GCM-SHA256 name = mail.go6lab.si depth = 0 Issuer CommonName = Let's Encrypt Authority X3 Issuer Organization = Let's Encrypt notBefore = 2017-10-29T12:33:26Z notAfter = 2018-01-27T12:33:26Z Subject CommonName = mail.go6lab.si pkey sha256 [matched] <- 3 1 1 39726a2fe2bb052cf00e6b95a8385f7 446c3ea76d015fd0644085d6e9f3e8853 depth = 1 Issuer CommonName = DST Root CA X3 Issuer Organization = Digital Signature Trust Co. notBefore = 2016-03-17T16:40:46Z notAfter = 2021-03-17T16:40:46Z Subject CommonName = Let's Encrypt Authority X3 Subject Organization = Let's Encrypt pkey sha256 [nomatch] <- 2 1 1 60b87575447dcba2a36b7d11ac09fb2 4a9db406fee12d2cc90180517616e8a18 mail.go6.si. IN A 91.239.96.61 ; AD=1 NoError mail.go6.si. IN AAAA 2001:67c:27e4::61 ; AD=1 NoError _25._tcp.mail.go6.si. IN TLSA 3 1 1 43fc43f31135d9542658b87356d 129ccfdd654f996499ac97cd6d34b2b8f9668 ; AD=1 NoError mail.go6.si[91.239.96.61]: pass: TLSA match: depth = 0, name = mail.go6.si TLS = TLS12 with ECDHE-RSA-AES256GCM-SHA384 name = mail.go6.si depth = 0 Issuer CommonName = Let's Encrypt Authority X3 Issuer Organization = Let's Encrypt notBefore = 2017-11-18T22:45:21Z notAfter = 2018-02-16T22:45:21Z Subject CommonName = mail.go6.si pkey sha256 [matched] <- 3 1 1 43fc43f31135d9542658b87356d129c cfdd654f996499ac97cd6d34b2b8f9668 depth = 1 Issuer CommonName = DST Root CA X3 Issuer Organization = Digital Signature Trust Co. notBefore = 2016-03-17T16:40:46Z notAfter = 2021-03-17T16:40:46Z Subject CommonName = Let's Encrypt Authority X3 Subject Organization = Let's Encrypt pkey sha256 [nomatch] <- 2 1 1 60b87575447dcba2a36b7d11ac09fb2 4a9db406fee12d2cc90180517616e8a18
If everything works as it should, then you should see similar output to the above.
Note the lines with a string of “AD=1 NoError” in it, that can be can be used in a script to automatically determine whether anything is wrong. For Go6lab purposes, we test each domain and “grep” for string “AD=” to extract messages from the tool’s output, and “grep -v” string “AD=1 NoError” to eliminate everything that seems OK and valid. If there is anything after the “grep” processing, omething is wrong with the domain and an e-mail can be sent flagging that.
Here’s an example how we monitor our DANE SMTP environment using a simple bash script:
file="/root/dane-smtp-monitor/domains.txt"
domains=`cat $file`
for domain in $domains; do
echo "Checking SMTP DANE for $domain"
/usr/local/bin/danecheck "$domain" | grep "AD=" | grep -v NoError \
> /root/dane-smtp-monitor/"$domain"
if [ -s /root/dane-smtp-monitor/"$domain" ]
then
echo "DANE for $domain is broken"
mail -s "DANE TLSA config for $domain is broken!!!" [email protected] \
< /root/dane-smtp-monitor/"$domain"
else
echo "All good for $domain ..."
fi
done
This shows how you can monitor all your domains that are entered line-by-line in the specified domains.txt file. You can also use the same script to monitor your https services, but of course it will need to be modified as the output of the tool is different, and you would probably want to use “grep -v “dane-validated successfully”” as an indicator that all is well and functioning.
DANE and DNSSEC are very nice add-ons for security and provide better verification of certificates and services. But if they’re not working properly then you’re not doing yourselves a favour, hence the importance of monitoring to enable action to be taken if something fails.
Jan Žorž