Some time ago Matthijs Mekking, one of the authors, maintainers and coders for the OpenDNSSEC project asked me why go6.si domain was not signed. My answer was that I need a short, precise and deployment/operations-oriented document with clearly described steps on how to setup the signing platform, how to add a zone and sign it.
Matthijs accepted the challenge and soon we got the first draft of the document to Go6lab. There were some errors in the procedure, so together we fixed them. Matthijs added some more information that he saw as needed for better understanding and re-tested the procedure again – and this time it worked. After some cycling through the Linux distributions we found that using Fedora and installing OpenDNSSEC from their repository usually brings you the latest version of the code – and the issue that we found in first round of testing was fixed in new version (the issue was that if you had OpenDNSSEC signer as “bump in a wire” between silent primary and secondaries that served the signed zone – signer did not understand “NOTIFY” message from primary and did not transfer new zone for signing).
We decided to install OpenDNSSEC signer as “bump in a wire”, fetching the unsigned zones from the “silent” primary server, where we edit and change the zones (with notify to signer), sign the zones on signer server and push them over AXFR to two secondary servers that acts like primary and secondary DNS for that zone. Seems easy – and it is, in a way 🙂
The “silent” primary server runs on PowerDNS with a mysql database backend. The two secondary servers that are serving the signed zones are both running bind9.
At first glance all worked fine, following the deployment guide we added go6.si, go6lab.si and zorz.si zones for signing (zorz.si was a test “mule” to see if the process works). The signer got the zones, signed them and pushed them to both name servers nsec1.go6lab.si and nsec2.go6lab.si. Our .si TLD dnsmaster Benjamin Zwittnig inserted for us DS records in .si zone and the whole thing started to work. We are pressing now our registrar to implement the tool to insert DS records to .si zone without bothering Benjamin. Hopefully this happens soon.
So, with DS records in parent zone we were able to test the whole thing and all was fine… until I stopped changing the zones and the primary server did not send any NOTIFY to signer server that there is a change in the zone for some time. And that “some time” was exactly the time that was set for a zone to expire. What happened was that signer server realized that the zone expired and instead of asking his primary name server for retransmit (AXFR) – the server expired the zone and stopped serving it.
So, zorz.si was off-the-Internet for 20 minutes! When my monitoring system alerted me that there is something wrong with DNS responses for that domain, I immediately figured out that signer expired the domain without refreshing it at the primary server.
As I needed to solve this issue quickly I created cronjob that forces a NOTIFY for all signed domains on hidden primary server every day at midnight – and so domains never expires on signer server. I also reported this bug to Matthijs and I think that the fix will be done in next release of the software.
It’s good to have a real environment where we can test and stress-out these new tools that are making the Internet a safer place – and also make these tools better with finding bugs before they hit somebody else in a big production environment.
You can now download the latest draft of Matthijs’ opendnssec-start-guide document that I used to set up the whole thing. Please note – it’s a draft, so all comments, suggestions and ideas are more than welcome.
How did we test if our DNSSEC signing was correct and valid? We used three tools:
- Verisign’s DNSSEC Debugger
- DNSViz (screenshot at right)
- ViewDNSSEC
… but there are many others out there, described at our DNSEC Tools page.
P.S. If you would like to get started with DNSSEC, please visit our Start Here page to find resources tailored for your type of organization or role.