Over the past day there have been a number of news reports talking about the brief outage that occurred yesterday, August 14, 2013, when sites ending in .GOV were unreachable if you were performing DNSSEC validation on those domain names. Many of those news reports are pointing at Johannes Ullrich’s post on the SANS ISC Diary site where he noted this issue.
The issue was fixed relatively quickly and the speculation on the dns-operations mailing list was later verified by a message sent from Verisign’s Duane Wessels to a number of mailing lists:
On the morning of August 14, a relatively small number of networks may have experienced an operational disruption related to the signing of the .gov zone. In preparation for a previously announced algorithm rollover, a software defect resulted in publishing the .gov zone signed only with DNSSEC algorithm 8 keys rather than with both algorithm 7 and 8. As a result .gov name resolution may have failed for validating recursive name servers. Upon discovery of the issue, Verisign took prompt action to restore the valid zone.
We can argue, perhaps, with the statement that “a relatively small number of networks” experienced this issue as those “networks” include all of Comcast’s 18 million users plus the millions of users out there who are using Google’s Public DNS services, as well as all the many other ISPs around the world who have enabled DNSSEC validation for their customers.
However, it may be true that a relatively small number of users of those networks happened to be visiting .GOV sites during the time period in question.
Regardless, the important part is to note here that this was an operational issue with the administration of DNSSEC for the .GOV domain rather than any particular issues related to the technology behind DNSSEC. As Duane Wessels had noted in an earlier message back on July 30, 2013, the .GOV zone is preparing to make a change to make its deployment of DNSSEC more secure:
An algorithm roll for the .gov zone will occur at the end of August, 2013. This notice is provided as a courtesy to the DNSSEC community. No action should be required on your part.
The .gov zone is currently signed with algorithm 7 (RSASHA1-NSEC3-SHA1) and will be changed to use algorithm 8 (RSA/SHA-256), bringing it in line with other top-level domains such as as .com, .net, and the root zone. The zone will be signed with both algorithms for a period of approximately 10 days.
Further scheduling details will be provided one week before the algorithm roll begins.
It seems that in Verisign’s preparations for that change an error was made and an incorrectly configured zone file was published instead. While obviously it would be preferable if the mistake had not been made, kudos to the team at Verisign for correcting the issue quickly and for also reporting back to the larger DNS / DNSSEC operations community on what the problem was that occurred.
Duane Wessels noted in his message today that Verisign is still planning to proceed with the algorithm rollover at the end of August and so we can expect to see more communication from them as they proceed with the change.