After the experiment with DPRIVE at IETF99, we thought we’d try to implement it in the Go6lab and see how this actually works in day-to-day reality.
The first step was to take a look at https://dnsprivacy.org/wiki/ as we had a feeling this might be the best source for information around this topic. There’s a ton of info about DNS over TLS, but what we were really looking for was simple instructions on how to setup a recursive DNS server to serve DNS responses over TLS (port 853), as well as how to setup a local client on our device that could talk to the server and accept local DNS queries over TLS, thereby protecting our DNS communications over the Internet.
We decided that running a TLS proxy was not the way to do it, so we used CentOS 7 VPS with Unbound installed. After some time and with extensive help from Willem Toorop from NLnet Labs (thanks Willem!!!) we managed to navigate the setup process for server and client.
Firstly, we installed the default Unbound from the CentOS7 default yum repositories, which turned out not to be a very good idea, as this version is 1.4.20 (March 2013) whilst intermediate certificates are only supported since version 1.5.5 (October 2015). So we had to add the GhettoForge repository to upgrade it to version 1.6.5, although for the sake of simplicity and easier setup for readers, we do not want to go into details of how to compile from the source as sometimes this gets messy 🙂
After we had Unbound installed, we started thinking about the certificates that will be used for TLS communication over port 853. Let’s Encrypt is our default choice, and installing one of the Let’s Encrypt clients was simple:
yum install certbot
After certbot was installed, we needed to obtain a new certificate for the DNS recursive server. In our case, it’s privacydns.go6lab.si.
certbot certonly -d privacydns.go6lab.si --standalone
As we don’t run an http server on this machine, we had to use the –standalone switch, so certbot starts an internal http server, exposes the authentication challenge file and shuts down after authentication is finished.
That command resulted in private and public certificates in the/etc/letsencrypt/live/privacydns.go6lab.si/ folder, although the default behaviour of certbot is to change the underlying private part of the certificate every time we renew it, and this is not what we’d like to do in our case. We had to solve the same problem when we experimented with DANE and our email system, but this time we just followed the process on DNSprivacy.org web site, where they explain how to force certbot to retain the private key when renewing.
Once we got this part right, we simply edited the unbound.conf file, found in /etc/unbound/ and set few parameters.
We don’t want this server to listen on a normal DNS port (53), so “interface 0.0.0.0/0” and “interface ::0” are commented out. Next thing to set is this:
interface: 0.0.0.0@853 interface: ::0@853
This instructs Unbound to start listening on port 853 that is currently the default experimental port for DNS over TLS communication. We then need to have a look at the access-control setting – set it up to your liking and preference – before configuring the TLS port and certificates:
ssl-service-key: "/etc/letsencrypt/live/privacydns.go6lab.si/privkey.pem" ssl-service-pem: "/root/dns-ssl-cert/privacydns.go6lab.si.pem" ssl-port: 853
ssl-service-key is our private key, usually found in the /etc/letsencrypt/live/[domain] directory.
We used this file to create a CSR during the previous step in order to keep the same private key across the renewals, so we need to specify it here.
ssl-service-pem is our certificate that results in the certbot key renewal with the same CSR.
ssl-port is set to 853, according to the current experimental default.
With this, we have a very basic setup of the Unbound server on port 853 done, so now it’s time to look at the client side.
Our first choice was “Stubby“, the product of GetDNSApi group (by NLnet Labs). There are clear instructions how to set it up, and since we use MacBooks in our environment, we decided to install it on one of our MacOS devices. We used the “compile from source” path described by the GetDNS Stubby page (at the bottom, search for “Download the getdns source”). Make sure you’re not using the default but outdated MacOS openssl library, but the newer one installed using the “brew” system.
Once you’ve successfully finished the “make install” part, you can start configuring the client. In our case, we installed it in /opt/stubby, so the config file was in /opt/stubby/etc/stubby/ directory.
By default it connects to several experimental DNS servers that offer DNS-over-TLS service over port 853, and we can just start the stubby daemon and see if it connects to these pre-set servers and test it by resolving a name of your choice (command issued from end-host):
dig @::1 www.go6lab.si
The result should be a valid DNS response.
The next step is to connect it to our own DNS-over-TLS recursive resolver, and this is where we got a bit stuck because in the config file you need to provide the certificate digest of the remote recursive server and we had no idea how to do that; even Googling did not return anything remotely useful. Luckily, Willem came to the rescue and sent us a command that connects to our server, fetches the certificate and prepares a digest for us:
echo | openssl s_client -connect '[2001:67c:27e4::35]:853' 2>/dev/null \ | openssl x509 -pubkey -noout | openssl pkey -pubin -outform der | \ openssl dgst -sha256 -binary | openssl enc -base64
…this command returned “g5lqtwHia/plKqWU/Fe2Woh4+7MO3d0JYqYJpj/iYAw=” that we entered into stubby.conf to connect it to our own recursor.
So the upstream_recursive_servers: section looked like this:
{ address_data: 2001:67c:27e4::35
, tls_auth_name: "privacydns.go6lab.si"
, tls_pubkey_pinset:
[ { digest: "sha256"
, value: g5lqtwHia/plKqWU/Fe2Woh4+7MO3d0JYqYJpj/iYAw=
} ]
}
…but after restarting stubby, it did not want to connect, returning:
macbook# openssl version OpenSSL 1.0.2l 25 May 2017 macbook# sudo ./bin/stubby [14:10:13.439329] STUBBY: Read config from file /opt/stubby/etc/stubby/stubby.conf [14:10:13.440072] STUBBY: Starting DAEMON.... [14:10:16.967643] STUBBY: 2001:67c:27e4::35 : Conn init : Transport=TLS - Profile=Strict [14:10:16.981397] STUBBY: 2001:67c:27e4::35 : Conn closed : Transport=TLS - *Failure* [14:10:16.981429] STUBBY: *FAILURE* no valid transports or upstreams available! [14:10:16.981513] STUBBY: 2001:67c:27e4::35 : Conn closed : Transport=TLS - Resp=0,Timeouts=0,Auth=None,Keepalive(ms)=0 [14:10:16.981520] STUBBY: 2001:67c:27e4::35 : Upstream stats: Transport=TLS - Resp=0,Timeouts=0,Best_auth=None
This was the result of not having an intermediate certificate in the cert file. We hoped that appending the Let’s Encrypt intermediate certificate to our pem key would solve the issue, but unfortunately this didn’t until we upgraded Unbound from default CentOS7 version to 1.6.5 as the old version does not support intermediate certificates,
You can test if your Unbound server has the intermediate cert support using:
gnutls-cli --print-cert -p 853 [server_IP_addr]
If this returns many certificates, than you are good to go 😉
Currently Stubby consists of all the settings that can be provided with the getdns_context_set_* functions described in Section 8 of the specification:
https://getdnsapi.net/documentation/spec/#8-dns-contexts
…plus all the things that can be specified via extensions, as described in Section 3 of the specification:
https://getdnsapi.net/documentation/spec/#3-extensions
The format uses a JSON like syntax, with a few extensions to allow for IP addresses, domain names, hex and base64 data, and getdns constant names to be given directly. Details about those extensions are described here:
https://getdnsapi.net/doxygen/group__Ustring2getdns__data.html
However, the plan for Stubby is to allow the configuration to be provided in yaml, which is not only a lot more user-friendly than JSON, but also allows comments so an example configuration file can be shipped describing all the options and their default values.
Once Stubby started working, we also setup DNS-over-TLS on our internal recursive resolvers and all was fine; at least until we closed the lid of the Macbook and reopened it again after 10 minutes when nothing worked. Connectivity was there, but Stubby did not return any responses to our DNS queries and the whole thing just did not wake up properly from the system “sleep/suspend”. The developers say this problem will be fixed, but currently that’s not the case and so we’ve decided to freeze the usage of Stubby until this is resolved.
More information on how to configure Stubby can be found on DNSPrivacy.org web page
So, how to setup a local resolver with DNS-over-TLS support that would actually run on a laptop for everyday work? I remembered that Unbound also has this “ssl-upstream:” directive in it’s config file, and as we’ve been using Unbound as a local resolver on our MacBook for many years anyway, we decided to try configuring that setting as “yes” and see what happens.
ssl-upstream: yes forward-zone: name: "." forward-addr: 2001:67c:27e4::35@853 forward-addr: 91.239.96.35@853 forward-ssl-upstream: yes
Without any certificate digests or other complications, our local resolver started working and sending queries to the recursive server in go6lab over TLS and port 853 – no drama, no questions asked 😉 If you require more upstreams, just add them to the list of “forward-addr” directives.
We are eagerly awaiting Stubby to get fixed and it will provide the out-of-order processing and also some other “goodies” that Unbound (currently) does not have.
We’ve been also told that adding “dnssec_return_status: GETDNS_EXTENSION_TRUE” to stubby.conf turns on DNSSEC validation in Stubby, but then we’d still need to do the trust-anchor management manually (with unbound-anchor for example). The upcoming getdns 1.2.0 will do DNSSEC validation without needing any prior configuration and without needing external/manual trust-anchor management, so let’s hope the new version comes out ASAP so we can do the test again!
A big shout-out to Willem Toorop from NLnet Labs and Sara Dickinson from SINODUN for their support, some of the text in this article, and reviewing everything!
You can install your own experimental DNS recursive resolver that serves DNS queries over TLS and port 853, and get on the list of servers at https://dnsprivacy.org/wiki/display/DP/DNS+Privacy+Test+Servers by sending an email to the DNSPrivacy team.
Jan Žorž
Editorial addition: During the reviews of the above text, additional information on configuring Stubby and Unbound was added to the DNSprivacy web pages as a result of our effort. Well done!