DevHeads.net

Reminder DNSSEC Root KSK roll today

In case you've not seen this many other places, just a friendly
reminder that ICANN is rolling the DNSSEC root KSK today. Make
sure your resolver (if it is validating) is ready. If you're
forwarding queries to an upstream resolver, you might also check
that the upstream is ready.

Comments

Re: Reminder DNSSEC Root KSK roll today

By Viktor Dukhovni at 10/11/2018 - 11:40

The new root zone is now live, with the DNSKEY RRset signed with
KSK2017 (id 20326), rather than KSK2010 (id 19036):

Before: <a href="http://dnsviz.net/d/root/W79zYQ/dnssec/" title="http://dnsviz.net/d/root/W79zYQ/dnssec/">http://dnsviz.net/d/root/W79zYQ/dnssec/</a>
During: <a href="http://dnsviz.net/d/root/W79zmg/dnssec/" title="http://dnsviz.net/d/root/W79zmg/dnssec/">http://dnsviz.net/d/root/W79zmg/dnssec/</a>
After: <a href="http://dnsviz.net/d/root/W790GQ/dnssec/" title="http://dnsviz.net/d/root/W790GQ/dnssec/">http://dnsviz.net/d/root/W790GQ/dnssec/</a>

As cached data expires, this should make its way into all working
caches over the next day or two.

Re: Reminder DNSSEC Root KSK roll today

By pg151 at 10/11/2018 - 12:27

On Thu, Oct 11, 2018, at 9:40 AM, Viktor Dukhovni wrote:
Thx for the reminder ... seems quite timely!

Can you comment just a bit further on 'ready'?

Literally not long after I received your notice above bout the roll, here, all queries stopped working, and server can't be restarted.

Logs on, e.g.,

dig A google.com

contain

...
Oct 11 10:09:00 ns01 named[4116]: 11-Oct-2018 10:09:00.435 resolver: debug 1: fetch: google.com/A
Oct 11 10:09:00 ns01 named[4116]: 11-Oct-2018 10:09:00.484 dnssec: info: view internal: validating com/DS: bad cache hit (./DNSKEY)
Oct 11 10:09:00 ns01 named[4116]: 11-Oct-2018 10:09:00.484 lame-servers: info: broken trust chain resolving 'google.com/A/IN': 2001:4860:4802:36::a#53
Oct 11 10:09:00 ns01 named[4116]: 11-Oct-2018 10:09:00.484 query-errors: debug 1: client @0x7efc441cd640 ::1#63498 (google.com): view internal: query failed (SERVFAIL) for google.com/IN/A at query.c:10692
...

Which seems related to the key roll.

Changing my local dns (named) config to

- dnssec-enable yes;
+ dnssec-enable no;
dnssec-lookaside no;
- dnssec-validation yes;
+ dnssec-validation no;

gets me back up & running, without DNSSEC of course.

Is 'ready' simply .... 'wait awhile' ?

Re: Reminder DNSSEC Root KSK roll today

By Jamie Nelson at 10/11/2018 - 13:03

This may help

<a href="https://www.icann.org/dns-resolvers-checking-current-trust-anchors" title="https://www.icann.org/dns-resolvers-checking-current-trust-anchors">https://www.icann.org/dns-resolvers-checking-current-trust-anchors</a>

Jamie

Re: Reminder DNSSEC Root KSK roll today

By pg151 at 10/11/2018 - 13:09

On Thu, Oct 11, 2018, at 11:03 AM, Jamie Nelson wrote:
was JUST looking for that! thx.

Re: Reminder DNSSEC Root KSK roll today

By Viktor Dukhovni at 10/11/2018 - 12:58

By "ready" I mean that it has a working "rfc5011" key rollover
implementation, and so has already long added KSK2017 to its list
of root trust anchors. Or alternatively, that some attentive
administrator has manually updated the trust-anchor file.

This does not look like a forwarder problem, your own trusted key
list is not up to date. Either it is manually maintained, or
automated updates are failing (perhaps permission problems to update
the files, the keys need to be writable by the running nameserver).

Definitely.

Waiting won't fix anything, you need to fix the trust anchor
management on your system.

Re: Reminder DNSSEC Root KSK roll today

By pg151 at 10/11/2018 - 13:08

On Thu, Oct 11, 2018, at 10:58 AM, Viktor Dukhovni wrote:
noted.

on it.

thx.

Re: Reminder DNSSEC Root KSK roll today

By Jim Reid at 10/11/2018 - 12:53

On 11 Oct 2018, at 18:27, <a href="mailto:pg151@dev-mail.net">pg151@dev-mail.net</a> wrote:
Although switching off DNSSEC validation will keep the mail flowing, it only kludges around the underlying problem. Which might or might not be related to the rollover of the root KSK a few hours ago. It’s hard to tell from the information you’ve provided. That said, you do appear to have a DNS server misconfiguration which is causing DNSSEC validation to fail. Clearly it would be wise to fix that before turning DNSSEC validation on again.

The switch to the new KSK seems the most likely cause, assuming DNSSEC validation always worked for you before then.

Maybe, maybe not. It depends on what is broken in your DNSSEC setup. If you’ve hard-wired the now dead root KSK, waiting a while won’t help. That key will still be dead when you re-enable DNSSEC validation. No matter how long or short you wait.

Consult ICANN’s web pages on the root KSK rollover. They have info on how to check that DNS configurations handle the KSK rollover properly and how to troubleshoot them when they don’t.

Re: Reminder DNSSEC Root KSK roll today

By pg151 at 10/11/2018 - 13:07

On Thu, Oct 11, 2018, at 10:53 AM, Jim Reid wrote:
It's been 'working' for ages. Yes, I could have been 'just lucky for a long time'. Bears looking at certainly.

Isn't 'hardwired' here afaict. Looking at the ICANN site -- again -- is probably best advice.

Thx!

Re: Reminder DNSSEC Root KSK roll today

By Bill Cole at 10/11/2018 - 16:33

Since you're running BIND, <a href="https://kb.isc.org/docs/aa-01182" title="https://kb.isc.org/docs/aa-01182">https://kb.isc.org/docs/aa-01182</a> might be
more specifically helpful, although I'm not sure that you can recover
from the key having actually rolled at this point by just setting
"dnssec-validation auto;"

Re: Reminder DNSSEC Root KSK roll today

By pg151 at 10/11/2018 - 17:44

On Thu, Oct 11, 2018, at 2:33 PM, Bill Cole wrote:

Managed to get it all sorted.

manually deleted the existing bind.keys, DL'd and copied over a current/updated copy, made sure all jnl's were synced, switched validation -> auto, and restarted.

resolver's up, running & working now, as least as verified with the usual

dig @127.0.0.1 dnssec-failed.org a +dnssec

not clear if all of that^ was needed, but it apparently did the trick.

thanks all.

Re: Reminder DNSSEC Root KSK roll today

By Viktor Dukhovni at 10/11/2018 - 17:51

Check the user "named" runs as after chroot and dropping privs has
write permissions to update the root trust-anchor file (may need
write permissions to the containing directory to make the update
atomic).

Re: Reminder DNSSEC Root KSK roll today

By pg151 at 10/11/2018 - 18:07

On Thu, Oct 11, 2018, at 3:51 PM, Viktor Dukhovni wrote:
thanks! I _think_ I'm set

ps aux | grep named
named 2561 0.0 0.3 243468 48724 ? Ssl 13:47 0:05 /usr/local/bind9/sbin/named -f -t /var/chroot/named -n 2 -S 1024 -u named -c /etc/named.conf

ls -al \
/var/chroot/named/keys/managed-keys/external.mkeys

-rw-r--r-- 1 named named 1.4K Oct 11 13:47 /var/chroot/named/keys/managed-keys/external.mkeys

where, given the bind.keys' init,

ls -al /usr/local/etc/named/bind.keys
-rw-r--r-- 1 named named 3.9K Oct 11 12:28 /usr/local/etc/named/bind.keys

matches in chroot,

cat /var/chroot/named/keys/managed-keys/external.mkeys
$ORIGIN .
$TTL 0 ; 0 seconds
@ IN SOA . . (
2 ; serial
0 ; refresh (0 seconds)
0 ; retry (0 seconds)
0 ; expire (0 seconds)
0 ; minimum (0 seconds)
)
KEYDATA 20181012204732 20181011204732 19700101000000 257 3 8 (
AwEAAagAIKlVZrpC6Ia7gEzahOR+9W29euxhJhVVLOyQ
bSEW0O8gcCjFFVQUTf6v58fLjwBd0YI0EzrAcQqBGCzh
/RStIoO8g0NfnfL2MTJRkxoXbfDaUeVPQuYEhg37NZWA
JQ9VnMVDxP/VHL496M/QZxkjf5/Efucp2gaDX6RS6CXp
oY68LsvPVjR0ZSwzz1apAzvN9dlzEheX7ICJBBtuA6G3
LQpzW5hOA2hzCTMjJPJ8LbqF6dsV6DoBQzgul0sGIcGO
Yl7OyQdXfZ57relSQageu+ipAdTTJ25AsRTAoub8ONGc
LmqrAmRLKBP1dfwhYB4N7knNnulqQxA+Uk1ihz0=
) ; KSK; alg = RSASHA256; key id = 19036
; next refresh: Fri, 12 Oct 2018 20:47:32 GMT
; trusted since: Thu, 11 Oct 2018 20:47:32 GMT
KEYDATA 20181012204732 20181011204732 19700101000000 257 3 8 (
AwEAAaz/tAm8yTn4Mfeh5eyI96WSVexTBAvkMgJzkKTO
iW1vkIbzxeF3+/4RgWOq7HrxRixHlFlExOLAJr5emLvN
7SWXgnLh4+B5xQlNVz8Og8kvArMtNROxVQuCaSnIDdD5
LKyWbRd2n9WGe2R8PzgCmr3EgVLrjyBxWezF0jLHwVN8
efS3rCj/EWgvIWgb9tarpVUDK/b58Da+sqqls3eNbuv7
pr+eoZG+SrDK6nWeL3c6H5Apxz7LjVc1uTIdsIXxuOLY
A4/ilBmSVIzuDWfdRUfhHdY6+cn8HFRm+2hM8AnXGXws
9555KrUB5qihylGa8subX2Nn6UwNR1AkUTV74bU=
) ; KSK; alg = RSASHA256; key id = 20326
; next refresh: Fri, 12 Oct 2018 20:47:32 GMT
; trusted since: Thu, 11 Oct 2018 20:47:32 GMT

Re: Reminder DNSSEC Root KSK roll today

By Jim Reid at 10/11/2018 - 14:14

DNSSEC is very brittle. Either it works perfectly or not at all. Luck has nothing to do with it. Ending up with a working DNSSEC setup is something that rarely if ever happens by accident. If your validators don’t/can’t maintain up to date trust anchors they *will* fail at some point. Today might well have been that day for you.

Ensuring trust anchor(s) are current is critical to DNSSEC validation. It’s not a matter of luck if this doesn’t get configured correctly. And it’s not a matter of luck if someone’s failed to plan for today’s KSK rollover or been unaware of this high profile event. Sorry.