dnssec failures cause nodes to be unable to resolve external addresses

Bug #1384334 reported by Jeff Lane on 2014-10-22
42
This bug affects 8 people
Affects Status Importance Assigned to Milestone
MAAS
Critical
Mike Pontillo

Bug Description

https://pastebin.canonical.com/119180/

MAAS 1.5.4 running on Trusty, fresh default install, the only things we've done are standard config (setting up network, adding DNS forwarder address, and then adding nodes once images were synced).

The problem was that I was unable to resolve addresses using the MAAS DNS forwarder. For example:
ubuntu@utsa-maas:/etc/bind$ dig @192.168.1.1 streams.canonical.com

; <<>> DiG 9.9.5-3-Ubuntu <<>> @192.168.1.1 streams.canonical.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 10035
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;streams.canonical.com. IN A

;; Query time: 0 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Wed Oct 22 11:32:29 CDT 2014
;; MSG SIZE rcvd: 50

I would get that, and a corresponding error in /var/log/syslog:
Oct 22 11:18:20 utsa-maas named[41376]: error (insecurity proof failed) resolving './NS/IN': 10.241.1.1#53

I believe the real cause is that dnssec-validation is set to auto by default in bind9, and those dnssec queries failed, which caused nodes in the MAAS cluster to be unable to resolve external addresses. This, in turn, caused juju bootstrap to fail:

Setting up libgoogle-perftools4 (2.1-2ubuntu1) ...
Setting up libsnappy1 (1.1.0-1ubuntu1) ...
Setting up juju-mongodb (2.4.9-0ubuntu3) ...
Processing triggers for libc-bin (2.19-0ubuntu6.3) ...
curl: (6) Could not resolve host: streams.canonical.com
tools from https://streams.canonical.com/juju/tools/releases/juju-1.18.4-trusty-amd64.tgz downloaded: HTTP 000; time 9.519s; size 0 bytes; speed 0.000 bytes/s ERROR bootstrap failed: rc: 1
Stopping instance...
ERROR rc: 1

After talking about this with Kiko, I set dnssec-validation to no:

dnssec-validation no;

in /etc/bind/named.conf.options

and after restarting bind, external forwarding started working properly.

Related branches

Andres Rodriguez (andreserl) wrote :

Did you add upstream DNS in MAAS settings page?

Jeff Lane (bladernr) wrote :

Yes... on the settings page I added in UTSA's DNS server. This was confirmed in the options file:

ubuntu@utsa-maas:/etc/bind/maas$ cat named.conf.options.inside.maas
    forwarders {
        10.241.1.1;
    };

Changed in maas:
status: New → Triaged
importance: Undecided → High
tags: added: dns trivial
Changed in maas:
milestone: none → next
Andres Rodriguez (andreserl) wrote :

So there's a few people who have hit this issue... I think it would probably be a good idea to add an option to either enable/disable DNSSEC validation on the config.. what do you think?

Narinder Gupta (narindergupta) wrote :

even at HP environmment i have top change it to dnssec-validation no; to make it work from day 1.

Graham Binns (gmb) wrote :

Julian, you've marked this as trivial, but it's not clear why. Do you think this is just a template change (as opposed to Andres's suggestion of a config option)?

Julian Edwards (julian-edwards) wrote :

Graham, either way it's trivial. The Config stuff makes it easy.

Changed in maas:
milestone: next → 1.8.0
importance: High → Critical
Changed in maas:
assignee: nobody → Mike Pontillo (mpontillo)
Changed in maas:
status: Triaged → In Progress
Changed in maas:
status: In Progress → Fix Committed
Changed in maas:
status: Fix Committed → Fix Released
Alexander List (alexlist) wrote :

This affects the current MAAS delivered with trusty (1.7) - can we expect a SRU?

Mike Pontillo (mpontillo) wrote :

We added the option to the MAAS settings page as a convenience for users, since we also allow users to specify forwarders there. (it made sense that, when specifying forwarders, you should consider whether or not those forwarders support DNSSEC.)

Since this wasn't really a MAAS problem (rather, a BIND configuration option that MAAS will now manage for users), I'm not sure an SRU can be justified. So I suggest using the workaround in the description. That is, set the following option in /etc/bind/named.conf.options:

dnssec-validation no;

Either that, or just use MAAS 1.8 out the gate. (it's in the 'stable' PPA.) Note that later when you upgrade to MAAS 1.8+, we'll attempt to parse the existing named.conf.options and migrate your DNSSEC setting to the MAAS database.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers