[enhancement] Allow MAAS DNS to resolve from subnets without modeling their fabric and vlan

[Jason Hobbs]

When operating as a forwarding DNS server, maas refuses queries from subnets it doesn't know about. A subnet must be added to MAAS in order for it to answer.

This is inconvenient because it requires adding subnets to MAAS which MAAS will never manage otherwise, and may not be known ahead of time. For example, when using DNS-HA OpenStack deployed on MAAS, all of the subnets OpenStack clients might come from must be added to MAAS so that MAAS can be used as the DNS server for those clients, which must also resolve non MAAS owned domains, even though MAAS will never own machines on those subnets, and we may not know all the subnets that will use the OpenStack.

Typically a network admin would control such access using firewall rules. In this case, they would also have to add subnets to MAAS.

I'm not sure why MAAS does this - it's not clear. It seems like ACLs like this should be left up to firewalls. There should at least be an option to disable this.

[Andres Rodriguez (andreserl)]

@Jason,

I wonder if what you are running against is https://bugs.launchpad.net/maas/+bug/1670886

Can you try manually using that option? If so we can fix MAAS to do it by default.

[Jason Hobbs (jason-hobbs)]

No, that's not what I'm running into. I'm running into this:

/etc/bind/maas/named.conf.options.inside.maas:allow-recursion { trusted; };

in /etc/bind/maas/named.conf.maas
# Access control for recursive queries. See named.conf.options.inside.maas
# for the directives used on this ACL.
acl "trusted" {
    10.245.208.0/20;
    192.168.36.0/26;
    192.168.33.0/24;
    10.244.40.0/21;
    10.244.32.0/21;
    192.168.35.0/26;
    localnets;
    localhost;
};

in src/maasserver/dns/config.py:
def get_trusted_networks():
    """Return the CIDR representation of all the Subnets we know about.

    :return: A list of CIDR-format subnet specifications.
    """
    return [
        str(subnet.cidr)
        for subnet in Subnet.objects.all()
    ]

which results in this:
ubuntu@production-cpe-62b5e6e9-20a2-4eb4-b857-9eefcaf5c0c6:~/cpe$ nslookup
> server 10.244.40.30
Default server: 10.244.40.30
Address: 10.244.40.30#53
> api.jujucharms.com
Server: 10.244.40.30
Address: 10.244.40.30#53

** server can't find api.jujucharms.com: REFUSED

and in the syslog on the maasserver this:
May 30 15:03:01 leafeon named[4730]: client 91.189.92.9#47343 (api.jujucharms.com): query (cache) 'api.jujucharms.com/A/IN' denied

[Jason Hobbs (jason-hobbs)]

The workaround in bug 1670886 has no effect here.

[Andres Rodriguez (andreserl)]

Ok, so I need some info:

1. Are you setting a forwarder ?
2. Can you provide /etc/bind/*

Also, AFAIK recursion happens when the MAAS dns server takes upon itself to resolve DNS queries for its clients. However, when using a forwarder, it would forward the request to other DNS servers. So in this case, if we have two things:

1. recursion - it would only work for maas known networks
2. forwarding - it would forward the not-known maas networks to the upstream DNS server

[Jason Hobbs (jason-hobbs)]

Yes we're setting an upstream dns server, and it works fine when I make the request from a client that MAAS has the subnet listed for.

ubuntu@production-cpe-62b5e6e9-20a2-4eb4-b857-9eefcaf5c0c6:~$ maas root maas get-config name=upstream_dns
Success.
Machine-readable output follows:
"10.245.208.1"

/etc/bind attached.

[Andres Rodriguez (andreserl)]

I've tried to reproduce this issue, and I'm unable to reproduce it. As such, I do not believe this is an issue with MAAS:

root@xenial:~# ip addr sh
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
43: eth0@if44: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:42:11:27 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.10.18/24 brd 192.168.10.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe42:1127/64 scope link
       valid_lft forever preferred_lft forever

root@xenial:~# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 192.168.1.13
search maaslab maas

root@xenial:~# ping br1.maas00.maas
PING br1.maas00.maas (192.168.1.13) 56(84) bytes of data.
64 bytes from br1.maas00.maas (192.168.1.13): icmp_seq=1 ttl=63 time=0.781 ms
64 bytes from br1.maas00.maas (192.168.1.13): icmp_seq=2 ttl=63 time=0.888 ms

======================

ubuntu@maas00:/etc/bind/maas$ cat named.conf.maas
include "/etc/bind/maas/named.conf.rndc.maas";

# Zone declarations.
zone "maas" {
    type master;
    file "/etc/bind/maas/zone.maas";
};
zone "maaslab" {
    type master;
    file "/etc/bind/maas/zone.maaslab";
};
zone "90.90.10.in-addr.arpa" {
    type master;
    file "/etc/bind/maas/zone.90.90.10.in-addr.arpa";
};
zone "100.168.192.in-addr.arpa" {
    type master;
    file "/etc/bind/maas/zone.100.168.192.in-addr.arpa";
};
zone "1.168.192.in-addr.arpa" {
    type master;
    file "/etc/bind/maas/zone.1.168.192.in-addr.arpa";
};

# Access control for recursive queries. See named.conf.options.inside.maas
# for the directives used on this ACL.
acl "trusted" {
    192.168.100.0/24;
    192.168.1.0/24;
    10.90.90.0/24;
    localnets;
    localhost;
};

[Andres Rodriguez (andreserl)]

root@xenial:~# nslookup br1.maas00.maas
Server: 192.168.1.13
Address: 192.168.1.13#53

Name: br1.maas00.maas
Address: 192.168.1.13

root@xenial:~# dig @192.168.1.13 br1.maas00.maas

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @192.168.1.13 br1.maas00.maas
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45246
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;br1.maas00.maas. IN A

;; ANSWER SECTION:
br1.maas00.maas. 30 IN A 192.168.1.13

;; AUTHORITY SECTION:
maas. 30 IN NS maas.

;; ADDITIONAL SECTION:
maas. 30 IN A 192.168.1.13

;; Query time: 3 msec
;; SERVER: 192.168.1.13#53(192.168.1.13)
;; WHEN: Wed May 30 16:12:15 UTC 2018
;; MSG SIZE rcvd: 90

[Andres Rodriguez (andreserl)]

Also, could you clarify something:

1. What is the MAAS version in use?
2. What is the client operating system?
3. What's the client /etc/resolv.conf configured?
4. Since you are talking about DNS HA, my understand that DNS HA simply meant that Openstack would use MAAS as a source for DNS for its own service, and not for the clients, can you please clarify whether this works and this issue is only for the openstack instances trying to use MAAS as DNS ?

[Andres Rodriguez (andreserl)]

From the same client that's on a subnet not owned by MAAS:

root@xenial:~# dig @192.168.1.13 api.jujucharms.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @192.168.1.13 api.jujucharms.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7870
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 13, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;api.jujucharms.com. IN A

;; ANSWER SECTION:
api.jujucharms.com. 519 IN A 162.213.33.121

;; AUTHORITY SECTION:
com. 172028 IN NS e.gtld-servers.net.
com. 172028 IN NS k.gtld-servers.net.
com. 172028 IN NS f.gtld-servers.net.
com. 172028 IN NS c.gtld-servers.net.
com. 172028 IN NS b.gtld-servers.net.
com. 172028 IN NS l.gtld-servers.net.
com. 172028 IN NS h.gtld-servers.net.
com. 172028 IN NS j.gtld-servers.net.
com. 172028 IN NS d.gtld-servers.net.
com. 172028 IN NS i.gtld-servers.net.
com. 172028 IN NS g.gtld-servers.net.
com. 172028 IN NS a.gtld-servers.net.
com. 172028 IN NS m.gtld-servers.net.

;; Query time: 4 msec
;; SERVER: 192.168.1.13#53(192.168.1.13)
;; WHEN: Wed May 30 17:17:33 UTC 2018
;; MSG SIZE rcvd: 287

root@xenial:~# nslookup api.jujucharms.com
Server: 192.168.1.13
Address: 192.168.1.13#53

Non-authoritative answer:
Name: api.jujucharms.com
Address: 162.213.33.121

[Jason Hobbs (jason-hobbs)]

1. 2.3.3
2. Ubuntu, any version
3. Irrelevant; you don't have to use /etc/resolv.conf, just use nslookup and set the nameserver directly:

jason@inire:~$ nslookup
> server 10.245.0.10
Default server: 10.245.0.10
Address: 10.245.0.10#53
> api.jujucharms.com
Server: 10.245.0.10
Address: 10.245.0.10#53

** server can't find api.jujucharms.com: REFUSED

4. No, DNS-HA for OpenStack means each openstack service gets a dns entry managed by maas (for example, keystone-public.maas). The charms pick which unit's IP it resolves to at any given point in time, and tell maas to keep that entry updated. Any clients talking to OpenStack use the DNS entry for it rather than IPs directly, so they always get the IP the charms have told maas to point the hostname at.

[Andres Rodriguez (andreserl)]

Seems that by setting:

allow-recursion { trusted; };
allow-query-cache { trusted; };

to

allow-recursion { any; };
allow-query-cache { any; };

allows this to happen. The concern I have about these options is the fact that doing so would allow *any* user to use MAAS DNS as a resolver. So if MAAS were to be on the internet, it would allow everybody to use it as a DNS server.

It doesn't seem like there is, but i think its worth talking to IS to see if there's anything else that could be done to address this.

[Jason Hobbs (jason-hobbs)]

The whole thing of MAAS 'trusting' any subnet it knows about isn't documented anywhere, and seems a little arbitrary.

[Andres Rodriguez (andreserl)]

FWIW:

https://bugs.launchpad.net/maas/+bug/1348364

and specially:

https://bugs.launchpad.net/maas/+bug/1348364/comments/1

Based on comment #1 in the above bug, I would be very concerned adding this option.

[Jason Hobbs (jason-hobbs)]

This big isn't about adding that option - that's just one possible solution.

This bug is about having to model subnet, vlan and fabric for every
client that needs to use the maas server for DNS.

I could see MAAS having a whitelist of allowed subnets, but it doesn't
make sense to require fabric and vlan.

On Wed, May 30, 2018 at 4:08 PM, Andres Rodriguez
<email address hidden> wrote:
> ** Tags added: dns
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1774206
>
> Title:
> [enhancement] Allow MAAS DNS to resolve from subnets it doesn't know
> about (or trust)
>
> Status in MAAS:
> Triaged
>
> Bug description:
> When operating as a forwarding DNS server, maas refuses queries from
> subnets it doesn't know about. A subnet must be added to MAAS in order
> for it to answer.
>
> This is inconvenient because it requires adding subnets to MAAS which
> MAAS will never manage otherwise, and may not be known ahead of time.
> For example, when using DNS-HA OpenStack deployed on MAAS, all of the
> subnets OpenStack clients might come from must be added to MAAS so
> that MAAS can be used as the DNS server for those clients, which must
> also resolve non MAAS owned domains, even though MAAS will never own
> machines on those subnets, and we may not know all the subnets that
> will use the OpenStack.
>
> Typically a network admin would control such access using firewall
> rules. In this case, they would also have to add subnets to MAAS.
>
> I'm not sure why MAAS does this - it's not clear. It seems like ACLs
> like this should be left up to firewalls. There should at least be an
> option to disable this.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1774206/+subscriptions

[Andres Rodriguez (andreserl)]

FWIW, the solution will be to allow whitelisting subnets not known to MAAS, by having a global option which will allow you to specify custom subnets.