NM fails to start dnsmasq such that it listens on ::1

[strainu]

I have IPv6 activated on Ubuntu 12.04 LTS and I receive an IPv6 address through DHCPv6. However, requests sent through IPv6 fail. For example:

strainu@emily:~$ dig aaaa -6 google.com

; <<>> DiG 9.8.1-P1 <<>> aaaa -6 google.com
;; global options: +cmd
;; connection timed out; no servers could be reached

But:

strainu@emily:~$ dig aaaa google.com

; <<>> DiG 9.8.1-P1 <<>> aaaa google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60426
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 4

;; QUESTION SECTION:
;google.com. IN AAAA

;; ANSWER SECTION:
google.com. 94 IN AAAA 2a00:1450:400d:805::1000

Looking into /etc/resolv.conf, I see:

strainu@emily:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1
search lan

And with netstat:

strainu@emily:~$ netstat -ln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN
tcp6 0 0 ::1:631 :::* LISTEN
udp 0 0 127.0.0.1:53 0.0.0.0:*
udp 0 0 0.0.0.0:68 0.0.0.0:*
udp 0 0 0.0.0.0:5353 0.0.0.0:*
udp 0 0 0.0.0.0:38399 0.0.0.0:*
udp6 0 0 :::47973 :::*
udp6 0 0 :::5353 :::*

Apparently, the local dnsmasq server does not listen on IPv6, which is wrong. The server should also listen on ::1:53, to allow IPv6-only connections. Alternatively, the resolvconf package should write the actual DNS server received through DHCPv6 in /etc/resolv.conf

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: resolvconf 1.63ubuntu14
ProcVersionSignature: Ubuntu 3.2.0-24.39-generic 3.2.16
Uname: Linux 3.2.0-24-generic x86_64
ApportVersion: 2.0.1-0ubuntu8
Architecture: amd64
Date: Sat Jun 9 02:27:02 2012
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release amd64 (20111012)
PackageArchitecture: all
SourcePackage: resolvconf
UpgradeStatus: Upgraded to precise on 2012-05-28 (11 days ago)

[Steve Langasek (vorlon)]

> Apparently, the local dnsmasq server does not listen on IPv6, which is wrong.

This may be wrong, but what should 'dig -6' do when the only nameserver specified in /etc/resolv.conf is an IPv4 address? I think the expected behavior here is underspecified.

[strainu (strainu)]

The expected behavior is to have an IPv6 server in /etc/resolv.conf. Quoting from man dig:

A typical invocation of dig looks like:

    dig @server name type

where:

server
    is the name or IP address of the name server to query. [...] If no server argument is provided, dig consults /etc/resolv.conf and queries the name servers listed there.

[strainu (strainu)]

I feel that I should elaborate. The final behavior that I'm looking for is to have a DNS server with IPv6 address active in my system. This means that an IPv6 address should be placed in /etc/resolv.conf. This can happen in a number of ways:

* the user adds one manually (through resolvconf or Network Manager) - OK as a temporary solution, but does not scale well
* the dnsmasq server used by Ubuntu listens on IPv6 as well and the line "nameserver=::1" is added in /etc/resolv.conf - looks like the simplest solution to me
* the dnsmasq server used by Ubuntu does _not listen on IPv6_, and then the IPv6 server received by DHCPv6 (available in /run/nm-dns-dnsmasq.conf) is used in /etc/resolv.conf - this solution might need further changes in resolvconf and breaks the behavior designed in [1] for IPv6-only clients

[1] https://blueprints.launchpad.net/ubuntu/+spec/foundations-p-dns-resolving

[Thomas Hood (jdthood)]

Strainu, thanks for the report.

Strainu wrote:
> Apparently, the local dnsmasq server does not listen on IPv6, which is wrong. The server should also listen on ::1:53, to allow IPv6-only connections.

The problem is certainly not that dnsmasq lacks IPv6 DNS *support*. It has very good support.

To begin with, the dnsmasq documentation says: "On IPv6-enabled boxes, dnsmasq can both talk to upstream servers via IPv6 and offer DNS service via IPv6. On dual-stack (IPv4 and IPv6) boxes it talks both protocols and can even act as IPv6-to-IPv4 or IPv4-to-IPv6 forwarder."

At the moment (due to a disk failure, but that's another story) I'm running Ubuntu 11.10 with network-manager and standalone dnsmasq (i.e., not dnsmasq run by NM). Dnsmasq *does* listen on ::1:53. The command

    netstat -nl

includes the following lines in its output.

    tcp6 0 0 :::53 :::* LISTEN
    udp6 0 0 :::53 :::*

The following command

    dig aaaa -6 google.com

works fine, giving the following output.

    ; <<>> DiG 9.7.3 <<>> aaaa -6 google.com
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11095
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

    ;; QUESTION SECTION:
    ;google.com. IN AAAA

    ;; ANSWER SECTION:
    google.com. 274 IN AAAA 2a00:1450:4007:804::1009

    ;; Query time: 1 msec
    ;; SERVER: ::1#53(::1)
    ;; WHEN: Sun Jun 10 21:53:03 2012
    ;; MSG SIZE rcvd: 56

The commands

    dig @127.0.0.1 aaaa -6 google.com
    dig @::1 aaaa -6 google.com

work equally well.

The following commands all work equally well.

    host google.com
    host google.com 127.0.0.1
    host google.com ::1

Furthermore the following works equally well

    ping google.com

whether "127.0.0.1" or "::1" is listed in /etc/resolv.conf.

Strainu wrote:
> Alternatively, the resolvconf package should write the actual DNS server [address] received through DHCPv6 in /etc/resolv.conf

Even if dnsmasq didn't support IPv6 this would not be an attractive solution since it would be inconsistent with the way the IPv4 case was handled.

Continuing in another comment.

[Thomas Hood (jdthood)]

There is no evidence in the report of a bug in resolvconf.

If dnsmasq is not responding at ::1 then this may be because dnsmasq has been started with the "--bind-interfaces" and "--listen-address=127.0.0.1" options which is the case, for example, when dnsmasq is controlled by NetworkManager. Please provide the output of "ps -elf|grep dnsmasq".

When I run the standalone dnsmasq with the aforementioned options, dnsmasq does not respond to either of the following commands

    dig aaaa -6 google.com
    dig @::1 aaaa -6 google.com

but it does respond correctly to the following command.

    dig @127.0.0.1 aaaa -6 google.com

Wireshark confirms that in the first case ("dig aaaa -6 google.com"), dig tries to contact ::1 rather than 127.0.0.1, even though its man page says "If no server argument is provided, dig consults /etc/resolv.conf and queries the name servers listed there."

So it looks as if there is a bug in dig and a bug in network-manager.

The bug in dig is that when "nameserver 127.0.0.1" is listed in /etc/resolv.conf, "dig aaaa -6 google.com" doesn't query 127.0.0.1 but ::1, contrary to what its man page says.

The bug in network-manager is that it starts dnsmasq such that the latter doesn't listen on ::1.

To fix the latter bug it may suffice for NM to invoke dnsmasq with an additional option "--listen-address=::1". With this option added to the other two all the "dig", "host" and "ping" commands I've mentioned in this and in my previous comment work properly.

[Thomas Hood (jdthood)]

See comment #6

[Thomas Hood (jdthood)]

I have reported the dig bug in #1011307.

[strainu (strainu)]

Thomas, you were right about the dnsmasq parameters:

strainu@emily:~$ ps -elf|grep dnsmasq
4 S nobody 6849 6764 0 80 0 - 7573 poll_s 08:24 ? 00:00:00 /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/var/run/sendsigs.omit.d/network-manager.dnsmasq.pid --listen-address=127.0.0.1 --conf-file=/var/run/nm-dns-dnsmasq.conf --cache-size=0 --proxy-dnssec

As to the dig bug, I consider it as being a common sense behavior, as it does not find any IPv6 servers in /etc/resolv.conf and it has to use IPv6 to connect. I will comment further in #1011307.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Thomas Hood (jdthood)]

One way to fix this would be to run standalone dnsmasq in addition to NetworkManager-controlled dnsmasq and have the former forward to the latter. Unlike NM-controlled dnsmasq, standalone dnsmasq listens on all interfaces, including ::1.

At present, standalone dnsmasq can't be started when NetworkManager-controlled dnsmasq is running. This is bug #959037. However, in comment #67 of that report I describe how to get both dnsmasq instances running in cascade.

[Thomas Hood (jdthood)]

Someone please set the Importance to wishlist.

[Josh Hill (ingenium)]

So there's really no way to just change the listen-address parameter for the NetworkManager started dnsmasq? I feel like it has to be in a script somewhere, but so far I've been unable to find the script that NetworkManager uses to start dnsmasq. This is a really annoying bug, as it introduces a noticeable lag whenever a AAAA record has to be resolved, and breaks AAAA resolution for some things.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Thomas Hood (jdthood)]

@Josh: There is no script; NM executes dnsmasq directly.

But this might help you:

network-manager (0.9.6.0~git201207271115.edb85e9-0ubuntu1) quantal; urgency=low

  * upstream snapshot 2012-07-27 11:15:54 (GMT)
    + edb85e972051aff3cfcd5990ea1bd8a28dddc9d9
    - dnsmasq: allow custom configuration via /etc/NetworkManager/dnsmasq.d

[Thomas Hood (jdthood)]

Josh wrote:
> it introduces a noticeable lag whenever a AAAA record has to be resolved, and breaks AAAA resolution for some things.

It doesn't introduce a lag here.

$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.1
$ ps -elf|grep dnsmasq
4 S nobody 3798 1 0 80 0 - 8265 poll_s 12:17 ? 00:00:00 /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/var/run/sendsigs.omit.d/network-manager.dnsmasq.pid --listen-address=127.0.0.1 --conf-file=/var/run/nm-dns-dnsmasq.conf --cache-size=0 --proxy-dnssec --enable-dbus --conf-dir=/etc/NetworkManager/dnsmasq.d

$ time dig a google.com

; <<>> DiG 9.8.1-P1 <<>> a google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63283
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;google.com. IN A

;; ANSWER SECTION:
google.com. 269 IN A 173.194.65.139
google.com. 269 IN A 173.194.65.113
google.com. 269 IN A 173.194.65.100
google.com. 269 IN A 173.194.65.101
google.com. 269 IN A 173.194.65.102
google.com. 269 IN A 173.194.65.138

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Aug 7 12:18:29 2012
;; MSG SIZE rcvd: 124

real 0m0.011s
user 0m0.000s
sys 0m0.008s
$ time dig aaaa google.com

; <<>> DiG 9.8.1-P1 <<>> aaaa google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64409
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;google.com. IN AAAA

;; ANSWER SECTION:
google.com. 145 IN AAAA 2a00:1450:8005::65

;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Aug 7 12:18:31 2012
;; MSG SIZE rcvd: 56

real 0m0.014s
user 0m0.004s
sys 0m0.008s

[Mathieu Trudel-Lapierre (cyphermox)]

I looked at this carefully; and wrote a (I hope) pretty complete blog post explaining the situation:
http://blog.cyphermox.net/2012/08/bug-1010724-why-doesnt-dnsmasq-listen.html

I'll mark prioritize this down to Wishlist for now, since there doesn't in fact appear to be lag caused by the current setup. It's going to be something to revisit later should there be time to look into the situation, but I'm not going to worry too much about it unless there's clearly an issue.

This is just a prioritization for the development work on NetworkManager by Ubuntu developers maintaining the package. It basically just means that *I* won't spend time looking into the situation further, but I'll be happy to sponsor patches should someone come up with a clever way to cover all the issues I noted in the above blog entry; namely:

- It should show up in /etc/resolv.conf; but that also seems like unnecessary duplication of information.
- Starting dnsmasq entries on ::1 may cause issues with other dnsmasq instances.

It was brought to my attention that draft-smith-v6ops-larger-ipv6-loopback-prefix-00 aims at fixing this, but it's going to take a while and lots of work for these solutions to be accepted and implemented.

[miip (cbubel)]

It's 2017 now, this is still broken. Please fix this.