NetworkManager restarts dnsmasq and adds host route on every IPv6 route lookup

Bug #1004775 reported by Steve Atwell
168
This bug affects 28 people
Affects Status Importance Assigned to Milestone
dnsmasq (Ubuntu)
Fix Released
Undecided
Unassigned
Precise
Fix Released
High
Mathieu Trudel-Lapierre
Quantal
Fix Released
Undecided
Unassigned
network-manager (Ubuntu)
Fix Released
High
Mathieu Trudel-Lapierre
Precise
Fix Released
High
Mathieu Trudel-Lapierre
Quantal
Fix Released
High
Mathieu Trudel-Lapierre

Bug Description

[Impact]
This issue affects all users of NetworkManager in a dual stack (IPv4 w/ IPv6 on the same network) environment. When any router advertisement is received, NetworkManager restarts dnsmasq in order to update the DNS configuration it uses.
This change modifies the method of update of dnsmasq to use DBus methods instead of writing a configuration file; which avoids having to restart the daemon to re-read the configuration.

kill -USR1 <dnsmasq pid> can be used to output to syslog the current DNS servers used by dnsmasq.

[Test Case]
1) Run NetworkManager in an dual-stack IPv4/IPv6 environment.
2) Verify that dnsmasq does not get restarted when a router advertisement is received.
3) Verify that DNS server settings are updated when stopping/starting a connection with NM.

[Regression Potential]
This change affects the underlying daemon used by NetworkManager to help with DNS resolution on the system. This daemon is updated via DBus. Since the method of update is changed; this may affect the resulting effective configuration of dnsmasq and break DNS resolution in case of a regression.

----

NetworkManager stops and starts dnsmasq very frequently (every few minutes). During the brief window that dnsmasq is restarting, hostname resolution doesn't work. dnsmasq is getting restarted frequently enough that I see brief hostname resolution errors in Chrome several times a day during normal web browsing.

This is on Precise with network-manager 0.9.4.0-0ubuntu4 and dnsmasq-base 2.59-4.

The frequent restarts seem to be triggered by IPv6 route advertisements on my network. (See log snippet below.) When NetworkManager gets a route advertisement, it reconfigures everything. In the case of dnsmasq, this means stopping dnsmasq, writing a new config file, and starting dnsmasq. NetworkManager should restart dnsmasq only if it its configuration changes.

I'm attaching a snippet from /var/log/daemon.log on my system that demonstrates a restart. This is with debug logging configured in /etc/NetworkManager/NetworkManager.conf.

Revision history for this message
Steve Atwell (satwell) wrote :
Thomas Hood (jdthood)
summary: - NetworkManager restarts dnsmasq very frequently
+ NetworkManager restarts dnsmasq on every IPv6 route advertisement, thus
+ very frequently
Revision history for this message
Launchpad Janitor (janitor) wrote : Re: NetworkManager restarts dnsmasq on every IPv6 route advertisement, thus very frequently

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in network-manager (Ubuntu):
status: New → Confirmed
Revision history for this message
Roman Yepishev (rye) wrote :

In my case I experience dnsmasq restart upon visiting an IPv6-enabled web site (e.g. http://www.sixxs.net) - the routes to the nodes that host web site get added to my local route table and dnsmasq is being restarted by NetworkManager. Are you sure that's happening only on radv for you?

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

The issue reported by Steve Atwell might have been resolved by installing package network-manager 0.9.4.0-0ubuntu4.1; where we apply patches to better handle route changes with new kernels, as well as RDNSS/DNSSL fixes.

Could you please verify that all packages are properly updated and attach /var/log/syslog to this bug report for further information?

It may be useful also to include debugging logs; which can be enabled as such (while NetworkManager is running):

sudo /usr/lib/NetworkManager/debug-helper.py --nm debug

This will generate all the debugging information for NM in /var/log/syslog.

Changed in network-manager (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Stéphane Graber (stgraber) wrote :

Launchpad doesn't seem to like me today and the attach functions is failing...

So anyway, you can find the debug log here: http://paste.ubuntu.com/1061432/

That's an up to date 12.04 system (including the latest NM SRU),

I didn't use the debug helper but simply started NetworkManager in debug mode (properly identical result).

Changed in network-manager (Ubuntu):
status: Incomplete → Triaged
importance: Undecided → High
Changed in network-manager (Ubuntu Precise):
milestone: none → ubuntu-12.04.1
status: New → Triaged
importance: Undecided → High
Revision history for this message
Stéphane Graber (stgraber) wrote :

Targeted for the point release.

Revision history for this message
James Tait (jamestait) wrote :

FWIW, I've checked the "Ignore automatically obtained routes" checkbox in the "Routes..." dialogue. I'm going to see how it goes. So far I've reconnected to Freenode, Facebook and G+ after unplugging my network cable to clear the route list, and so far the route list hasn't grown from the default routes set up when I reconnected the cable.

Changed in network-manager (Ubuntu Precise):
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
Changed in network-manager (Ubuntu Quantal):
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
Revision history for this message
Ben Jencks (bjencks) wrote :

The automatic route adds were causing me trouble (getting in the way of an unmanaged vpn), so I did some digging and experiments.

Without "Ignore automatically obtained routes": any IPv6 route lookup for a host that hasn't been looked up already causes NetworkManager to do two things, both of which are undesirable:
* It restarts dnsmasq
* It adds a host route (/128) for the address that was looked up, via the current default gateway (not sure whether it's based on a kernel route lookup, or NM's idea of the default gateway, since they're congruent in my configuration).

I can reproduce this without sending any traffic, a simple "ip -6 route get foo" triggers it.

Turning on debugging shows that NM receives an add/del route netlink message to trigger this. My guess based on that is that there's a bug in the route handling code, and it's interpreting a message from the kernel saying "this route was looked up" as a message saying "this route was added". Then it adds it to its internal table and syncs the internal table back to the kernel table, adding the spurious route to the kernel.

I don't know enough about netlink messages to actually find the bug, though; trying to read the code has me completely lost.

With "Ignore automatically obtained routes" turned on, it behaves correctly. Route lookups have no special effects, and it still handles the default route it automatically obtained from RAs. Documentation implies that it will reject routes learned via RA and DHCPv6, which it doesn't do.

Ben Jencks (bjencks)
summary: - NetworkManager restarts dnsmasq on every IPv6 route advertisement, thus
- very frequently
+ NetworkManager restarts dnsmasq and adds host route on every IPv6 route
+ lookup
Changed in network-manager (Ubuntu Quantal):
status: Triaged → In Progress
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

I've been working on a workaround to dnsmasq being restarted all the time. Instead of overwriting the dnsmasq configuration and restarting dnsmasq; NM will not write any configuration file but start dnsmasq with --enable-dbus and push DNS changes using DBus. This can still mean there would be a small period of time where the DNS nameservers might not be up to date, but at least it should try to resolve with the old nameservers before the new ones are set.

This won't fix the extra host routes in the IPv6 routing table, but I think we should really consider that as a completely separate bug... (and IMO a cosmetic one, as I haven't been able to pinpoint any actual real-world effect of these extra routes to break anything).

For testing on Precise as I start the SRU process for that fix (it needs to land in quantal first, and takes some time to be reviewed, then will be in proposed and need testing there...); I've already make NetworkManager and an accompanying patch to dnsmasq available in my ppa -- ppa:mathieu-tl/nm . Testing of NM with dnsmasq using the dbus method for updating nameservers would be much appreciated.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package network-manager - 0.9.6.0~git201207271115.edb85e9-0ubuntu1

---------------
network-manager (0.9.6.0~git201207271115.edb85e9-0ubuntu1) quantal; urgency=low

  * upstream snapshot 2012-07-27 11:15:54 (GMT)
    + edb85e972051aff3cfcd5990ea1bd8a28dddc9d9
    - dnsmasq: allow custom configuration via /etc/NetworkManager/dnsmasq.d
  * debian/patches/dnsmasq-dbus-updates.patch: do dnsmasq server updates via
    DBus, instead of respawning the process every time. (LP: #1004775)
  * debian/patches/dnsmasq-vpn-dns-filtering.patch: refreshed.
  * debian/patches/nm-change-dnsmasq-parameters.diff: refreshed.
  * debian/patches/add_sendsigs_omissions.patch: refreshed.
 -- Mathieu Trudel-Lapierre <email address hidden> Mon, 30 Jul 2012 09:13:32 -0400

Changed in network-manager (Ubuntu Quantal):
status: In Progress → Fix Released
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Please use ppa:mathieu-tl/nm for initial testing of the same patch in Precise.

Revision history for this message
Attila Szalay (sasa-u) wrote :

I installed the new network manager and dnsmasq-base and will test it.

Revision history for this message
Steve Atwell (satwell) wrote :

I've installed network-manager 0.9.4.0-0ubuntu4.2~mtrudel6 and dnsmasq-base 2.59-4ubuntu0.1~mtrudel2 on Precise.

So far things look good. With the normal Precise versions, I would see dnsmasq restart 30-40 times(!) during boot, according to daemon.log. No restarts during boot with the new versions. I'll let you know if I see any DNS failures during normal use, but the fix looks promising so far.

Revision history for this message
Steve Atwell (satwell) wrote :

Been using network-manager 0.9.4.0-0ubuntu4.2~mtrudel6 for a few days now and I haven't noticed any resolver blips. The fix looks good to me.

Changed in network-manager (Ubuntu Precise):
milestone: ubuntu-12.04.1 → ubuntu-12.04.2
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Awesome. Thanks Steve, I'll start preparing this for Precise.

The issue with host routes will be covered in bug 1038541.

Changed in network-manager (Ubuntu Precise):
status: Triaged → In Progress
Revision history for this message
Adam Stokes (adam-stokes) wrote :

Mathieu,

Just pinging to make sure precise was on your list to get pushed, I can go through and do the debdiff if your workload is above normal and need some assistance.

Thanks
Adam

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

This bug basically requires adapting the dbus changes (and possibly another patch) to dnsmasq. Enabling dbus itself isn't huge, but if the binding code needs to be backported too, it's likely to be non-trivial.

As for NM, it's a matter of cherry-picking the patches, which should apply pretty cleanly on top of the package for precise.

It's still on my list; so let's aim at getting this checked off before the end of the week.

Changed in dnsmasq (Ubuntu):
status: New → Fix Released
Changed in dnsmasq (Ubuntu Quantal):
status: New → Fix Released
Changed in dnsmasq (Ubuntu Precise):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Mathieu Trudel-Lapierre (mathieu-tl)
Revision history for this message
Josh Hill (ingenium) wrote :

I'm still having the issue with 0.9.4.0-0ubuntu4.2~mtrudel7 in precise. I've also checked the box to have network manager ignore automatically obtained routes but it doesn't help. I even tried using the network-manager package in precise-proposed (0.9.4.0-0ubuntu4.2), and it also still has the issue.

It happens pretty regularly with 0.9.4.0-0ubuntu4.2~mtrudel7. It happens exactly every 20 or 30 minutes (sometimes it's exactly 20 minutes, other times it's exactly 30 minutes). This could just be related to when the IPv6 announce timing is though.

Revision history for this message
Mark Russell (marrusl) wrote :

I think this should be marked "Fixed Released" for Precise now.

Revision history for this message
Erno Kuusela (erno-iki) wrote :

@marrusl: fix doesn't seem to be in precise or even precise-updates yet, the version mentioned byt @ingenium is probably from ppa:mathieu-tl/nm.

Colin Watson (cjwatson)
Changed in network-manager (Ubuntu Precise):
milestone: ubuntu-12.04.2 → ubuntu-12.04.3
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

No, not Fix Released for Precise just yet:

This change requires the an update to dnsmasq currently in precise-proposed (https://launchpad.net/ubuntu/+source/dnsmasq/2.59-4ubuntu0.1) which needs to be verified by someone having the issue, to go along with the network-manager - 0.9.4.0-0ubuntu4.3~mtrudel1 package in my ppa (ppa:mathieu-tl/nm).

Once that's done we can push the NM changes to proposed for verification.

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

dnsmasq is now in updates, and there is no SRU blocking network-manager -- I'll upload the fix for this in -proposed today.

Changed in dnsmasq (Ubuntu Precise):
status: In Progress → Fix Released
description: updated
Revision history for this message
Steve Langasek (vorlon) wrote :

As I understand it, the NM SRU to precise updates the package to unconditionally use the dbus interface for updates. This means that the package needs a versioned dependency on the SRU version of dnsmasq-base that *enables* the dbus interface; otherwise this can break for users if they happen to install the NM SRU without the dnmasq SRU.

I'm rejecting the package from the precise-proposed queue. Mathieu, can you please reupload with a fix for the dnsmasq versioned dependency?

description: updated
Revision history for this message
Adam Stokes (adam-stokes) wrote :

Friendly ping to see what the status is on this issue wrt comment #23

Thanks!
Adam

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

I uploaded a new package to the precise queue last week.

Revision history for this message
Adam Stokes (adam-stokes) wrote :

Thanks Mathieu for the upload!

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Steve, or anyone else affected,

Accepted network-manager into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/network-manager/0.9.4.0-0ubuntu4.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in network-manager (Ubuntu Precise):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Alec Warner (antarus) wrote :

I am the replacement for Steve Atwell (who no longer does Goobuntu stuff.) This package fixes the issue for me.

-A

Steve Langasek (vorlon)
tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package network-manager - 0.9.4.0-0ubuntu4.3

---------------
network-manager (0.9.4.0-0ubuntu4.3) precise-proposed; urgency=low

  * debian/patches/dnsmasq-dbus-updates.patch: do DNS updates to dnsmasq via
    DBUS rather than restarting it all the time. (LP: #1004775)
  * debian/network-manager.dirs: add /etc/NetworkManager/dnsmasq.d as a path
    to install.
  * debian/control: network-manager depends on at least dnsmasq-base
    2.59-4ubuntu0.1, which is the lowest version that provides dbus.
 -- Mathieu Trudel-Lapierre <email address hidden> Thu, 23 May 2013 11:50:20 -0400

Changed in network-manager (Ubuntu Precise):
status: Fix Committed → Fix Released
Revision history for this message
Colin Watson (cjwatson) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Howard Chu (hyc) wrote :

I just now discovered this was finally fixed. It only took 5 years for someone to reinvent my patch... https://mail.gnome.org/archives/networkmanager-list/2008-September/msg00042.html

Hopefully upstream will take this soon. Thanks for your work integrating this much-needed feature.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.