[3.11.0-12.18 regression] "Failed name lookup - disconnected path" in dhclient D-BUS access

Bug #1244157 reported by Martin Pitt on 2013-10-24
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
apparmor (Ubuntu)
Undecided
Unassigned
linux (Ubuntu)
Low
John Johansen
network-manager (Ubuntu)
Undecided
Martin Pitt

Bug Description

On October 9 the NetworkManager tests started failing (https://jenkins.qa.ubuntu.com/view/Saucy/view/AutoPkgTest/job/saucy-adt-network-manager/?). Unfortunately the more recent saucy logs got lost, but the trusty ones have the information, like in

  https://jenkins.qa.ubuntu.com/view/Trusty/view/AutoPkgTest/job/trusty-adt-network-manager/1/ARCH=i386,label=adt/

In these, dhclient that gets called through NetworkManager and /usr/lib/NetworkManager/nm-dhcp-client.action cannot access the system D-BUS any more:

----------- NM log ------------
NetworkManager[24451]: <info> Activation (eth42) Stage 3 of 5 (IP Configure Start) complete.
Internet Systems Consortium DHCP Client 4.2.4
Copyright 2004-2012 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Error: could not get the system bus. Make sure the message bus daemon is running! Message: (org.freedesktop.D
Bus.Error.AccessDenied) Failed to connect to socket /var/run/dbus/system_bus_socket: Permission denied
----------- NM log ------------

In syslog, you see this at that time:

------------ syslog ---------
Oct 21 14:11:24 autopkgtest kernel: [ 288.320754] type=1400 audit(1382364684.505:21): apparmor="DENIED" operation="connect" info="Failed name lookup - disconnected path" error=-13 parent=18759 profile="/usr/lib/NetworkManager/nm-dhcp-client.action" name="run/dbus/system_bus_socket" pid=18760 comm="nm-dhcp-client." requested_mask="rw" denied_mask="rw" fsuid=0 ouid=0
Oct 21 14:11:24 autopkgtest kernel: [ 288.333814] type=1400 audit(1382364684.517:22): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 parent=18752 profile="/sbin/dhclient" name="dev/log" pid=18759 comm="dhclient" requested_mask="w" denied_mask="w" fsuid=0 ouid=0
------------ syslog ---------

This gets fixed if I do "sudo /etc/init.d/apparmor teardown". But it does not seem to be a problem with the policy itself; if I do "sudo aa-complain dhclient" then dmesg changes to

[ 8054.314704] type=1400 audit(1382609088.727:672): apparmor="ALLOWED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 parent=24451 profile="/sbin/dhclient" name="dev/log" pid=24736 comm="dhclient" requested_mask="w" denied_mask="w" fsuid=0 ouid=0
[ 8054.341409] type=1400 audit(1382609088.755:673): apparmor="ALLOWED" operation="connect" info="Failed name lookup - disconnected path" error=-13 parent=24736 profile="/usr/lib/NetworkManager/nm-dhcp-client.action" name="run/dbus/system_bus_socket" pid=24737 comm="nm-dhcp-client." requested_mask="rw" denied_mask="rw" fsuid=0 ouid=0

So this doesn't look like a problem with the policy but rather with some internal AppArmor name parsing?

I found an old bug 955892 with the same error message, but that got fixed a while ago, and this does not involve ecryptfs or anythign similar. It's just a plain trusty VM with the NetworkManager autopkgtest.

Martin Pitt (pitti) on 2013-10-24
summary: - "Failed name lookup - disconne cted path" in dhclient D-BUS access
+ "Failed name lookup - disconnected path" in dhclient D-BUS access
tags: added: autopkgtest
Martin Pitt (pitti) wrote :

Reproducer:
- Start a minimal saucy VM (run-adt-test -sUl -r saucy), or boot a live system in KVM, etc.
- Install dependencies: sudo apt-get install --no-install-recommends python3 gir1.2-networkmanager-1.0 network-manager hostapd iw python3-dbusmock
- Get the tests: apt-get source network-manager; cd network-manager-*
- Run a single test: sudo debian/tests/nm ColdplugEthernet.test_auto_ip4

This will fail with log output, and reproduce above error.

Martin Pitt (pitti) wrote :

This also reproduces on a current trusty installation, with the same reproducer (except for the "start VM" part, of course). This will kick you off your "real" network though.

Martin Pitt (pitti) wrote :

If I install and boot https://launchpad.net/ubuntu/+source/linux/3.11.0-11.17 on a current trusty system this bug goes away. So https://launchpad.net/ubuntu/+source/linux/3.11.0-12.18 was the one that caused the regression.

Martin Pitt (pitti) on 2013-10-24
tags: added: rls-t-incoming

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1244157

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete

More kernel logs aren't relevant here, and I already stated the exact kernel version which introduced this regression.

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
summary: - "Failed name lookup - disconnected path" in dhclient D-BUS access
+ [3.11.0-12.18 regression] "Failed name lookup - disconnected path" in
+ dhclient D-BUS access
Martin Pitt (pitti) wrote :

From discussion with Jamie: Changing /etc/apparmor.d/sbin.dhclient to

   /usr/lib/NetworkManager/nm-dhcp-client.action flags=(attach_disconnected) {

does the trick. Apparently this happens because the NM tests unshare the file system namespace.

They do this because they mount tmpfses over /run/NetworkManager, /etc/NetworkManager and similar, so that they don't destroy the real files on the production system.

Martin Pitt (pitti) wrote :

Is there any way to temporarily add this attach_disconnected flag without having to touch files in /etc? I wouldn't want to actually sed -i there as these tests are supposed to be running on a desktop installation too (that's why I go through all the fuss in the first place).

Jamie Strandboge (jdstrand) wrote :

For adding the attach_disconnected flag, unfortunately there isn't a convenient way to apply attach_disconnected without modifying the profile directly.

John Johansen (jjohansen) wrote :

So yes this is because of the unshare of the file system namespace. Currently the only work around is the use of the attach_disconnected flag. Alternate solutions are coming as part of the work to support lxc

Martin:
The only way to temporarily add the attach_disconnected flag is to manually replace the profile with a version that has the flag added. The manually loaded profile can be from anywhere
  apparmor_parser -Kr <profile file>

Martin Pitt (pitti) wrote :

@John: Thanks a lot for the tip with "apparmor_parser -Kr <profile file>", that works perfectly!

I'll add that to the NM tests. If this is supposed to work that way, please feel free to close the AppArmor task. If it's just a workaround and D-BUS access is supposed to continue to work from an unshared file system, it can be kept open.

Changed in network-manager (Ubuntu):
status: New → In Progress
assignee: nobody → Martin Pitt (pitti)
Martin Pitt (pitti) on 2013-10-25
Changed in network-manager (Ubuntu):
status: In Progress → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package network-manager - 0.9.8.4-0ubuntu3

---------------
network-manager (0.9.8.4-0ubuntu3) trusty; urgency=low

  * Set "allow-stderr" for debian/tests/apparmor_workaround to avoid failure
    on AppArmor warnings.
 -- Martin Pitt <email address hidden> Tue, 05 Nov 2013 09:48:28 +0100

Changed in network-manager (Ubuntu):
status: Fix Committed → Fix Released
Shuduo Sang (sangshuduo) wrote :

I meet this problem on saucy too. So will the patch be backported to saucy soon? Thanks.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apparmor (Ubuntu):
status: New → Confirmed
tags: added: regression-release

Martin Pitt, could you please provide the linux commit that this regressed in as per https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1244157/comments/4 via https://wiki.ubuntu.com/Kernel/KernelBisection ?

tags: added: bot-stop-nagging
Martin Pitt (pitti) wrote :

Christopher, that's probably something that the AppArmor devs (John/Tylor) should look into. When I discussed this with John and Tylor, they already seemed to know which area of AppArmor hits this.

tags: removed: rls-t-incoming
Changed in apparmor (Ubuntu):
status: Confirmed → In Progress
status: In Progress → Invalid
Changed in linux (Ubuntu):
status: Confirmed → In Progress
assignee: nobody → John Johansen (jjohansen)
importance: Undecided → Low
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers