full fix for disconnected path (paths)

[Jamie Strandboge]

With the apparmor 3 RC1 upload, there is an incomplete bug fix for disconnected paths. This bug is to track that work.

This denial may be related:
Sep 23 10:10:50 localhost kernel: [40262.517799] audit: type=1400 audit(1411485050.722:2862): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/rsyslogd" name="dev/log" pid=7011 comm="logger" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

This is related to bug 1375410

[Jamie Strandboge (jdstrand)]

Here is another:
Sep 10 09:06:00 callisto kernel: audit: type=1400 audit(1410332760.203:112): apparmor="DENIED" operation="connect" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/cupsd" name="run/dbus/system_bus_socket" pid=3608 comm="cupsd" requested_mask="rw" denied_mask="rw" fsuid=0 ouid=0

[Jamie Strandboge (jdstrand)]

I'm going to need to add attach_disconnected to the cups profile as a temporary workaround. When this bug is fixed, we need to undo that.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cups - 1.7.5-3ubuntu1

---------------
cups (1.7.5-3ubuntu1) utopic; urgency=medium

  * debian/local/apparmor-profile:
    - fix peer on signal rule to use /usr/sbin/cupsd//third_party
      (LP: #1376611)
    - temporarily use attach_disconnected to work around LP: #1373070. This
      should be undone once 1373070 is properly fixed
 -- Jamie Strandboge <email address hidden> Thu, 02 Oct 2014 08:22:36 -0500

[Simon Déziel (sdeziel)]

To add one more data point, my Trusty server using the Utopic HWE kernel also exhibits the problem:

May 21 12:27:28 xeon kernel: [95104.918686] audit: type=1400 audit(1432225648.230:57): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/rsyslogd" name="dev/log" pid=3444 comm="logger" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

$ apt-cache policy apparmor linux-image-3.16.0-38-generic rsyslog
apparmor:
  Installed: 2.8.95~2430-0ubuntu5.2
  Candidate: 2.8.95~2430-0ubuntu5.2
  Version table:
 *** 2.8.95~2430-0ubuntu5.2 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2.8.95~2430-0ubuntu5.1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     2.8.95~2430-0ubuntu5 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
linux-image-3.16.0-38-generic:
  Installed: 3.16.0-38.52~14.04.1
  Candidate: 3.16.0-38.52~14.04.1
  Version table:
 *** 3.16.0-38.52~14.04.1 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
rsyslog:
  Installed: 7.4.4-1ubuntu2.6
  Candidate: 7.4.4-1ubuntu2.6
  Version table:
 *** 7.4.4-1ubuntu2.6 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     7.4.4-1ubuntu2.3 0
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     7.4.4-1ubuntu2 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

[Pavel Malyshev (afunix)]

I'm affected by this bug too at Trusty + Vivid HWE

# lsb_release -rd
Description: Ubuntu 14.04.3 LTS
Release: 14.04

# uname -a
Linux amanda 3.19.0-42-generic #48~14.04.1-Ubuntu SMP Fri Dec 18 10:25:23 UTC 2015 i686 i686 i686 GNU/Linux

# dpkg -l | grep linux-image-generic
ii linux-image-generic 3.13.0.74.80 i386 Generic Linux kernel image
ii linux-image-generic-lts-vivid 3.19.0.42.27 i386 Generic Linux kernel image

# dpkg -l | grep -e rsyslog -e apparmor
ii apparmor 2.8.95~2430-0ubuntu5.3 i386 User-space parser utility for AppArmor
ii apparmor-profiles 2.8.95~2430-0ubuntu5.3 all Profiles for AppArmor Security policies
...
ii rsyslog 7.4.4-1ubuntu2.6 i386 reliable system and kernel logging daemon

# grep 'audit:' /var/log/syslog | grep DENIED
Dec 26 09:39:48 amanda kernel: [11627.614510] audit: type=1400 audit(1451111988.687:54): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/rsyslogd" name="dev/log" pid=20376 comm="logger" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

[Rafael David Tinoco (rafaeldtinoco)]

Pavel, Déziel,

Im reproducing the same issue with dnsmasq + openstack + neutron:

Feb 16 18:35:01 juju-inaddy-machine-12 kernel: [ 4357.680900] audit: type=1400 audit(1455647701.796:121): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/dnsmasq" name="dev/log" pid=15176 comm="dnsmasq" requested_mask="w" denied_mask="w" fsuid=65534 ouid=0

AND when using :

/usr/sbin/dnsmasq flags=(attach_disconnected) {

in usr.sbin.dnsmasq profile, I'm mitigating the problem (just as the cups patch).

I'll try reproducing using rsyslog so I can have a simple reproducer in order to bisect kernel 3.13 -> 3.19 and check what caused apparmor's regression (likely related to apparmor's filesystem labeling mechanism).

Thank you

-inaddy

[Rafael David Tinoco (rafaeldtinoco)]

I am able to reproduce this just by having apparmor.d profile usr.sbin.rsyslogd removed from disable/ directory.

[ 674.165128] audit: type=1400 audit(1456491880.616:134): apparmor="DENIED" operation="sendmsg" profile="/usr/sbin/rsyslogd" name="/dev/log" pid=3639 comm="dhclient" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 674.165178] audit: type=1400 audit(1456491880.616:135): apparmor="DENIED" operation="sendmsg" profile="/usr/sbin/rsyslogd" name="/dev/log" pid=3639 comm="dhclient" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

OR

[ 522.429097] audit: type=1400 audit(1456491728.880:113): apparmor="DENIED" operation="sendmsg" profile="/usr/sbin/rsyslogd" name="/dev/log" pid=3184 comm="sshd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 527.268883] audit: type=1400 audit(1456491733.720:114): apparmor="DENIED" operation="sendmsg" profile="/usr/sbin/rsyslogd" name="/dev/log" pid=3239 comm="sshd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

[Christian Boltz (cboltz)]

As expected, that's a totally different issue.

Please add
    /dev/log r,
to your rsyslogd profile.

[Rafael David Tinoco (rafaeldtinoco)]

Yep, you're right. It was getting /dev/log from abstractions/base for write only. My bad.

Though,

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1373070/comments/6

Shows same issue.

[Rafael David Tinoco (rafaeldtinoco)]

Though,

For comments:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1373070/comments/7

If you remove /dev/log rwx from /etc/apparmor.d/usr.sbin.rsyslog :

Using kernel Ubuntu-3.13.x DOES NOT show any DENIALS (Ubuntu-3.16, Ubuntu-3.19 and Ubuntu-4.2 HWE kernels shows).

Using upstream kernels 3.13, 3.16, 3.19 and 4.2 DOES NOT show any DENIALS.

I wonder why only Ubuntu >= 3.16 kernels show the denials.

[Rafael David Tinoco (rafaeldtinoco)]

Okay, so, I had more time to dig a bit into this and, after some analysis, I got:

Errors being reproduced:

[1668392.078137] audit: type=1400 audit(1459311786.129:1375455): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/dnsmasq" name="dev/log" pid=15735 comm="dnsmasq" requested_mask="w" denied_mask="w" fsuid=0 ouid=0

And apparmor dnsmasq profile:

#/usr/sbin/dnsmasq flags=(attach_disconnected) {
#/usr/sbin/dnsmasq flags=(complain) {
/usr/sbin/dnsmasq {

Without any flags.

And the command causing the apparmor errors:

root 16877 0.0 0.2 66416 3648 ? S 13:23 0:00 sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ip netns exec qdhcp-37d013b6-f6fa-4652-8073-5e7d2c418a9d env NEUTRON_NETWORK_ID=37d013b6-f6fa-4652-8073-5e7d2c418a9d dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=ns-aa95fe20-ff --except-interface=lo --pid-file=/var/lib/neutron/dhcp/37d013b6-f6fa-4652-8073-5e7d2c418a9d/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/37d013b6-f6fa-4652-8073-5e7d2c418a9d/host --addn-hosts=/var/lib/neutron/dhcp/37d013b6-f6fa-4652-8073-5e7d2c418a9d/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/37d013b6-f6fa-4652-8073-5e7d2c418a9d/opts --dhcp-leasefile=/var/lib/neutron/dhcp/37d013b6-f6fa-4652-8073-5e7d2c418a9d/leases --dhcp-range=set:tag0,192.168.21.0,static,86400s --dhcp-lease-max=256 --conf-file=/etc/neutron/dnsmasq.conf --domain=openstacklocal

It is a "sudo-like" approach from openstack (rootwrap) to execute dnsmasq in a new network namespace with different privileges.

Ubuntu kernel 3.13.X has apparmor 3 alpha 6 code: https://pastebin.canonical.com/152812/
Ubuntu kernel 3.16 and 3.19 has apparmor 3 rc 1 code: https://pastebin.canonical.com/152813/

From apparmor I could see that the error comes from "aa_path_name" called by either:

- path_name *
- aa_remount
- aa_bind_mount
- aa_mount_change_type
- aa_move_mount
- aa_new_mount
- aa_unmount
- aa_pivotroot

So, since the job is being restarted by neutron (or at least it is trying to re-start it, causing the apparmor to block the access), I created a systemtap script to monitor path_name and check for dnsmasq trying to open "log" (allegedly /dev/log) file.

probe kernel.function("path_name").call {
 funcname = execname();
 if (funcname == "dnsmasq") {
  filename = reverse_path_walk($path->dentry);
  if (filename == "log") {
   printf("(%s) %s\n", execname(), filename);
   print_backtrace();
  }
 }
}

And got the backtrace from the denials:

(dnsmasq) log

 0xffffffff8132deb0 : path_name+0x0/0x140 [kernel]
 0xffffffff8132e413 : aa_path_perm+0xa3/0x130 [kernel]
 0xffffffff81337e26 : aa_unix_peer_perm+0x536/0x990 [kernel]
 0xffffffff8132c653 : apparmor_unix_may_send+0x73/0x150 [kernel]
 0xffffffff812eb8a6 : security_unix_may_send+0x16/0x20 [kernel]
 0xffffffff817019db : unix_dgram_connect+0x23b/0x250 [kernel]
 0xffffffff8164a987 : SYSC_connect+0xe7/0x120 [kernel]
 0xffffffff8164b68e : sys_connect+0xe/0x10 [kernel]
 0xffffffff817700cd : system_call_fastpath+0x1a/0x1f [kernel]

 When trying to check if "log" could be converted to "fullpath" by using systemtap function:

  return task_den...

[John Johansen (jjohansen)]

Correct.

There are actually several ways to get disconnected paths and this specific one is being caused by the new file ns. The proper fix for this is delegating access to the object that would not normally be accessible, however delegation is not available in the current releases of apparmor and the HACK of attach disconnected is being used to work around this.

As for apparmor not complaining about disconnected path failures, it should be unless attach disconnected is specified. The info field in the apparmor audit message will be
  info="Failed name lookup - disconnected path"

[Christian Ehrhardt  (paelzer)]

Hi,
I think bug 1594202 is another data point for this:

Jun 20 01:49:24 omicron kernel: [ 962.491873] audit: type=1400 audit(1466380164.941:90): apparmor="ALLOWED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/lib/dovecot/log" name="run/systemd/journal/dev-log" pid=2175 comm="log" requested_mask="w" denied_mask="w" fsuid=0 ouid=0

But before I close-as-dup and open a dovecot task here I'd ask if one that has worked on this issue take a look if that is true?

If so are we still supposed to add workarounds like the attach_disconnected or were there updates to this issue which didn't make it to the bug yet?

[Christian Ehrhardt  (paelzer)]

Actually the dovecot profiles are in apparmor and not dovecot source packages - so it would be an apparmor task then.

[John Johansen (jjohansen)]

possibly. There isn't actually enough information in that bug to be sure if it is an actual namespacing issue or it is a separate bug to do with unix domain sockets.

Unfortunately the workaround of attach_disconnect is still required to deal with these issues.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[LGB [Gábor Lénárt] (lgb)]

Same problem with powerdns, I can't run it with apparmor profile, because it complains:

operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/pdns_server" name="run/systemd/journal/dev-log" pid=17236 comm="pdns_server" requested_mask="w" denied_mask="w" fsuid=0 ouid=0

I am not an expert, but I tried to put run/systemd/journal/dev-log into the profile, but it is not accepted as it does not start with / ... But this is what kernel log suggest, so what can I do otherwise?

Note: I have: /usr/sbin/pdns_server flags=(complain,attach_disconnected)

But still ... (now I have only complain mode).

If I exclude pdns from systemd it works btw, and no wonder as it seems the problem somehow connected to systemd's journal, so it's better not to use systemd if possible since it renders apparmor unusable in my experience :( But for sure, I would be more than happy to have a better option, rather than deleting systemd's unit file each time after upgrade pdns ... Or so.

this is up-to-date Ubuntu 16.04.3 LTS 64 bit, fresh install, but I have about a dozen of servers with this problem with different daemons as well, not only powerdns.

[Seth Arnold (seth-arnold)]

Gábor, systemd is well-meaning in providing namespacing features so the thousands of daemons that are in the world don't have to re-implement something similar. But of course the kernel hook points used by AppArmor don't provide sufficient information to know what pathname to reconstruct when the named object isn't visible in the namespace where it was used.

Add /run/systemd/journal/dev-log w, to the profile, make sure attach_disconnected is used, and then you can return to using the systemd unit file. (Which is probably better than falling back to the sysv-init compatibility shims systemd uses.)

Thanks