Apparmor uses rsyslogd profile for different processes - utopic HWE

[Pavel Malyshev]

[apparmor impact]

This bug generates false positives when using the apparmor regression
tests on the HWE kernels (utopic and newer), which means the kernel team
needs to examine test output to ensure that addiitional failures didn't
occur when testing new kernels.

[apparmor test case]

1) install hwe kernel libapparmor-dev libdbus-1-dev attr
2) apt-get source apparmor
3) cd apparmor-2.8.95~2430/tests/regression/apparmor/
4) make USE_SYSTEM=1
5) sudo bash unix_socket_file.sh

If the bug has not been addressed, this test script will fail with the
following messages:

Error: unix_socket_file failed. Test 'socket file (dgram); confined server / access (w)' was expected to 'pass'. Reason for failure 'FAIL CLIENT - connect: Permission denied
FAIL - poll timed out'
Error: unix_socket_file failed. Test 'socket file (dgram); confined client w/ access (rw)' was expected to 'pass'. Reason for failure 'FAIL CLIENT - connect: Permission denied
FAIL - poll timed out'

and a return code of 2 (echo $?). If it has been fixed it should return
silently, with a return code of 0.

[apparmor regression potential]

The patch for this bug only affects the test suite for apparmor, which
is a loosening of the policy used in the specific failing testcases.
There should be no effect on the apparmor implementation proper from
this fix.

[apparmor additional info]

This testsuite is run as part of the test-apparmor.py test script
from lp:qa-regression-testing, and used as part of the kernel update
process, but is useful for ensuring that apparmor is functioning
properly.

[Original description]
I've noticed that apparmor loads /usr/sbin/rsyslogd profile for completely unrelated processes:

Feb 25 08:36:19 emma kernel: [ 134.796218] audit: type=1400 audit(1424842579.429:245): apparmor="DENIED" operation="sendmsg" profile="/usr/sbin/rsyslogd" name="/dev/log" pid=4002 comm="sshd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Feb 25 08:36:23 emma kernel: [ 139.330989] audit: type=1400 audit(1424842583.965:246): apparmor="DENIED" operation="sendmsg" profile="/usr/sbin/rsyslogd" name="/dev/log" pid=4080 comm="sudo" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Feb 25 08:35:42 emma kernel: [ 97.912402] audit: type=1400 audit(1424842542.565:241): apparmor="DENIED" operation="sendmsg" profile="/usr/sbin/rsyslogd" name="/dev/log" pid=2436 comm="whoopsie" requested_mask="r" denied_mask="r" fsuid=103 ouid=0
Feb 25 08:34:43 emma kernel: [ 38.867998] audit: type=1400 audit(1424842483.546:226): apparmor="DENIED" operation="sendmsg" profile="/usr/sbin/rsyslogd" name="/dev/log" pid=3762 comm="ntpd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

I'm not sure how apparmor decides which profile to use for which task, but is shouldn't load '/usr/sbin/rsyslogd' profile for sshd/ntpd/etc.

I'm running:
# lsb_release -rd
Description: Ubuntu 14.04.2 LTS
Release: 14.04

# dpkg -l | grep apparmor
ii apparmor 2.8.95~2430-0ubuntu5.1 amd64 User-space parser utility for AppArmor
ii apparmor-profiles 2.8.95~2430-0ubuntu5.1 all Profiles for AppArmor Security policies
ii apparmor-utils 2.8.95~2430-0ubuntu5.1 amd64 Utilities for controlling AppArmor
ii libapparmor-perl 2.8.95~2430-0ubuntu5.1 amd64 AppArmor library Perl bindings
ii libapparmor1:amd64 2.8.95~2430-0ubuntu5.1 amd64 changehat AppArmor library
ii python3-apparmor 2.8.95~2430-0ubuntu5.1 amd64 AppArmor Python3 utility library
ii python3-libapparmor 2.8.95~2430-0ubuntu5.1 amd64 AppArmor library Python3 bindings

# uname -a
Linux emma 3.16.0-31-generic #41~14.04.1-Ubuntu SMP Wed Feb 11 19:30:13 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[Pavel Malyshev (afunix)]

This errors started appearing in the log since Feb 21 when I've rebooted to complete Utopic HWE installation:
# dpkg -l | grep utopic
ii linux-generic-lts-utopic 3.16.0.31.24 amd64 Complete Generic Linux kernel and headers
ii linux-headers-generic-lts-utopic 3.16.0.31.24 amd64 Generic Linux kernel headers
ii linux-image-generic-lts-utopic 3.16.0.31.24 amd64 Generic Linux kernel image

[Pavel Malyshev (afunix)]

I can confirm that the audit message is gone from /var/log/syslog after I got back to the original 3.13.0-46 Ubuntu 14.04.2 kernel.

[Seth Arnold (seth-arnold)]

There's a chance this is the AppArmor "cross profile" IPC check; when one process performs an IPC operation with a second process, in different profiles, both profiles have to allow the operation.

If my guess is right, these ought to be silenced by changing:

  /dev/log wl,
to
  /dev/log rwl,

in the /etc/apparmor.d/usr.sbin.rsyslogd file and reloading the policy with apparmor_parser --replace /etc/apparmor.d/usr.sbin.rsyslogd

Thanks

[John Johansen (jjohansen)]

Seth is correct,

when a message is sent across a unix based file socket their is a cross check (one check for the sender, and another for the receiver). In this case the receiving end of the message (rsyslogd) does not have the correct permission (r perm).

The reason this is reported with the pid and comm of other processes is that the check (for both sender and receiver) is done in the kernel context of the sending process.

[Pavel Malyshev (afunix)]

First of all, Trusty kernel worked somehow differently. It didn't complain for /dev/log.

Yes, my /etc/apparmor.d/usr.sbin.rsyslogd does not have 'r' for /dev/log:
/dev/log wl,

So does it mean that the rsyslog profile has to be patched for trusty?
Or at least some kind of Utopic HWE release notes has to be updated?

I've updated /etc/apparmor.d/local/usr.sbin.rsyslogd and can confirm that message is gone with Utopic kernel.

[John Johansen (jjohansen)]

Are you using the apparmor user space from trusty or the utopic or backport apparmor userspace?

When using the trusty userspace the HWE kernel should behave like the trusty kernel, however when using a newer userspace with the HWE kernel policy will need to be updated.

Ubuntu have chosen to not update the policy and userspace for trusty even when the HWE kernel is used.

[Pavel Malyshev (afunix)]

John,

I'm using Ubuntu Trusty with Utopic HWE, so my apparmor is from Trusty.
I didn't install anything from backports or Utopic by hand to not mess package system and LTS state of distribution.
But somehow the system does not behave like trusty.

Note that I've updated this system from 12.04 w/ Trusty HWE to 14.04 with do-release-upgrade few months ago.

$ LANG=C apt-cache policy apparmor apparmor-profiles apparmor-utils
apparmor:
  Installed: 2.8.95~2430-0ubuntu5.1
  Candidate: 2.8.95~2430-0ubuntu5.1
  Version table:
 *** 2.8.95~2430-0ubuntu5.1 0
        500 http://ru.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.8.95~2430-0ubuntu5 0
        500 http://ru.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
apparmor-profiles:
  Installed: 2.8.95~2430-0ubuntu5.1
  Candidate: 2.8.95~2430-0ubuntu5.1
  Version table:
 *** 2.8.95~2430-0ubuntu5.1 0
        500 http://ru.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.8.95~2430-0ubuntu5 0
        500 http://ru.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
apparmor-utils:
  Installed: 2.8.95~2430-0ubuntu5.1
  Candidate: 2.8.95~2430-0ubuntu5.1
  Version table:
 *** 2.8.95~2430-0ubuntu5.1 0
        500 http://ru.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
        100 /var/lib/dpkg/status
     2.8.95~2430-0ubuntu5 0
        500 http://ru.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

[John Johansen (jjohansen)]

Pavel,

okay thanks, this does sound like a bug in the kernel. I will look into it more

[John Johansen (jjohansen)]

So after looking into this more it looks like this is actually a bug in the trusty kernel, that has been fixed on utopic.

Trusty needs a kernel patch and an update to its policy.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1425398

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Simon Déziel (sdeziel)]

Same problem here (Trusty+HWE kernel) and adding "/dev/log r," to /etc/apparmor.d/local/usr.sbin.rsyslogd does not help.

[Steve Beattie (sbeattie)]

The rsyslog apparmor policy is shipped in the rsyslog package, not apparmor. Opening a task against rsyslog, closing the apparmor userspace task.

[Steve Beattie (sbeattie)]

This is fixed in the rsyslog policy in vivid; closing there.

[Pavel Malyshev (afunix)]

As I said before, I don't use Vivid, I use Trusty which is stated to be supported until 2019!

The bug is against trusty, please don't close it as fixed in an non-LTS release

[Pavel Malyshev (afunix)]

Also if you read comments of other Ubuntu _developers_, they are rather confident that the bug is in Utopic HWE for Trusty

[Seth Arnold (seth-arnold)]

Pavel, note that Steve closed the rsyslog/vivid task because that has been fixed; the rsyslog/trusty task is still open, as are the kernel portions of the bug.

Thanks

[Steve Beattie (sbeattie)]

John, what changes are needed kernel side to address this issue?

Attached is a debdiff to address the rsyslog policy side of things.

[John Johansen (jjohansen)]

The issue here is that one end of the socket is an fs socket and the other end is anonymous. When the check is done from the socket at the anonymous end, the check is being dropped.

the patch is a backport of what is in utopic/vivid but is currently untested. I am building a test kernel

[Steve Beattie (sbeattie)]

It turns out that there is a small bit of the AppArmor userspace that needs to be addressed, the regression tests need to be slightly adjusted to take the permissions change into account.

[Marc Deslauriers (mdeslaur)]

ACK on the debdiff in comment #20. Uploaded for processing by the SRU team. Thanks!

[Chris J Arges (arges)]

Hello Pavel, or anyone else affected,

Accepted rsyslog into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/rsyslog/7.4.4-1ubuntu2.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Pavel Malyshev (afunix)]

It seems that issue is gone with this fix.

1. Installed the package from trusty-proposed
# LANG=C apt-cache policy rsyslog
rsyslog:
  Installed: 7.4.4-1ubuntu2.6
  Candidate: 7.4.4-1ubuntu2.6
  Version table:
 *** 7.4.4-1ubuntu2.6 0
        400 http://archive.ubuntu.com/ubuntu/ trusty-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     7.4.4-1ubuntu2.5 0
        500 http://ru.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
     7.4.4-1ubuntu2.3 0
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     7.4.4-1ubuntu2 0
        500 http://ru.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

2. commented out the workaround "/dev/log rwl," from /etc/apparmor.d/local/usr.sbin.rsyslogd

3. Rebooted to the latest Utopic HWE kernel
# uname -a
Linux emma 3.16.0-37-generic #51~14.04.1-Ubuntu SMP Wed May 6 15:23:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

4. The issue is gone:
# dmesg | grep DENIED
#

[Simon Déziel (sdeziel)]

Works here too, thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package rsyslog - 7.4.4-1ubuntu2.6

---------------
rsyslog (7.4.4-1ubuntu2.6) trusty-proposed; urgency=medium

  * debian/usr.sbin.rsyslog: grant read access to /dev/log to cope with
    behaviorial change of apparmor in utopic HWE kernel (LP: #1425398)

 -- Steve Beattie <email address hidden> Thu, 30 Apr 2015 12:20:51 -0700

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for rsyslog has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Steve Beattie (sbeattie)]

Here is the patch to address the apparmor userspace component of this bug as part of a trusty SRU. It's already been addressed in utopic and later.

[Steve Beattie (sbeattie)]

For the apparmor trusty SRU, I reproduced the failure with the source for apparmor 2.8.95~2430-0ubuntu5.1 from trusty-updates with a hardware enablement kernel and confirmed that apparmor 2.8.95~2430-0ubuntu5.2 addresses the issue. I also verified that the rest of the apparmor regression tests passed.

[Simon Déziel (sdeziel)]

Oddly enough, I'm still seeing some variation of this error:

May 21 12:27:28 xeon kernel: [95104.918686] audit: type=1400 audit(1432225648.230:57): apparmor="DENIED" operation="sendmsg" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/rsyslogd" name="dev/log" pid=3444 comm="logger" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

$ dpkg -l| grep rsyslog
ii rsyslog 7.4.4-1ubuntu2.6 amd64 reliable system and kernel logging daemon
$ uname -a
Linux xeon 3.16.0-38-generic #52~14.04.1-Ubuntu SMP Fri May 8 09:43:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[Jamie Strandboge (jdstrand)]

Simon, that is a different issue unrelated to this update. It is being tracked in bug #1373070.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apparmor - 2.8.95~2430-0ubuntu5.2

---------------
apparmor (2.8.95~2430-0ubuntu5.2) trusty-proposed; urgency=medium

  * debian/patches/php5-Zend_semaphore-lp1401084.patch: allow php5
    abstraction access to Zend opcache files (LP: #1401084)
  * debian/patches/dnsmasq-lxc_networking-lp1403468.patch: update
    profile for lxc support (LP: #1403468)
  * debian/patches/profiles-texlive_font_generation-lp1010909.patch:
    allow generation of texlive fonts by sanitized-helpers
    (LP: #1010909)
  * debian/apport/source_apparmor.py: fix the apparmor apport hook
    so it does not raise an exception if a non-unicode character is
    found in /var/log/kern.log or in /var/log/syslog. This should
    work under python3 or python2.7 (LP: #1304447)
  * debian/patches/profiles-dovecot-updates-lp1296667.patch: update
    dovecot profiles to address several missing permissions.
    (LP: #1296667)
  * debian/patches/profiles-adjust_X_for_lightdm-lp1339727.patch:
    adjust X abstraction for LightDM xauthority location (LP: #1339727)
  * debian/patches/libapparmor-fix_memory_leaks-lp1340927.patch; fix
    memory leaks in log parsing component of libapparmor (LP: #1340927)
  * debian/patches/libapparmor-another_audit_format-lp1399027.patch:
    add support for another log format style (LP: #1399027)
  * debian/patches/tests-workaround_for_unix_socket_change-lp1425398.patch:
    work around apparmor kernel behavioral change in regression tests
    (LP: #1425398)
  * debian/control: add breaks on python3-apparmor against older
    apparmor-utils that used to be where python bits lived
    (LP: #1373259)
  * debian/patches/utils-update_to_2.9.2.patch: update the python
    utilities to the upstream 2.9.2 (LP: #1449769, incorporating a
    large number of fixes and improvements, including:
    - fix aa-genprof traceback with apparmor 2.8.95 (LP: #1294797)
    - fix aa-genprof crashing when selecting scan on Ubuntu 14.04 server
      (LP: #1319829)
    - make aa-logprof read profile instead of program binary
      (LP: #1317176, LP: #1324154)
    - aa-complain: don't traceback when marking multiple profiles
      (LP: #1378095)
    - make python tools able to parse mounts with UTF-8 non-ascii
      characters (LP: #1310598)

 -- Steve Beattie <email address hidden> Thu, 30 Apr 2015 12:18:08 -0700