apparmor="DENIED" operation="create" profile="/usr/sbin/cups-browsed" w/ 4.14-rc2 and later

[Paul Menzel]

With Ubuntu 16.04.3 LTS (Xenial Xerus), and apparmor 2.10.95-0ubuntu2.7, in the system log each second the error message below is printed to.

```
[…]
[Mi Okt 4 16:57:52 2017] audit: type=1400 audit(1507129072.882:554): apparmor="DENIED" operation="create" profile="/usr/sbin/cups-browsed" pid=939 comm="cups-browsed" family="unix" sock_type="stream" protocol=0 requested_mask="create" denied_mask="create"
[Mi Okt 4 16:57:53 2017] audit: type=1400 audit(1507129073.886:555): apparmor="DENIED" operation="create" profile="/usr/sbin/cups-browsed" pid=939 comm="cups-browsed" family="unix" sock_type="stream" protocol=0 requested_mask="create" denied_mask="create"
[Mi Okt 4 16:57:54 2017] audit: type=1400 audit(1507129074.886:556): apparmor="DENIED" operation="create" profile="/usr/sbin/cups-browsed" pid=939 comm="cups-browsed" family="unix" sock_type="stream" protocol=0 requested_mask="create" denied_mask="create"
[Mi Okt 4 16:57:55 2017] audit: type=1400 audit(1507129075.886:557): apparmor="DENIED" operation="create" profile="/usr/sbin/cups-browsed" pid=939 comm="cups-browsed" family="unix" sock_type="stream" protocol=0 requested_mask="create" denied_mask="create"
[…]
```

[Doug Smythies (dsmythies)]

Which kernel are you using?
On my development 17.10 Desktop, I get the same as you but only for mainline kernels 4.14-rc2 and 4.14-rc3. Earlier kernels, including mainline 4.14-rc1, seem to be fine with respect to this issue.

[Paul Menzel (paulmenzel)]

Dear Doug,

Thank you for your reply.

On 10/06/17 21:16, Doug Smythies wrote:
> Which kernel are you using?

I am using Linux 4.14-rc3+.

> On my development 17.10 Desktop, I get the same as you but only for mainline
> kernels 4.14-rc2 and 4.14-rc3. Earlier kernels, including mainline 4.14-rc1 > seem to be fine with respect to this issue.

I don’t know, when it started. The problem wasn’t there with Linux 4.13.
So, your observation seems plausible.

Kind regards,

Paul

[Doug Smythies (dsmythies)]

Paul wrote:

> I am using Linux 4.14-rc3+.

O.K. that is/was really important information.

While I have been calling this a kernel regression, it might be that a great number of apparmor profiles need to be updated to accommodate the new security stuff that was introduced in kernel 4.14-rc2 (it missed -rc1 due to some issues, and it is not normal to include such major changes in anything other than -rc1). i.e. It might be that this is a non-backwards compatible kernel change.

Anyway, I know nothing about this area of the kernel, and will step back now.

To be able to proceed with my own work, I have compiled kernel 4.14-rc3 (and now 4.14-rc4) with apparmor disabled.

[Mario Limonciello (superm1)]

I've found that it's more than just cups blows up, some networking related items (DHCP client, network manager IIRC) also explode.

[Mario Limonciello (superm1)]

I've personally confirmed this with both artful and xenial userspace with 4.14-rc4.
A temporary solution other than compiling without apparmor is to do teardown/stop
# /etc/init.d/apparmor teardown
# /etc/init.d/apparmor stop

[Doug Smythies (dsmythies)]

> I've found that it's more than just cups blows up, some networking
> related items (DHCP client, network manager IIRC) also explode.

yes, and libvirtd and mysql.

I was not aware of "teardown". I'll try it when I get a chance.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jamie Strandboge (jdstrand)]

This isn't really an *Ubuntu* issue per se as we've never claimed to support apparmor profiles with non-Ubuntu kernels. I do think it is interesting that there are 'unix' denials on a kernel that isn't supposed to support unix rules.

John, can you comment on this?

[Mario Limonciello (superm1)]

>This isn't really an *Ubuntu* issue per se as we've never claimed to support apparmor profiles with non-Ubuntu kernels.

So I think the problem is that kernel team maintains a PPA of mainline kernels and often will ask users to check stuff with mainline kernel when there are bugs that come up. This will make that more difficult if it's not fixed.

[Mario Limonciello (superm1)]

And FWIW the /sbin/dhclient and /usr/lib/NetworkManager/nm-dhcp-helper errors are also family="unix" denying create operations.

[John Johansen (jjohansen)]

As of 4.13 the upstream kernel does support basic socket mediation which does include unix sockets. This denial is not due to fine grained unix socket mediation.

[John Johansen (jjohansen)]

@Doug,

not a kernel regression and not an incompatible kernel change either. The kernel does support the older abi, however the compiled policy being sent to the kernel is for the new abi that the kernel is now advertising as being supported.

The kernel advertises its supported feature set and abis through the /sys/kernel/security/apparmor/features directory.

The userspace side of things can choose to take advantage of the current kernel feature set/abi or to pin its supported feature set by setting the features file. This is not being done on ubuntu so the newest version of kernel features is always being supported, generally the userspace has been ahead of kernel features so it is more than willing to compile for them.

What is odd, is that Ubuntu carries profiles with fine grained unix socket rules and these should be downgraded to basic the basic socket rules that the 4.13 kernel supports.

[John Johansen (jjohansen)]

err make that 4.14 not 4.13 in my above explanation

[John Johansen (jjohansen)]

@Doug,

I forgot to mention this in my above explanation the reason you see this with 4.14-rc2 and not 4.14-rc1 is because there was a problem with the security tree merge and Linus ended up pulling the security changes in between rc1 and rc2.

[John Johansen (jjohansen)]

Could someone who is having this issue also attach a profile cache file for the profile that is failing? So I can verify what your local compiles are doing.

you can grab the binary cache file out of
  /etc/apparmor.d/cache/sbin.dhclient

or compile it with
  apparmor_parser -o output_file /etc/apparmor.d/sbin.dhclient

[Mario Limonciello (superm1)]

Here you go. This is from a kernel built on 4.14-rc4 right after boot where dhclient is failing.

[John Johansen (jjohansen)]

Ubuntu's parser is missing upstream commit r3700, resulting in this failure.

[Doug Smythies (dsmythies)]

John wrote:
> Ubuntu's parser is missing upstream commit r3700, resulting in this failure.

Is there any boot option that would allow Ubuntu mainline kernels to work?
For my own work, and as mentioned in comment #3, I am compiling with "# CONFIG_SECURITY_APPARMOR is not set".

[John Johansen (jjohansen)]

This bug is annoying in that there isn't a single switch to toggle to work around it. You can pin the feature file but getting the feature file you want requires some editing, or booting into a 4.13 upstream kernel (at which point you loose the other features landed in 4.14).

To pin the features file you will want to edit /etc/apparmor/parser.conf and add

#Pin the used features to
features-file=/etc/apparmor/features

To obtain the features file you can reboot into an upstream 4.13 kernel copy the features file from the cache (remember this will result in loss of other features landed in 4.14)
  cp /etc/apparmor.d/cache/.features /etc/apparmor/features

Or you use the hand edited features 4.14 feature file attached.

Remember that once this feature file is set you won't be picking up new features so ideally you will need to remove the feature file pinning at some point in the future.

[Jamie Strandboge (jdstrand)]

John,

It sounds like we should backport r3700 to all Ubuntu releases?

[John Johansen (jjohansen)]

Yes. Ideally we would grab the upstream maintenance releases with the patches in them. But upstream hasn't had time to release them yet. It should happen this week

[Doug Smythies (dsmythies)]

@John:

I tried your suggestion on my main 16.04.3 test server. I edited /etc/apparmor/parser.conf, keeping an "original copy" first.
And used "the hand edited features 4.14 feature file attached".

It made things worse, as in addition to mysql and libvirt not starting, now the network doesn't start either.

I restored /etc/apparmor/parser.conf and changed my GRUB_CMDLINE_LINUX_DEFAULT in grub, adding "apparmor=0". and now everything is fine using the Ubuntu mainline kernel 4.14-rc5.

[Rocko (rockorequin)]

If it helps anyone, I've got 4.14-rc5 and apparmor working. I've posted a patch at the duplicate bug https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1724450.

[John Johansen (jjohansen)]

Rocko: thanks for the patch, just so people know this is a work around patch which adjusts policy instead of fixing the bug in the parser.

[John Johansen (jjohansen)]

@Doug,

thanks for testing, I've managed to track down a bug in the kernel, I'll try to get a fix merged before 4.14 final,

also I have apparmor userspace fixes building in the apparmor ppa and will post those up for further test once they are done

[John Johansen (jjohansen)]

Alright userspace packages with the parser fix are available in
https://launchpad.net/~apparmor-dev/+archive/ubuntu/apparmor-devel

zesty is still building.

So to recap which solutions are needed where.
ubuntu kernel + apparmor 2.11.X - no patches needed

upstream 4.14-rc6 or earlier - policy patch from #23 OR 2.11.1 userspace OR earlier userspace with patch from #17 (in ppa)

And if you are using feature pinning a pre 4.14 upstream feature set in combination with a 4.14 kernel - you will need a (2.11.1 user space or 2.11.0 userspace + patch) and the kernel fix that is in testing and has not been attached yet

[Paul Menzel (paulmenzel)]

I’d really like to try the Linux kernel fix. Can a get it from somewhere?

[John Johansen (jjohansen)]

On 10/24/2017 02:32 AM, Paul Menzel wrote:
> I’d really like to try the Linux kernel fix. Can a get it from
> somewhere?
>
commit 8baea25455c08173713fdbceac99309192518ffb
Author: John Johansen <email address hidden>
Date: Mon Oct 23 08:51:24 2017 -0700

    apparmor: fix regression in network mediation when using feature pinning

    When the 4.14-rc6 and earlier kernels are used with an upstream 4.13
    or earlier pinned feature set, there is a regression in network
    mediation where policy is not being correctly enforced, because the
    compilation is completely dropping the af mediation table as expected
    by pre 4.14 kernels but the 4.14 kernel is not accounting for this.

    Resulting in network denials that can not be fixed by policy.

    Fixes: 651e28c5537a ("apparmor: add base infastructure for socket mediation")
    Signed-off-by: John Johansen <email address hidden>

diff --git a/security/apparmor/policy_unpack.c b/security/apparmor/policy_unpack.c
index 5a2aec358322..e348f8dec45d 100644
--- a/security/apparmor/policy_unpack.c
+++ b/security/apparmor/policy_unpack.c
@@ -755,6 +755,10 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name)
   }
   if (!unpack_nameX(e, AA_ARRAYEND, NULL))
    goto fail;
+ } else {
+ /* support policy pre AF socket mediation */
+ for (i = 0; i < AF_MAX; i++)
+ profile->net.allow[i] = 0xffff;
  }
  if (VERSION_LT(e->version, v7)) {
   /* pre v7 policy always allowed these */

[John Johansen (jjohansen)]

Several people have asked for the patch

[Paul Menzel (paulmenzel)]

Dear John,

On 10/24/17 12:55, John Johansen wrote:
> On 10/24/2017 02:32 AM, Paul Menzel wrote:
>> I’d really like to try the Linux kernel fix. Can a get it from
>> somewhere?
>>
> commit 8baea25455c08173713fdbceac99309192518ffb
> Author: John Johansen <email address hidden>
> Date: Mon Oct 23 08:51:24 2017 -0700
>
> apparmor: fix regression in network mediation when using feature pinning
>
> When the 4.14-rc6 and earlier kernels are used with an upstream 4.13
> or earlier pinned feature set, there is a regression in network
> mediation where policy is not being correctly enforced, because the
> compilation is completely dropping the af mediation table as expected
> by pre 4.14 kernels but the 4.14 kernel is not accounting for this.
>
> Resulting in network denials that can not be fixed by policy.
>
> Fixes: 651e28c5537a ("apparmor: add base infastructure for socket mediation")
> Signed-off-by: John Johansen <email address hidden>
>
> diff --git a/security/apparmor/policy_unpack.c b/security/apparmor/policy_unpack.c
> index 5a2aec358322..e348f8dec45d 100644
> --- a/security/apparmor/policy_unpack.c
> +++ b/security/apparmor/policy_unpack.c
> @@ -755,6 +755,10 @@ static struct aa_profile *unpack_profile(struct aa_ext *e, char **ns_name)
> }
> if (!unpack_nameX(e, AA_ARRAYEND, NULL))
> goto fail;
> + } else {
> + /* support policy pre AF socket mediation */
> + for (i = 0; i < AF_MAX; i++)
> + profile->net.allow[i] = 0xffff;
> }
> if (VERSION_LT(e->version, v7)) {
> /* pre v7 policy always allowed these */

Thank you. Can I pull it from a tree? Trying [1], I am asked for
credentials.

```
$ git remote add ubuntu
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source
$ git fetch ubuntu
Username for 'https://git.launchpad.net':
```

Kind regards,

Paul

[1]
https://code.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/saucy/+ref/mako

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Fix regression in network mediation" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Doug Smythies (dsmythies)]

@John: That patch works great, thanks.

On kernel 4.14-rc6 + patch, I re-did the stuff from my comment #22, which in turn was implementing one of the methods from your comment #19. This time it worked.

[Paul Menzel (paulmenzel)]

@John, thank youf or the patch, but maybe I misunderstood it. Applying that patch to Linus’ master branch, should fix the regression, right? No user space change needed, correct?

```
$ git log --oneline -2
4a4a4a7 apparmor: fix regression in network mediation when using feature pinning
6cff0a1 Merge tag 'platform-drivers-x86-v4.14-3' of git://git.infradead.org/linux-platform-drivers-x86
```

With a Linux kernel built from that, NetworkManager still does not work, and even brightness controls don’t work on a Dell XPS 13 9360.

```
$ journalctl -k
[…]
Okt 24 16:54:05 Ixpees kernel: ath10k_pci 0000:3a:00.0: board_file api 2 bmi_id N/A crc32 07ee144e
Okt 24 16:54:05 Ixpees kernel: audit: type=1400 audit(1508856845.907:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/lightdm/lightdm-guest-session" pid=828 comm="apparmor_parse
Okt 24 16:54:05 Ixpees kernel: audit: type=1400 audit(1508856845.907:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/lightdm/lightdm-guest-session//chromium" pid=828 comm="appa
Okt 24 16:54:05 Ixpees kernel: audit: type=1400 audit(1508856845.991:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/sbin/dhclient" pid=830 comm="apparmor_parser"
Okt 24 16:54:05 Ixpees kernel: audit: type=1400 audit(1508856845.991:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=830 comm="apparmo
Okt 24 16:54:05 Ixpees kernel: audit: type=1400 audit(1508856845.991:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=830 comm="apparmor_parse
Okt 24 16:54:05 Ixpees kernel: audit: type=1400 audit(1508856845.991:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=830 comm="apparmor_par
Okt 24 16:54:06 Ixpees kernel: audit: type=1400 audit(1508856846.055:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="webbrowser-app" pid=833 comm="apparmor_parser"
Okt 24 16:54:06 Ixpees kernel: audit: type=1400 audit(1508856846.055:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="webbrowser-app//oxide_helper" pid=833 comm="apparmor_parser"
Okt 24 16:54:06 Ixpees kernel: audit: type=1400 audit(1508856846.115:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/cups-browsed" pid=856 comm="apparmor_parser"
Okt 24 16:54:06 Ixpees kernel: audit: type=1400 audit(1508856846.155:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=852 comm="apparmor_parser"
Okt 24 16:54:06 Ixpees kernel: ath10k_pci 0000:3a:00.0: Unknown eventid: 90118
Okt 24 16:54:06 Ixpees kernel: ath10k_pci 0000:3a:00.0: htt-ver 3.32 wmi-op 4 htt-op 3 cal otp max-sta 32 raw 0 hwcrypto 1
[…]
Okt 24 16:54:06 Ixpees kernel: ath: EEPROM regdomain: 0x6c
Okt 24 16:54:06 Ixpees kernel: ath: EEPROM indicates we should expect a direct regpair map
Okt 24 16:54:06 Ixpees kernel: ath: Country alpha2 being used: 00
Okt 24 16:54:06 Ixpees kernel: ath: Regpair used: 0x6c
Okt 24 16:54:06 Ixpe...

[Christian Boltz (cboltz)]

> ... apparmor="DENIED" operation="create" ... family="unix" sock_type="stream"

With the pinned-down feature set, you probably "lost" support for unix rules.

In theory, apparmor_parser will downgrade those rules to "network unix," - but in practise a bug in apparmor_parser prevented it.This bug was fixed in the point releases some days ago.

Can you please test with the latest apparmor_parser? "Latest" means 2.11.1, 2.10.3 or 2.9.5 - or, if you want to test only the bugfix, apply the patch from bzr trunk r3700 - http://bazaar.launchpad.net/~apparmor-dev/apparmor/master/revision/3700

[Paul Menzel (paulmenzel)]

Dear Christian,

Am 24.10.2017 um 19:14 schrieb Christian Boltz:
>> ... apparmor="DENIED" operation="create" ... family="unix"
> sock_type="stream"
>
> With the pinned-down feature set, you probably "lost" support for unix
> rules.

Sorry, I have no clue about the internals. I just use what’s shipped in
Ubuntu 16.04.

> In theory, apparmor_parser will downgrade those rules to "network unix,"
> - but in practise a bug in apparmor_parser prevented it. This bug was
> fixed in the point releases some days ago.

Just a note, that the no regression policy of Linux actually demands
that the latest Linux kernel also works with buggy user space software.

> Can you please test with the latest apparmor_parser? "Latest" means
> 2.11.1, 2.10.3 or 2.9.5 - or, if you want to test only the bugfix, apply
> the patch from bzr trunk r3700 - http://bazaar.launchpad.net/~apparmor-
> dev/apparmor/master/revision/3700

The system is an up-to-date Ubuntu 16.04 installation. So that should be
already installed? I can check tomorrow.

Kind regards,

Paul

[John Johansen (jjohansen)]

@Paul,

sorry no. At least not unless you are doing some very specific pinning of the kernel features abi as I suggested as a solution in #19.

You will need the userspace fix in the ppa until ubuntu can land an SRU of either patch r3700 or a full SRU of the current maintenance releases. With the userspace fix ubuntu policy should just work under the 4.14-rcX kernels

[Paul Menzel (paulmenzel)]

I integrated the PPA, but under Ubuntu 16.04.3 LTS no updates are available. The package *apparmor* 2.10.95-0ubuntu2.7 is installed.

```
$ sudo add-apt-repository ppa:apparmor-dev/apparmor-devel
$ sudo apt-get update
```

[Doug Smythies (dsmythies)]

Further to my comment #32: That setup then breaks lots of stuff if I subsequently boot a normal default kernel (i.e. 4.4.0-96-generic). I'm going back to just booting with apparmor disabled.

[John Johansen (jjohansen)]

@Doug,

can you attach your breakage?

[Doug Smythies (dsmythies)]

@John: O.K., I think this excerpt from kern.log is what you might be looking for.

[John Johansen (jjohansen)]

Okay thankyou everyone for your feedback.

The kernel patch causing the issue has been reverted. So 4.14-rc7 should work as pre 4.14-rc2

This bug has become a dumping ground for multiple issues so I am going to create new bugs to track the issues individually and close this bug down. Please see the following bugs if you have further issues.

the userspace policy compiler rule downgrades is bug 1728120
the network rule regression for policy that doesn't support network rules is bug 1728123
the need for better policy versioning so that a developer mode compile doesn't happen unless explicit requested is bug 1728130

[intrigeri (intrigeri)]

> The kernel patch causing the issue has been reverted. So 4.14-rc7 should work as pre 4.14-rc2

Great! (Modulo Linus' commit message…)

[John Johansen (jjohansen)]

Yes, that stings but wasn't unexpected. It will take awhile to get features going back up stream but in the long term this will actually benefit apparmor, as it is forcing the development of fine grained policy version which has been needed for year but never a top priority.