/proc/sys/kernel/random/boot_id rule missing from abstractions/nameservice

Bug #1872564 reported by Simon Déziel on 2020-04-14
72
This bug affects 16 people
Affects Status Importance Assigned to Milestone
apparmor (Ubuntu)
Undecided
Sergio Durigan Junior
Focal
Undecided
Sergio Durigan Junior

Bug Description

[Impact]

On a default Focal install, systemd is used when looking up passwd and group information:

# grep systemd /etc/nsswitch.conf
passwd: files systemd
group: files systemd

Daemons confined by Apparmor that also query those "databases" will cause this Apparmor denial:

audit: type=1400 audit(1586825456.411:247): apparmor="DENIED" operation="open" namespace="root//lxd-fb1_<var-snap-lxd-common-lxd>" profile="/usr/sbin/named" name="/proc/sys/kernel/random/boot_id" pid=7370 comm="named" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000

Many daemons confined by Apparmor also happen to downgrade their privileges so they always end up looking up user/group information.

To fix this problem, we had to backport an upstream patch which adds new directives to the 'nameservices' apparmor profile.

[Test Case]

In order to reproduce the bug, one can:

1) launch a Focal container (named fb1 here)
$ lxc launch images:ubuntu/focal fb1

2) setup apparmor inside the container (already done on official Ubuntu images)
$ lxc exec fb1 -- apt update && lxc exec fb1 -- apt install apparmor -y

3) install bind9
$ lxc exec fb1 -- apt install bind9 -y

4) check kernel logs for DENIED
$ journalctl -o cat -b0 -k | grep 'apparmor="DENIED"' | grep -F 'profile="/usr/sbin/named"'

or, depending on how logging is configured:

$ dmesg | grep 'apparmor="DENIED"' | grep -F 'profile="/usr/sbin/named"'

Step 4, should not return anything. Because systemd is involved in the user/group lookups, it currently returns the following:

audit: type=1400 audit(1586826072.115:266): apparmor="DENIED" operation="open" namespace="root//lxd-fb1_<var-snap-lxd-common-lxd>" profile="/usr/sbin/named" name="/proc/sys/kernel/random/boot_id" pid=13756 comm="named" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000
audit: type=1400 audit(1586826072.115:267): apparmor="DENIED" operation="open" namespace="root//lxd-fb1_<var-snap-lxd-common-lxd>" profile="/usr/sbin/named" name="/proc/sys/kernel/random/boot_id" pid=13756 comm="named" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000
audit: type=1400 audit(1586826072.115:268): apparmor="DENIED" operation="open" namespace="root//lxd-fb1_<var-snap-lxd-common-lxd>" profile="/usr/sbin/named" name="/proc/sys/kernel/random/boot_id" pid=13756 comm="named" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000
audit: type=1400 audit(1586826072.115:269): apparmor="DENIED" operation="open" namespace="root//lxd-fb1_<var-snap-lxd-common-lxd>" profile="/usr/sbin/named" name="/proc/sys/kernel/random/boot_id" pid=13756 comm="named" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000
audit: type=1400 audit(1586826072.115:270): apparmor="DENIED" operation="open" namespace="root//lxd-fb1_<var-snap-lxd-common-lxd>" profile="/usr/sbin/named" name="/proc/sys/kernel/random/boot_id" pid=13756 comm="named" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000

[Regression Potential]

In order to fix this issue, 3 separate patches had to be backported. They are simple and self-contained, especially two of them, whose purposes are to add the definition of the @{run} variable and then to add a trailing slash at the end of the "/run" pathname.

The other patch, albeit very simple, adds three statements to the 'nameservice' profile in order to let processes access (read-only) files under "/run/systemd/userdb" and "/proc/sys/kernel/random/boot_id". After thinking about the possible cases, the only possible problem I could envision was for a program that, not being able to access some of these files before, will now be able to do that and therefore exercise a part of its codebase which was not being used, possibly uncovering latent bugs in this software. But this is not a regression of apparmor per se.

[Original Description]

(Description and Test Case were moved above)

# Workaround

1) remove systemd from nsswitch.conf
$ lxc exec fb1 -- sed -i 's/ systemd$/ # systemd/' /etc/nsswitch.conf
2) restart named
$ lxc exec fb1 -- service named restart
3) notice no more denials in kernel logs

# Additional information

root@fb1:~# apt-cache policy apparmor
apparmor:
  Installed: 2.13.3-7ubuntu4
  Candidate: 2.13.3-7ubuntu4
  Version table:
 *** 2.13.3-7ubuntu4 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status

root@fb1:~# uname -a
Linux fb1 5.3.0-46-generic #38~18.04.1-Ubuntu SMP Tue Mar 31 04:17:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

root@fb1:~# lsb_release -rd
Description: Ubuntu Focal Fossa (development branch)
Release: 20.04

Related branches

Simon Déziel (sdeziel) wrote :

On all my machines and using various daemons, the denial messages always have fsuid==ouid. As such, I believe it would be OK to use the 'owner' specifier like this:

  owner @{PROC}/sys/kernel/random/boot_id r,

Simon Déziel (sdeziel) wrote :

Scratch that. Using 'owner' on a root-owned but world readable file is probably ill-advised in an abstraction. It seems plausible for an application to do NSS lookup for user/group while running as non-root.

Dimitri John Ledkov (xnox) wrote :

Which lxd are you using? Because more recent ones, should be creating a per-container boot_id.

Simon Déziel (sdeziel) wrote :

`snap info lxd` says:
installed: 4.0.1 (14890) 72MB -

And indeed, there is a tmpfs mounted there:

root@bind:~# mount | grep boot
none on /proc/sys/kernel/random/boot_id type tmpfs (ro,nosuid,nodev,noexec,relatime,size=492k,mode=755,uid=1524288,gid=1524288)

That said, I don't think there is anything lxd specific to this issue as similar behavior is observable on physical/virtual machines where lxd is not used at all.

Andreas Hasenack (ahasenack) wrote :

Squid is failing to start due to this apparmor deny:
[ 7271.822230] audit: type=1400 audit(1588602033.905:516): apparmor="DENIED" operation="open" namespace="root//lxd-autopkgtest-lxd-sljvrl_<var-snap-lxd-common-lxd>" profile="/usr/sbin/squid" name="/proc/sys/kernel/random/boot_id" pid=289530 comm="squid" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000

which results in:
2020/05/04 14:20:34 kid1| WARNING: failed to send start-up notification to systemd
    sd_notify() error: (13) Permission denied

and
# time systemctl start squid
Job for squid.service failed because a timeout was exceeded.
See "systemctl status squid.service" and "journalctl -xe" for details.

real 2m6.317s
user 0m0.014s
sys 0m0.011s

Andreas Hasenack (ahasenack) wrote :

That was squid 4.11, for groovy, btw. squid as shipped in focal is working fine.

Simon Déziel (sdeziel) wrote :

squid in focal is indeed another package that triggers that denial but it is non fatal there as mentioned by Andreas.

@ahasenack, with 4.11, squid's systemd unit moved from Type=forking to Type=notify and with the error you showed, I would expect you to see a denial trying to write to /run/systemd/notify. I don't think a rule for /run/systemd/notify was added in any abstraction (yet) and I don't see any such rule in squid's profile itself.

Simon Déziel (sdeziel) wrote :

The missing rule for boot_id was added to Apparmor 2.13 (https://gitlab.com/apparmor/apparmor/-/blob/apparmor-2.13/profiles/apparmor.d/abstractions/nameservice#L35) and was later refined in the master branch. As such, marking as fix committed.

Changed in apparmor (Ubuntu):
status: New → Fix Committed

I'm building a PPA with the backported fix here:

https://launchpad.net/~sergiodj/+archive/ubuntu/apparmor-bug1872564

Changed in apparmor (Ubuntu):
assignee: nobody → Sergio Durigan Junior (sergiodj)
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apparmor (Ubuntu Focal):
status: New → Confirmed
Changed in apparmor (Ubuntu Focal):
assignee: nobody → Sergio Durigan Junior (sergiodj)
description: updated
Daniel Richard G. (skunk) wrote :

Thanks for being on top of this, Sergio. I'm surprised that a LP search for "boot_id" in this project did not turn up this existing bug report.

description: updated
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apparmor - 2.13.3-7ubuntu6

---------------
apparmor (2.13.3-7ubuntu6) groovy; urgency=medium

  * Add missing "boot_id" rule to abstractions/nameservice. (LP: #1872564)
    - d/p/upstream-commit-454fca7-Add-run-variable.patch: Add the
      definition for the "@{run}" variable.
    - d/p/upstream-commit-ef591a67-Add-trailing-slash-to-the-run-variable-definition.patch:
      Add trailing slash to the "@{run}" variable.
    - d/p/upstream-commit-1f319c3870-abstractions-nameservice-allow-accessing-run-systemd-user.patch:
      Add a missing rule to allow systemd to access
      @{PROC}/sys/kernel/random/boot_id and @{run}/systemd/userdb.
    - d/apparmor.install: Install new file 'tunables/run' under '/etc/apparmor.d'.

 -- Sergio Durigan Junior <email address hidden> Mon, 11 May 2020 09:55:16 -0400

Changed in apparmor (Ubuntu):
status: Fix Committed → Fix Released
Jamie Strandboge (jdstrand) wrote :

@Sergio, I didn't see that you uploaded anything to the queue so to expedite the SRU since there are a number of duplicates, I created a smaller backport of the fix and uploaded it to focal-proposed just now: http://launchpadlibrarian.net/480473812/apparmor_2.13.3-7ubuntu5_2.13.3-7ubuntu5.1.diff.gz

(I hope that is alright).

Changed in apparmor (Ubuntu Focal):
status: Confirmed → In Progress

On Tuesday, May 19 2020, Jamie Strandboge wrote:

> @Sergio, I didn't see that you uploaded anything to the queue so to
> expedite the SRU since there are a number of duplicates, I created a
> smaller backport of the fix and uploaded it to focal-proposed just now:
> http://launchpadlibrarian.net/480473812/apparmor_2.13.3-7ubuntu5_2.13.3-7ubuntu5.1.diff.gz
>
> (I hope that is alright).

Thanks, Jamie! That's quite alright. There's an MP opened about this,
but we got sidetracked and forgot to follow up.

Thanks again.

--
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0 EB2F 106D A1C8 C3CB BF14

Jamie Strandboge (jdstrand) wrote :

@Sergio - assuming you are ok with my patch, do you still plan to follow through on the SRU verification once it is accepted into focal-proposed?

On Tuesday, May 19 2020, Jamie Strandboge wrote:

> @Sergio - assuming you are ok with my patch, do you still plan to follow
> through on the SRU verification once it is accepted into focal-proposed?

Hi Jamie,

Yes, I can take care of the verification if no one else does it.

Thanks,

--
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0 EB2F 106D A1C8 C3CB BF14

Simon Déziel (sdeziel) wrote :

To save you some work, I'll be happy to do the verification as soon as something lands in focal-proposed. Thanks

On Wednesday, May 20 2020, Simon Déziel wrote:

> To save you some work, I'll be happy to do the verification as soon as
> something lands in focal-proposed. Thanks

Thanks, Simon! Much appreciated.

--
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0 EB2F 106D A1C8 C3CB BF14

Hello Simon, or anyone else affected,

Accepted apparmor into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apparmor/2.13.3-7ubuntu5.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in apparmor (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal
Simon Déziel (sdeziel) wrote :

After pulling apparmor 2.13.3-7ubuntu5.1 from focal-proposed:

Get:18 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 apparmor amd64 2.13.3-7ubuntu5.1 [494 kB]
...
Unpacking apparmor (2.13.3-7ubuntu5.1) over (2.13.3-7ubuntu5) ...
Setting up libapparmor1:amd64 (2.13.3-7ubuntu5.1) ...
Setting up apt-utils (2.0.3) ...
Setting up apparmor (2.13.3-7ubuntu5.1) ...
Installing new version of config file /etc/apparmor.d/abstractions/nameservice ...
Reloading AppArmor profiles
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
...

I'm happy to report the bug is fixed, thanks so much!

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal

All autopkgtests for the newly accepted apparmor (2.13.3-7ubuntu5.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

systemd/245.4-4ubuntu3 (ppc64el)
libreoffice/1:6.4.3-0ubuntu0.20.04.1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#apparmor

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Marco Davids (mdavids) on 2020-05-27
Changed in apparmor (Ubuntu Focal):
status: Fix Committed → Fix Released
Jamie Strandboge (jdstrand) wrote :

@Marco, this issue is not yet fixed in Focal. Marking back to Fix Committed.

Changed in apparmor (Ubuntu Focal):
status: Fix Released → Fix Committed
Jamie Strandboge (jdstrand) wrote :

The autopkgtest failures seem unrelated. I triggered reruns just now.

Jamie Strandboge (jdstrand) wrote :

FYI, those re-runs passed and the package is green in https://people.canonical.com/~ubuntu-archive/pending-sru.html. When ubuntu-sru goes through the queue, this will be published.

On Monday, June 01 2020, Jamie Strandboge wrote:

> FYI, those re-runs passed and the package is green in
> https://people.canonical.com/~ubuntu-archive/pending-sru.html. When
> ubuntu-sru goes through the queue, this will be published.

Thanks for taking care of this one, Jamie!

--
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0 EB2F 106D A1C8 C3CB BF14

Brian Murray (brian-murray) wrote :

I don't see the following step from the Test Case performed in comment #20. Was it?

4) check kernel logs for DENIED
$ journalctl -o cat -b0 -k | grep 'apparmor="DENIED"' | grep -F 'profile="/usr/sbin/named"'

or, depending on how logging is configured:

$ dmesg | grep 'apparmor="DENIED"' | grep -F 'profile="/usr/sbin/named"'

Step 4, should not return anything. Because systemd is involved in the user/group lookups, it currently returns the following:

tags: added: verification-needed verification-needed-focal
removed: verification-done verification-done-focal
Simon Déziel (sdeziel) on 2020-06-02
tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Simon Déziel (sdeziel) wrote :

@Brian, I did go through the full test case when marking it as verified in comment #20.

Do I really need to repeat the full test case when verifying a bug?

$ lxc launch images:ubuntu/focal fb1
$ lxc exec fb1 -- apt update && lxc exec fb1 -- apt install apparmor -y
$ lxc exec fb1 -- apt install bind9 -y

# Confirms the problem:
$ journalctl -o cat -b0 -k | grep 'apparmor="DENIED"' | grep -F 'profile="/usr/sbin/named"'
audit: type=1400 audit(1591130868.387:930): apparmor="DENIED" operation="open" namespace="root//lxd-fb1_<var-snap-lxd-common-lxd>" profile="/usr/sbin/named" name="/proc/sys/kernel/random/boot_id" pid=21656 comm="named" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000
audit: type=1400 audit(1591130868.387:931): apparmor="DENIED" operation="open" namespace="root//lxd-fb1_<var-snap-lxd-common-lxd>" profile="/usr/sbin/named" name="/proc/sys/kernel/random/boot_id" pid=21656 comm="named" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000
audit: type=1400 audit(1591130868.387:932): apparmor="DENIED" operation="open" namespace="root//lxd-fb1_<var-snap-lxd-common-lxd>" profile="/usr/sbin/named" name="/proc/sys/kernel/random/boot_id" pid=21656 comm="named" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000
audit: type=1400 audit(1591130868.387:933): apparmor="DENIED" operation="open" namespace="root//lxd-fb1_<var-snap-lxd-common-lxd>" profile="/usr/sbin/named" name="/proc/sys/kernel/random/boot_id" pid=21656 comm="named" requested_mask="r" denied_mask="r" fsuid=1000000 ouid=1000000

Bringing in the fix from -proposed:

$ echo 'deb http://archive.ubuntu.com/ubuntu focal-proposed main' | lxc exec fb1 -- tee /etc/apt/sources.list
$ lxc exec fb1 -- apt update
$ lxc exec fb1 -- apt install apparmor
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  apparmor-profiles-extra apparmor-utils
The following packages will be upgraded:
  apparmor
1 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.
Need to get 494 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 apparmor amd64 2.13.3-7ubuntu5.1 [494 kB]
Fetched 494 kB in 1s (929 kB/s)
Preconfiguring packages ...
(Reading database ... 14968 files and directories currently installed.)
Preparing to unpack .../apparmor_2.13.3-7ubuntu5.1_amd64.deb ...
Unpacking apparmor (2.13.3-7ubuntu5.1) over (2.13.3-7ubuntu5) ...
Setting up apparmor (2.13.3-7ubuntu5.1) ...
Installing new version of config file /etc/apparmor.d/abstractions/nameservice ...
Reloading AppArmor profiles
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Processing triggers for systemd (245.4-4ubuntu3.1) ...
$ lxc exec fb1 -- systemctl restart named

No *new* DENIED messages in 'journalctl -k', so marking as verification-done.

Chris Halse Rogers (raof) wrote :

You don't *have* to include the full output of the test cases when verifying a bug (although, depending on how much output there is, it can be nice).

I don't think it was clear that you *had* gone through the full test-case in your verification comment - I'm not entirely sure what gave that impression, but I think it might have been the combination of *some* output (the apt/dpkg bit) and saying “the bug is fixed, thanks” without making reference to the test case.

Thanks for testing!

The verification of the Stable Release Update for apparmor has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apparmor - 2.13.3-7ubuntu5.1

---------------
apparmor (2.13.3-7ubuntu5.1) focal-proposed; urgency=medium

  * upstream-lp1872564.patch: adjust nameservice abstraction for nss-systemd
    - LP: #1872564

 -- Jamie Strandboge <email address hidden> Tue, 19 May 2020 16:59:49 +0000

Changed in apparmor (Ubuntu Focal):
status: Fix Committed → Fix Released

On 2020-06-02 8:50 p.m., Chris Halse Rogers wrote:
> You don't *have* to include the full output of the test cases when
> verifying a bug (although, depending on how much output there is, it can
> be nice).

OK, good, thanks for clarifying!

> I don't think it was clear that you *had* gone through the full test-
> case in your verification comment - I'm not entirely sure what gave that
> impression, but I think it might have been the combination of *some*
> output (the apt/dpkg bit) and saying “the bug is fixed, thanks” without
> making reference to the test case.

True, I should have been more explicit, duly noted!

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers