Apparmor denial on /var/lib/dpkg/arch

[L W R]

[ Impact ]

Systems with a /var/lib/dpkg/arch file will trigger an apparmor DENIED log entry when the esm-cache service tries to access that file.

Not all systems will have /var/lib/dpkg/arch. It can be created, probably among other scenarios, when a subarchitecture is added. For example, on amd64 systems, it's quite common to also have i386 added via the command

  sudo dpkg --add-architecture i386

That is enough to create /var/lib/dpkg/arch populated with both am64 and i386, and trigger this bug.

Within the Pro client, we determined that the bug is triggered when a) that file exists; and b) when the Pro client, as part of running the esm-cache.service service, calls `apt-cache policy`. That will trigger an access to /var/lib/dpkg/arch under the dpkg and other apparmor subprofiles defined in /etc/apparmor.d/ubuntu_pro_esm_cache, and result in apparmor denying that access.

After learning of this bug, we ran the upstream test suite with the bug trigger in place, without the fix, and no tests have been found that failed because of this bug (other than the check for apparmor DENIED logs). Even so, this influx of apparmor logs can be troubling and noisy, or we could have missed a scenario where it really triggers an incorrect behavior in the Pro client. Given that the fix is simple, and easy to test, we decided to proceed with this SRU.

[ Test Plan ]

a) very specific test for this issue. Needs to be run in a VM, not LXD, otherwise apparmor will block /dev/pts/* which affects this test (but does not affect the esm-cache.service -- see test (b))

- install the Pro client version to be tested
- run these commands:

  sudo touch /var/lib/dpkg/arch
  sudo aa-exec -p ubuntu_pro_esm_cache//dpkg dpkg --print-foreign-architectures
  sudo aa-exec -p ubuntu_pro_esm_cache apt-cache policy

Without the fix, they will produce apparmor DENIED messages in the dmesg logs showing an attempted access to /var/lib/dpkg/arch, and in addition to that, the dpkg one will fail (apt-cache policy won't fail)

b) esm-cache.service test (only in an LTS)
- install the Pro client version to be tested
- run these commands in sequence as root:

  touch /var/lib/dpkg/arch
  rm -rf /var/lib/apt/periodic/*
  systemctl start esm-cache.service

Without the fix, the dmesg logs will contain apparmor DENIED messages showing attempted accesses to /var/lib/dpkg/arch.

[ Where problems could occur ]

A syntax error in the apparmor profile would prevent it from loading, and remove its protection entirely. To account for that, the package build process runs an apparmor static check on the generated profiles, and if that fails, the package build fails. It could still be susceptible to errors at profile load-time regarding the running kernel, which is likely different than the running kernel in the launchpad builders.

Another type of mistake that could happen is inadvertently opening up the profile more than is needed. But the extra access we are giving here is read-only, and the affected profiles do need that access.

[ Other Info ]

Upstream bug report: https://github.com/canonical/ubuntu-pro-client/issues/3137

Unfortunately this wasn't caught by the extensive Pro test suite because the test units (vms, lxd containers) never had a /var/lib/dpkg/arch file in them. Likewise, the development container where this profile was first created also didn't have that file.

[ Original Description ]

ubuntu-advantage-tools 32.3~18.04 is causing a new apparmor denial on Bionic when updating:

[ 8091.769560] audit: type=1400 audit(1717273124.410:121): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=10358 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Fix:

--- /etc/apparmor.d/ubuntu_pro_esm_cache.orig 2024-06-01 22:31:28.276735437 +0200
+++ /etc/apparmor.d/ubuntu_pro_esm_cache 2024-06-01 22:31:07.163884846 +0200
@@ -174,6 +174,8 @@

     /etc/dpkg/** r,

+ /var/lib/dpkg/** r,
+
     /{,usr/}bin/dpkg mr,

   }

[Andreas Hasenack (ahasenack)]

Hi,

we haven't seen this denial in our testing, could you please help to narrow it down to which conditions trigger it? Can you perhaps map the apparmor deny timestamp with something in your system logs, like /var/log/syslog or /var/log/ubuntu-advantage.log?

[L W R (nopheer)]

When I execute "apt update", then the denial happens. It also happens upon boot. I did not see anything relevant in /var/log/ubuntu-advantage.log, but here is a snippet of /var/log/syslog on bootup:

Jun 1 20:04:07 droid systemd[1]: Starting Update APT News...
Jun 1 20:04:07 droid systemd[1]: Starting Update the local ESM caches...
Jun 1 20:04:07 droid systemd[1]: Started Update APT News.
Jun 1 20:04:07 droid kernel: [ 14.897120] kauditd_printk_skb: 48 callbacks suppressed
Jun 1 20:04:07 droid kernel: [ 14.897125] audit: type=1400 audit(1717265047.937:60): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=2380 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jun 1 20:04:08 droid kernel: [ 14.994224] audit: type=1400 audit(1717265048.033:61): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=2381 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jun 1 20:04:08 droid kernel: [ 14.997591] audit: type=1400 audit(1717265048.037:62): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=2382 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jun 1 20:04:08 droid kernel: [ 15.000624] audit: type=1400 audit(1717265048.041:63): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=2383 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

[Andreas Hasenack (ahasenack)]

If you run this command, does it print the architecture, or fail in any way? And does it log a new apparmor DENIED message?

  sudo aa-exec -p ubuntu_pro_esm_cache//dpkg dpkg --print-architecture

[L W R (nopheer)]

When I run the command you mentioned, it prints the correct architecture successfully and it does not log a new apparmor DENIED message.

[Andreas Hasenack (ahasenack)]

Ok, we found a reproducer:

sudo dpkg --add-architecture i386

Then either command will trigger the DENIED:

sudo aa-exec -p ubuntu_pro_esm_cache//dpkg dpkg --print-foreign-architecture
sudo aa-exec -p ubuntu_pro_esm_cache apt-cache policy

Just the presence of /var/lib/dpkg/arch will trigger it, even if it's empty. If the file does not exist, then there is no apparmor DENIED.

We will apply your patch.

[Andreas Hasenack (ahasenack)]

Upstream issue: https://github.com/canonical/ubuntu-pro-client/issues/3137

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 32.3.1

---------------
ubuntu-advantage-tools (32.3.1) oracular; urgency=medium

  * Adjust the esm_cache apparmor profile to allow reading of dpkg data
    directory (LP: #2067810):
    - d/apparmor/ubuntu_pro_esm_cache.jinja2: allow /var/lib/dpkg/** for dpkg
      and other profiles
    - features/steps/machines.py: trigger the bug in the behave test suite,
      which tests the fix
  * version.py: update version to 32.3.1

 -- Andreas Hasenack <email address hidden> Fri, 07 Jun 2024 14:52:55 -0300

[Mauricio Faria de Oliveira (mfo)]

Hello L, or anyone else affected,

Accepted ubuntu-advantage-tools into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/32.3.1~24.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mauricio Faria de Oliveira (mfo)]

Hello L, or anyone else affected,

Accepted ubuntu-advantage-tools into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/32.3.1~23.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mauricio Faria de Oliveira (mfo)]

Hello L, or anyone else affected,

Accepted ubuntu-advantage-tools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/32.3.1~22.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mauricio Faria de Oliveira (mfo)]

Hello L, or anyone else affected,

Accepted ubuntu-advantage-tools into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/32.3.1~20.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mauricio Faria de Oliveira (mfo)]

Hello L, or anyone else affected,

Accepted ubuntu-advantage-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/32.3.1~18.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mauricio Faria de Oliveira (mfo)]

Hello L, or anyone else affected,

Accepted ubuntu-advantage-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/32.3.1~16.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted ubuntu-advantage-tools (32.3.1~24.04) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

software-properties/unknown (amd64, arm64, armhf, i386, ppc64el, s390x)
wsl-pro-service/0.1.4 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#ubuntu-advantage-tools

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted ubuntu-advantage-tools (32.3.1~22.04) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

update-motd/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#ubuntu-advantage-tools

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Andreas Hasenack (ahasenack)]

Noble verification

# Reproducing the error

root@n-uat-2067810:~# apt-cache policy ubuntu-pro-client
ubuntu-pro-client:
  Installed: 32.3~24.04
  Candidate: 32.3~24.04
  Version table:
 *** 32.3~24.04 500
        500 http://br.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     31.2.3 500
        500 http://br.archive.ubuntu.com/ubuntu noble/main amd64 Packages

root@n-uat-2067810:~# pro version
32.3~24.04

Test case (a):

root@n-uat-2067810:~# aa-exec -p ubuntu_pro_esm_cache//dpkg dpkg --print-foreign-architectures
root@n-uat-2067810:~#

dmesg shows:
[Thu Jun 20 21:06:45 2024] audit: type=1400 audit(1718917605.728:124): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=3074 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

root@n-uat-2067810:~# aa-exec -p ubuntu_pro_esm_cache apt-cache policy
Package files:
 100 /var/lib/dpkg/status
     release a=now
 500 http://br.archive.ubuntu.com/ubuntu noble-security/restricted amd64 Packages
     release v=24.04,o=Ubuntu,a=noble-security,n=noble,l=Ubuntu,c=restricted,b=amd64
     origin br.archive.ubuntu.com
...
(normal output)
...
Pinned packages:
root@n-uat-2067810:~#

dmesg shows:
[Thu Jun 20 21:07:09 2024] audit: type=1400 audit(1718917629.197:125): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=3086 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[Thu Jun 20 21:07:09 2024] audit: type=1400 audit(1718917629.202:126): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=3087 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Test case (b):
root@n-uat-2067810:~# systemctl start esm-cache.service
root@n-uat-2067810:~#

dmesg shows multiple DENIED entries, like:
[Thu Jun 20 21:10:04 2024] audit: type=1400 audit(1718917804.553:139): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=3505 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[Thu Jun 20 21:10:04 2024] audit: type=1400 audit(1718917804.558:140): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=3506 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[Thu Jun 20 21:10:05 2024] audit: type=1400 audit(1718917805.323:141): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//apt_methods_gpgv" name="/var/lib/dpkg/arch" pid=3515 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=42 ouid=0

# Verifying the fix

Upgrading to version from proposed:
root@n-uat-2067810:~# apt-cache policy ubuntu-pro-client
ubuntu-pro-client:
  Installed: 32.3.1~24.04
  Candidate: 32.3.1~24.04
  Version table:
 *** 32.3.1~24.04 100
        100 http://br.archive.ubuntu.com/ubuntu noble-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     32.3~24.04 500
        500 http://br.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages
     31.2.3 500
        500 http://br.archive.ubuntu.com/ubuntu noble/...