Apparmor denial on /var/lib/dpkg/arch

[L W R]

[ Impact ]

Systems with a /var/lib/dpkg/arch file will trigger an apparmor DENIED log entry when the esm-cache service tries to access that file.

Not all systems will have /var/lib/dpkg/arch. It can be created, probably among other scenarios, when a subarchitecture is added. For example, on amd64 systems, it's quite common to also have i386 added via the command

  sudo dpkg --add-architecture i386

That is enough to create /var/lib/dpkg/arch populated with both am64 and i386, and trigger this bug.

Within the Pro client, we determined that the bug is triggered when a) that file exists; and b) when the Pro client, as part of running the esm-cache.service service, calls `apt-cache policy`. That will trigger an access to /var/lib/dpkg/arch under the dpkg and other apparmor subprofiles defined in /etc/apparmor.d/ubuntu_pro_esm_cache, and result in apparmor denying that access.

After learning of this bug, we ran the upstream test suite with the bug trigger in place, without the fix, and no tests have been found that failed because of this bug (other than the check for apparmor DENIED logs). Even so, this influx of apparmor logs can be troubling and noisy, or we could have missed a scenario where it really triggers an incorrect behavior in the Pro client. Given that the fix is simple, and easy to test, we decided to proceed with this SRU.

[ Test Plan ]

a) very specific test for this issue. Needs to be run in a VM, not LXD, otherwise apparmor will block /dev/pts/* which affects this test (but does not affect the esm-cache.service -- see test (b))

- install the Pro client version to be tested
- run these commands:

  sudo touch /var/lib/dpkg/arch
  sudo aa-exec -p ubuntu_pro_esm_cache//dpkg dpkg --print-foreign-architectures
  sudo aa-exec -p ubuntu_pro_esm_cache apt-cache policy

Without the fix, they will produce apparmor DENIED messages in the dmesg logs showing an attempted access to /var/lib/dpkg/arch, and in addition to that, the dpkg one will fail (apt-cache policy won't fail)

b) esm-cache.service test (only in an LTS)
- install the Pro client version to be tested
- run these commands in sequence as root:

  touch /var/lib/dpkg/arch
  rm -rf /var/lib/apt/periodic/*
  systemctl start esm-cache.service

Without the fix, the dmesg logs will contain apparmor DENIED messages showing attempted accesses to /var/lib/dpkg/arch.

[ Where problems could occur ]

A syntax error in the apparmor profile would prevent it from loading, and remove its protection entirely. To account for that, the package build process runs an apparmor static check on the generated profiles, and if that fails, the package build fails. It could still be susceptible to errors at profile load-time regarding the running kernel, which is likely different than the running kernel in the launchpad builders.

Another type of mistake that could happen is inadvertently opening up the profile more than is needed. But the extra access we are giving here is read-only, and the affected profiles do need that access.

[ Other Info ]

Upstream bug report: https://github.com/canonical/ubuntu-pro-client/issues/3137

Unfortunately this wasn't caught by the extensive Pro test suite because the test units (vms, lxd containers) never had a /var/lib/dpkg/arch file in them. Likewise, the development container where this profile was first created also didn't have that file.

[ Original Description ]

ubuntu-advantage-tools 32.3~18.04 is causing a new apparmor denial on Bionic when updating:

[ 8091.769560] audit: type=1400 audit(1717273124.410:121): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=10358 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Fix:

--- /etc/apparmor.d/ubuntu_pro_esm_cache.orig 2024-06-01 22:31:28.276735437 +0200
+++ /etc/apparmor.d/ubuntu_pro_esm_cache 2024-06-01 22:31:07.163884846 +0200
@@ -174,6 +174,8 @@

     /etc/dpkg/** r,

+ /var/lib/dpkg/** r,
+
     /{,usr/}bin/dpkg mr,

   }

[Andreas Hasenack (ahasenack)]

Hi,

we haven't seen this denial in our testing, could you please help to narrow it down to which conditions trigger it? Can you perhaps map the apparmor deny timestamp with something in your system logs, like /var/log/syslog or /var/log/ubuntu-advantage.log?

[L W R (nopheer)]

When I execute "apt update", then the denial happens. It also happens upon boot. I did not see anything relevant in /var/log/ubuntu-advantage.log, but here is a snippet of /var/log/syslog on bootup:

Jun 1 20:04:07 droid systemd[1]: Starting Update APT News...
Jun 1 20:04:07 droid systemd[1]: Starting Update the local ESM caches...
Jun 1 20:04:07 droid systemd[1]: Started Update APT News.
Jun 1 20:04:07 droid kernel: [ 14.897120] kauditd_printk_skb: 48 callbacks suppressed
Jun 1 20:04:07 droid kernel: [ 14.897125] audit: type=1400 audit(1717265047.937:60): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=2380 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jun 1 20:04:08 droid kernel: [ 14.994224] audit: type=1400 audit(1717265048.033:61): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=2381 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jun 1 20:04:08 droid kernel: [ 14.997591] audit: type=1400 audit(1717265048.037:62): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=2382 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jun 1 20:04:08 droid kernel: [ 15.000624] audit: type=1400 audit(1717265048.041:63): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=2383 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

[Andreas Hasenack (ahasenack)]

If you run this command, does it print the architecture, or fail in any way? And does it log a new apparmor DENIED message?

  sudo aa-exec -p ubuntu_pro_esm_cache//dpkg dpkg --print-architecture

[L W R (nopheer)]

When I run the command you mentioned, it prints the correct architecture successfully and it does not log a new apparmor DENIED message.

[Andreas Hasenack (ahasenack)]

Ok, we found a reproducer:

sudo dpkg --add-architecture i386

Then either command will trigger the DENIED:

sudo aa-exec -p ubuntu_pro_esm_cache//dpkg dpkg --print-foreign-architecture
sudo aa-exec -p ubuntu_pro_esm_cache apt-cache policy

Just the presence of /var/lib/dpkg/arch will trigger it, even if it's empty. If the file does not exist, then there is no apparmor DENIED.

We will apply your patch.

[Andreas Hasenack (ahasenack)]

Upstream issue: https://github.com/canonical/ubuntu-pro-client/issues/3137

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-advantage-tools - 32.3.1

---------------
ubuntu-advantage-tools (32.3.1) oracular; urgency=medium

  * Adjust the esm_cache apparmor profile to allow reading of dpkg data
    directory (LP: #2067810):
    - d/apparmor/ubuntu_pro_esm_cache.jinja2: allow /var/lib/dpkg/** for dpkg
      and other profiles
    - features/steps/machines.py: trigger the bug in the behave test suite,
      which tests the fix
  * version.py: update version to 32.3.1

 -- Andreas Hasenack <email address hidden> Fri, 07 Jun 2024 14:52:55 -0300

[Mauricio Faria de Oliveira (mfo)]

Hello L, or anyone else affected,

Accepted ubuntu-advantage-tools into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/32.3.1~24.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mauricio Faria de Oliveira (mfo)]

Hello L, or anyone else affected,

Accepted ubuntu-advantage-tools into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/32.3.1~23.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mauricio Faria de Oliveira (mfo)]

Hello L, or anyone else affected,

Accepted ubuntu-advantage-tools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/32.3.1~22.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mauricio Faria de Oliveira (mfo)]

Hello L, or anyone else affected,

Accepted ubuntu-advantage-tools into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/32.3.1~20.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mauricio Faria de Oliveira (mfo)]

Hello L, or anyone else affected,

Accepted ubuntu-advantage-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/32.3.1~18.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mauricio Faria de Oliveira (mfo)]

Hello L, or anyone else affected,

Accepted ubuntu-advantage-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-advantage-tools/32.3.1~16.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted ubuntu-advantage-tools (32.3.1~24.04) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

software-properties/unknown (amd64, arm64, armhf, i386, ppc64el, s390x)
wsl-pro-service/0.1.4 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#ubuntu-advantage-tools

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted ubuntu-advantage-tools (32.3.1~22.04) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

update-motd/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#ubuntu-advantage-tools

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Andreas Hasenack (ahasenack)]

Noble verification

# Reproducing the error

root@n-uat-2067810:~# apt-cache policy ubuntu-pro-client
ubuntu-pro-client:
  Installed: 32.3~24.04
  Candidate: 32.3~24.04
  Version table:
 *** 32.3~24.04 500
        500 http://br.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     31.2.3 500
        500 http://br.archive.ubuntu.com/ubuntu noble/main amd64 Packages

root@n-uat-2067810:~# pro version
32.3~24.04

Test case (a):

root@n-uat-2067810:~# aa-exec -p ubuntu_pro_esm_cache//dpkg dpkg --print-foreign-architectures
root@n-uat-2067810:~#

dmesg shows:
[Thu Jun 20 21:06:45 2024] audit: type=1400 audit(1718917605.728:124): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=3074 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

root@n-uat-2067810:~# aa-exec -p ubuntu_pro_esm_cache apt-cache policy
Package files:
 100 /var/lib/dpkg/status
     release a=now
 500 http://br.archive.ubuntu.com/ubuntu noble-security/restricted amd64 Packages
     release v=24.04,o=Ubuntu,a=noble-security,n=noble,l=Ubuntu,c=restricted,b=amd64
     origin br.archive.ubuntu.com
...
(normal output)
...
Pinned packages:
root@n-uat-2067810:~#

dmesg shows:
[Thu Jun 20 21:07:09 2024] audit: type=1400 audit(1718917629.197:125): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=3086 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[Thu Jun 20 21:07:09 2024] audit: type=1400 audit(1718917629.202:126): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=3087 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Test case (b):
root@n-uat-2067810:~# systemctl start esm-cache.service
root@n-uat-2067810:~#

dmesg shows multiple DENIED entries, like:
[Thu Jun 20 21:10:04 2024] audit: type=1400 audit(1718917804.553:139): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=3505 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[Thu Jun 20 21:10:04 2024] audit: type=1400 audit(1718917804.558:140): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=3506 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[Thu Jun 20 21:10:05 2024] audit: type=1400 audit(1718917805.323:141): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//apt_methods_gpgv" name="/var/lib/dpkg/arch" pid=3515 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=42 ouid=0

# Verifying the fix

Upgrading to version from proposed:
root@n-uat-2067810:~# apt-cache policy ubuntu-pro-client
ubuntu-pro-client:
  Installed: 32.3.1~24.04
  Candidate: 32.3.1~24.04
  Version table:
 *** 32.3.1~24.04 100
        100 http://br.archive.ubuntu.com/ubuntu noble-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     32.3~24.04 500
        500 http://br.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages
     31.2.3 500
        500 http://br.archive.ubuntu.com/ubuntu noble/...

[Andreas Hasenack (ahasenack)]

I'm using this script for the remaining verifications.

[Andreas Hasenack (ahasenack)]

# Testing series mantic
###########################################
Creating mantic-2067810
Starting mantic-2067810
Waiting for container IP
.............Waiting for container ssh
..Connection to 10.0.102.90 22 port [tcp/ssh] succeeded!
Waiting for cloud-init to be done

# Latest u-a-t is installed
###########################################
ubuntu-advantage-tools:
  Installed: 32.3~23.10
  Candidate: 32.3~23.10
  Version table:
 *** 32.3~23.10 500
        500 http://br.archive.ubuntu.com/ubuntu mantic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     29.4 500
        500 http://br.archive.ubuntu.com/ubuntu mantic/main amd64 Packages
###########################################

# Creating conditions for the bug
###########################################
###########################################

# Reproducing the bug
###########################################

# Triggering apparmor DENIED messages
###########################################

# aa-exec -p ubuntu_pro_esm_cache//dpkg dpkg --print-foreign-architectures

# dmesg:
[Fri Jun 21 20:39:14 2024] audit: type=1400 audit(1719002354.512:106): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=1672 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

# aa-exec -p ubuntu_pro_esm_cache apt-cache policy | head
Package files:
 100 /var/lib/dpkg/status
     release a=now
 500 http://br.archive.ubuntu.com/ubuntu mantic-security/multiverse amd64 Packages
     release v=23.10,o=Ubuntu,a=mantic-security,n=mantic,l=Ubuntu,c=multiverse,b=amd64
     origin br.archive.ubuntu.com
 500 http://br.archive.ubuntu.com/ubuntu mantic-security/universe amd64 Packages
     release v=23.10,o=Ubuntu,a=mantic-security,n=mantic,l=Ubuntu,c=universe,b=amd64
     origin br.archive.ubuntu.com
 500 http://br.archive.ubuntu.com/ubuntu mantic-security/restricted amd64 Packages

# dmesg:
[Fri Jun 21 20:39:14 2024] audit: type=1400 audit(1719002354.908:107): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=1679 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[Fri Jun 21 20:39:15 2024] audit: type=1400 audit(1719002354.944:108): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=1680 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

# Skipping esm-cache.service test because mantic is not an LTS

# Updating to proposed
###########################################
deb http://archive.ubuntu.com/ubuntu mantic-proposed main
dpkg-preconfigure: unable to re-open stdin: No such file or directory
ubuntu-advantage-tools:
  Installed: 32.3.1~23.10
  Candidate: 32.3.1~23.10
  Version table:
 *** 32.3.1~23.10 100
        100 http://archive.ubuntu.com/ubuntu mantic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     32.3~23.10 500
        500 http://br.archive.ubuntu.com/ubuntu mantic-updates/main amd64 Packages
     29.4 500
        500 http://br.archive.ubuntu.com/ubuntu mantic/main amd64 Packages

# Now there must be no apparmor DENIED messages
###############################...

[Andreas Hasenack (ahasenack)]

# Testing series jammy
###########################################
Creating jammy-2067810
Starting jammy-2067810
Waiting for container IP
.......Waiting for container ssh
....Connection to 10.0.102.211 22 port [tcp/ssh] succeeded!
Waiting for cloud-init to be done

# Latest u-a-t is installed
###########################################
ubuntu-advantage-tools:
  Installed: 32.3~22.04
  Candidate: 32.3~22.04
  Version table:
 *** 32.3~22.04 500
        500 http://br.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     27.7~22.04.1 500
        500 http://br.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
###########################################

# Creating conditions for the bug
###########################################
###########################################

# Reproducing the bug
###########################################

# Triggering apparmor DENIED messages
###########################################

# aa-exec -p ubuntu_pro_esm_cache//dpkg dpkg --print-foreign-architectures

# dmesg:
[Fri Jun 21 20:38:29 2024] audit: type=1400 audit(1719002309.477:43): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=1435 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

# aa-exec -p ubuntu_pro_esm_cache apt-cache policy | head
Package files:
 100 /var/lib/dpkg/status
     release a=now
 500 http://br.archive.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages
     release v=22.04,o=Ubuntu,a=jammy-security,n=jammy,l=Ubuntu,c=multiverse,b=amd64
     origin br.archive.ubuntu.com
 500 http://br.archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages
     release v=22.04,o=Ubuntu,a=jammy-security,n=jammy,l=Ubuntu,c=universe,b=amd64
     origin br.archive.ubuntu.com
 500 http://br.archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages

# dmesg:
[Fri Jun 21 20:38:29 2024] audit: type=1400 audit(1719002309.833:44): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=1442 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[Fri Jun 21 20:38:29 2024] audit: type=1400 audit(1719002309.853:45): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=1443 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

# esm-cache.service test
###########################################

# /var/lib/apt/periodic/ contents
###########################################
total 8
drwxr-xr-x 2 root root 4096 Jun 21 20:38 .
drwxr-xr-x 5 root root 4096 Jun 21 20:38 ..

# systemctl start esm-cache.service

# dmesg:
[Fri Jun 21 20:38:30 2024] audit: type=1400 audit(1719002310.581:46): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=1456 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[Fri Jun 21 20:38:30 2024] audit: type=1400 audit(1719002310.581:47): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=1457 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[Fri Jun 21 20:38:31 2024] audit: type=14...

[Andreas Hasenack (ahasenack)]

# Testing series focal
###########################################
Creating focal-2067810
Starting focal-2067810
Waiting for container IP
..............Waiting for container ssh
....Connection to 10.0.102.70 22 port [tcp/ssh] succeeded!
Waiting for cloud-init to be done

# Latest u-a-t is installed
###########################################
ubuntu-advantage-tools:
  Installed: 32.3~20.04
  Candidate: 32.3~20.04
  Version table:
 *** 32.3~20.04 500
        500 http://br.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     20.3 500
        500 http://br.archive.ubuntu.com/ubuntu focal/main amd64 Packages
###########################################

# Creating conditions for the bug
###########################################
###########################################

# Reproducing the bug
###########################################

# Triggering apparmor DENIED messages
###########################################

# aa-exec -p ubuntu_pro_esm_cache//dpkg dpkg --print-foreign-architectures

# dmesg:
[Fri Jun 21 20:35:31 2024] audit: type=1400 audit(1719002131.528:40): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=1599 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

# aa-exec -p ubuntu_pro_esm_cache apt-cache policy | head
Package files:
 100 /var/lib/dpkg/status
     release a=now
 500 http://br.archive.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages
     release v=20.04,o=Ubuntu,a=focal-security,n=focal,l=Ubuntu,c=multiverse,b=amd64
     origin br.archive.ubuntu.com
 500 http://br.archive.ubuntu.com/ubuntu focal-security/universe amd64 Packages
     release v=20.04,o=Ubuntu,a=focal-security,n=focal,l=Ubuntu,c=universe,b=amd64
     origin br.archive.ubuntu.com
 500 http://br.archive.ubuntu.com/ubuntu focal-security/restricted amd64 Packages

# dmesg:
[Fri Jun 21 20:35:31 2024] audit: type=1400 audit(1719002131.908:41): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=1607 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[Fri Jun 21 20:35:31 2024] audit: type=1400 audit(1719002131.928:42): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=1608 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

# esm-cache.service test
###########################################

# /var/lib/apt/periodic/ contents
###########################################
total 8
drwxr-xr-x 2 root root 4096 Jun 21 20:35 .
drwxr-xr-x 5 root root 4096 Jun 21 20:35 ..

# systemctl start esm-cache.service

# dmesg:
[Fri Jun 21 20:35:32 2024] audit: type=1400 audit(1719002132.644:43): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=1622 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[Fri Jun 21 20:35:32 2024] audit: type=1400 audit(1719002132.644:44): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=1623 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[Fri Jun 21 20:35:33 2024] audit: type=1400...

[Andreas Hasenack (ahasenack)]

For bionic I had to change the script a bit to be able to launch a lxd bionic VM. In bionic I also didn't always get an apparmor DENIED with the dpkg --print-foreign-architectures, but I did get it when I logged in interactively. Since the other checks produced a DENIED message, and later with the update did not, I decided it was not worth debugging further.

Same with xenial.

# Testing series bionic
###########################################
Creating bionic-2067810
Device config added to bionic-2067810
Waiting for container IP
Waiting for container ssh
Connection to 10.0.102.136 22 port [tcp/ssh] succeeded!
Waiting for cloud-init to be done
timeout: unrecognized option '--verbose'
Try 'timeout --help' for more information.
cloud-init status --wait failed on container bionic-2067810

dpkg-preconfigure: unable to re-open stdin: No such file or directory

# Latest u-a-t is installed
###########################################
ubuntu-advantage-tools:
  Installed: 32.3~18.04
  Candidate: 32.3~18.04
  Version table:
 *** 32.3~18.04 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     17 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
###########################################

# Creating conditions for the bug
###########################################
###########################################

# Reproducing the bug
###########################################

# Triggering apparmor DENIED messages
###########################################

# aa-exec -p ubuntu_pro_esm_cache//dpkg dpkg --print-foreign-architectures

# dmesg:

# aa-exec -p ubuntu_pro_esm_cache apt-cache policy | head
Package files:
 100 /var/lib/dpkg/status
     release a=now
 500 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages
     release v=18.04,o=Ubuntu,a=bionic-security,n=bionic,l=Ubuntu,c=multiverse,b=amd64
     origin security.ubuntu.com
 500 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages
     release v=18.04,o=Ubuntu,a=bionic-security,n=bionic,l=Ubuntu,c=universe,b=amd64
     origin security.ubuntu.com
 500 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages

# dmesg:
[Fri Jun 21 22:13:00 2024] audit: type=1400 audit(1719007981.644:28): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=2251 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[Fri Jun 21 22:13:00 2024] audit: type=1400 audit(1719007981.664:29): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=2252 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

# esm-cache.service test
###########################################

# /var/lib/apt/periodic/ contents
###########################################
total 8
drwxr-xr-x 2 root root 4096 Jun 21 22:13 .
drwxr-xr-x 5 root root 4096 Jun 21 22:12 ..

# systemctl start esm-cache.service

# dmesg:
[Fri Jun 21 22:13:03 2024] audit: type=1400 audit(1719007984.315:30): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=2...

[Andreas Hasenack (ahasenack)]

I had to change the script a bit to be able to launch a xenial lxd vm. Here are the results:

# Testing series xenial
###########################################
Creating xenial-2067810
Device config added to xenial-2067810
Waiting for container IP
Waiting for container ssh
Connection to 10.0.102.82 22 port [tcp/ssh] succeeded!
Waiting for cloud-init to be done
timeout: unrecognized option '--verbose'
Try 'timeout --help' for more information.
cloud-init status --wait failed on container xenial-2067810

dpkg-preconfigure: unable to re-open stdin: No such file or directory

# Latest u-a-t is installed
###########################################
ubuntu-advantage-tools:
  Installed: 32.3~16.04
  Candidate: 32.3~16.04
  Version table:
 *** 32.3~16.04 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status
###########################################

# Creating conditions for the bug
###########################################
###########################################

# Reproducing the bug
###########################################

# Triggering apparmor DENIED messages
###########################################

# aa-exec -p ubuntu_pro_esm_cache//dpkg dpkg --print-foreign-architectures

# dmesg:
[Fri Jun 21 21:45:19 2024] audit: type=1400 audit(1719006320.072:25): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=2471 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

# aa-exec -p ubuntu_pro_esm_cache apt-cache policy | head
Package files:
 100 /var/lib/dpkg/status
     release a=now
 500 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages
     release v=16.04,o=Ubuntu,a=xenial-security,n=xenial,l=Ubuntu,c=multiverse,b=amd64
     origin security.ubuntu.com
 500 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages
     release v=16.04,o=Ubuntu,a=xenial-security,n=xenial,l=Ubuntu,c=universe,b=amd64
     origin security.ubuntu.com
 500 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages

# dmesg:
[Fri Jun 21 21:45:20 2024] audit: type=1400 audit(1719006321.448:26): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=2478 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[Fri Jun 21 21:45:20 2024] audit: type=1400 audit(1719006321.452:27): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=2479 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

# esm-cache.service test
###########################################

# /var/lib/apt/periodic/ contents
###########################################
total 8
drwxr-xr-x 2 root root 4096 Jun 21 21:45 .
drwxr-xr-x 6 root root 4096 Jun 21 21:45 ..

# systemctl start esm-cache.service

# dmesg:

# Updating to proposed
###########################################
deb http://archive.ubuntu.com/ubuntu xenial-proposed main
dpkg-preconfigure: unable to re-open stdin: No such file or directory
ubuntu-advantage-tools:
  Installed: 32.3.1~16.04
  Candidate: 32.3.1~16.04
  Version table:
 *** 32.3.1~16.04...

[Andreas Hasenack (ahasenack)]

v2 of the script, needed for xenial and bionic