AppArmor profiles missing in kernel 5.15.0-1051+ release

[Philip Roche]

After the kernel roll to linux-gcp-5.15 to version 5.15.0-1051.59_20.04.1 the public cloud team pre publication test were failing on our snap_preseed_optimized test which checks to ensure that snaps are preseeded correctly

This test checks the output of `snap debug seeding` to assert `seed-completion` is present and not empty.

``
❯ snap debug seeding
seeded: true
preseeded: true
image-preseeding: 39.367s
seed-completion: 1.335s
```

If `/var/lib/snapd/seed/seed.yaml` exists it also asserts that `preseeded` is present and not empty.

With the recent kernel update this test is failing which indicates a kernel feature mismatch between
the running kernel and the feature set hard-coded in livecd-rootfs for this image.
Boot will be slowed by ~200ms until this is resolved in livecd-rootfs.

This solution is to add a 5.15 apparmor configuration to the focal branch of livecd-rootfs

The issue is also present with the recent 5.15 kernels in Jammy.

Related bugs LP: #2031943 and LP: #2045384

[ Impact ]

Boot will be slowed by ~200ms until this is resolved in livecd-rootfs

[ Test Plan ]

 * for focal build any cloud image with preseeded snaps with HWE 5.15 kernel
 * for jammy build any cloud image with preseeded snaps with up to date 5.15 kernel
 * boot
 * run `snap debug seeding`
 * assert the test described above passes

[ Where problems could occur ]

 * Similar patches already exist for later releases 6.2, 6.5 kernel etc. and have been used on other private customer kernels and all kernels released after 22.04, so there is already a good track record for this patchset and it shouldn't create any issues.

[ Other Info ]

 * This is a time-sensitive issue for a paying customer

[John Chittum (jchittum)]

upstream linux bug that tracked the change.

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2045384

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Philip Roche (philroche)]

This also affects the recent Jammy 5.15 kernel. Updating title, description and target

[Timo Aaltonen (tjaalton)]

only the focal version seems to have been uploaded at this stage?

[Philip Roche (philroche)]

Jammy fix is still awaiting review @ https://code.launchpad.net/~philroche/livecd-rootfs/+git/livecd-rootfs/+merge/460929

[Mauricio Faria de Oliveira (mfo)]

Marking Jammy as Incomplete per #5, #6 (MR still awaiting review).

[Andreas Hasenack (ahasenack)]

> This also affects the recent Jammy 5.15 kernel. Updating title, description and target

Please also update the test plan to include a jammy scenario.

[Philip Roche (philroche)]

Test plan updated for Jammy too

[Andreas Hasenack (ahasenack)]

Still missing a jammy upload. The linked PR was approved just 2h ago, though, so looks like it's close! :)

[Philip Roche (philroche)]

Now uploaded to Jammy queue @ https://launchpad.net/ubuntu/jammy/+queue?queue_state=1&queue_text=livecd-rootfs

[Brian Murray (brian-murray)]

Hello Philip, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.41 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[John Chittum (jchittum)]

Verified Jammy:

Steps:

1. using bartender, built an image using the livecd-rootfs 2.765.41 code pulled from https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.41

bartender \
--hook-extras-branch jammy \
--livecd-rootfs-dir /home/jchittum/dev01/troubleshooting/various-rootfs/2.765.41/livecd-rootfs \
--build-provider aws \
--aws-keypair-name ${AWS_BUILD_KEY} \
--aws-profile image_builder \
-- \
--series jammy \
--project ubuntu-cpc \
--image-target qcow2

### NOTE ###
You don't have to provide hook-extras-branch build a qcow2. I just default to always using it...

2. untarred the contents
3. launched with QEMU
qemu-system-x86_64 \
-cpu host -machine type=q35,accel=kvm -m 2048 \
-nographic \
-snapshot \
-netdev id=net00,type=user,hostfwd=tcp::2222-:22 \
-device virtio-net-pci,netdev=net00 \
-drive if=virtio,format=qcow2,file=livecd.ubuntu-cpc.img \
-cdrom cloud_init.iso \
-bios /usr/share/OVMF/OVMF_CODE.fd

### NOTE ###
cloud-init.iso just adds my ssh key into the image.

4. logged in and ran `sudo snap debug seeding`
sudo snap debug seeding
seeded: true
preseeded: true
image-preseeding: 8.498s
seed-completion: 1.823s

5. and just to double check, yes there are snaps

snap list
Name Version Rev Tracking Publisher Notes
core20 20240111 2182 latest/stable canonical✓ base
lxd 5.0.3-babaaf8 27948 5.0/stable/… canonical✓ -
snapd 2.61.2 21184 latest/stable canonical✓ snapd

[Andreas Hasenack (ahasenack)]

Hi all, I'm looking at the focal upload in unapproved.

I see it has this additional change in live-build/functions that is not mentioned in d/changelog:
--- a/live-build/functions
+++ b/live-build/functions
@@ -726,13 +726,44 @@ snap_preseed() {

 snap_validate_seed() {
     local CHROOT_ROOT=$1
-
+ local kern_major_min=undefined
+ local boot_filename=undefined
+
+ # ppc64el still uses /boot/vmlinux so we need to determine the boot file name as non ppc64el use /boot/vmlinuz
+ # We don't need to query the arch as we can use existence of the file to determine the boot file name. Both
+ # will never be present at the same time.
+ if [ -e ${CHROOT_ROOT}/boot/vmlinuz ]; then
+ boot_filename=vmlinuz
+ elif [ -e ${CHROOT_ROOT}/boot/vmlinux ]; then
+ boot_filename=vmlinux
+ fi
+ if [ ${boot_filename} != undefined ]; then # we have a known boot file so we can proceed with checking for features to mount
+ kern_major_min=$(readlink --canonicalize --no-newline ${CHROOT_ROOT}/boot/${boot_filename} | grep --extended-regexp --only-matching --max-count 1 '[0-9]+\.[0-9]+')
+ if [ -d /usr/share/livecd-rootfs/live-build/apparmor/${kern_major_min} ]; then
+ # if an Ubuntu version has different kernel apparmor features between LTS and HWE kernels
+ # a snap pre-seeding issue can occur, where the incorrect apparmor features are reported
+ # basic copy of a directory structure overriding the "generic" feature set
+ # which is tied to the LTS kernel
+
+ # Bind kernel apparmor directory to feature directory for snap preseeding
+ umount "${CHROOT_ROOT}/sys/kernel/security/apparmor/features/"
+ mount --bind /usr/share/livecd-rootfs/live-build/apparmor/${kern_major_min} "${CHROOT_ROOT}/sys/kernel/security/apparmor/features/"
+ fi
+ fi
+
     if [ -e "${CHROOT_ROOT}/var/lib/snapd/seed/seed.yaml" ]; then
         snap debug validate-seed "${CHROOT_ROOT}/var/lib/snapd/seed/seed.yaml"
         /usr/lib/snapd/snap-preseed --reset $(realpath "${CHROOT_ROOT}")
         /usr/lib/snapd/snap-preseed $(realpath "${CHROOT_ROOT}")
         chroot "${CHROOT_ROOT}" apparmor_parser --skip-read-cache --write-cache --skip-kernel-load --verbose -j `nproc` /etc/apparmor.d
     fi
+
+ # Unmount kernel specific apparmor feature
+ # mount generic apparmor feature again (cleanup)
+ if [ -d /build/config/hooks.d/extra/apparmor/${kern_major_min} ]; then
+ umount "${CHROOT_ROOT}/sys/kernel/security/apparmor/features/"
+ mount -o bind /usr/share/livecd-rootfs/live-build/apparmor/generic "${CHROOT_ROOT}/sys/kernel/security/apparmor/features/"
+ fi
 }

 snap_from_seed() {

Looking at the jammy code changes and changelog, this seems to fix LP: #2038957):

      * Enable snap preseeding with ppc64el images where /boot/vmlinux is used
        instead of /boot/vmlinuz. (LP: #2038957)

That bug has a focal task, but it was marked as "won't fix" in https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/2038957/comments/4. Yet here we are with the change.

What do you want to do?
- include this change, and therefore...

[John Chittum (jchittum)]

The statement in the bug was correct -- we had not anticipated or thought an apparmor change would get backported to an LTS branch that would necessitate the backport of the functionality in `snap_validate_seed`. but now we have a break, where the HWE of focal (5.15) and LTS of jammy (5.15) got changed to include things.

What is really required is the functionality added to match on ${kern_major_min}, so it's a few commits / bugs deep.

goes back to the origin commit in ubuntu/jammy : bd1690bd16c70f9631ee2798514b51ed2dc973d5

which was never backported because there weren't going to be new kernel versions of 20.04 (5.15 was already out) and we didn't believe there'd be an addition to apparmor that'd break it:

https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/2015596

and follow up:

https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/2024639

So, i'd say, no, it's not about fixing LP #2038957 specifically, it's about how a kernel change caused us to require the functionality to special case on kernel version. Actually, the original bug doesn't say anything about focal: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2045384

so this may be a _side effect_. Let me ping that ticket quickly to see if it was intentional to release this to 20.04 5.15 (it's abnormal to put something in LTS Kernel and then _not_ in HWE of $PREVIOUS_SUITE, but it's not listed on the bug)

[Andreas Hasenack (ahasenack)]

An upload of livecd-rootfs to focal-proposed has been rejected from the upload queue for the following reason: "The focal upload contains undocumented changes to live-build/functions. After discussion, it was agreed that a new bug will be filed specifically for that change, and included in a new upload.".

[Catherine Redfield (catred)]

Patch for updating the changelog to cover the added function as well as the new apparmor directory

[Philip Roche (philroche)]

MP with catred's proposed changes @ https://code.launchpad.net/~philroche/livecd-rootfs/+git/livecd-rootfs/+merge/463488

[Philip Roche (philroche)]

Focal livecd-rootfs `2.664.53` with these proposed changes now in upload queue https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=livecd-rootfs

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.765.41

---------------
livecd-rootfs (2.765.41) jammy; urgency=medium

  [Jess Jang ]
  * add 5.15 apparmor directory for snap preseeding with 5.15 kernel
    (LP: #2052789)

 -- Philip Roche <email address hidden> Wed, 21 Feb 2024 11:31:50 +0000

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "LP2059730.patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Timo Aaltonen (tjaalton)]

Hello Philip, or anyone else affected,

Accepted livecd-rootfs into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.664.53 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Philip Roche (philroche)]

Verified Focal:

This exact proposed version of livecd-rootfs has been used in cloud image build for many weeks now and has met all of the steps listed in the test plan.

GCE daily minimal image daily-ubuntu-minimal-2004-focal-v20240405`` being one such image with snaps and built using this version of livecd-rootfs.

Steps:

1. Launch `daily-ubuntu-minimal-2004-focal-v20240405` from project `ubuntu-os-cloud-devel` in GCE

```
gcloud compute instances create $(petname) --zone=europe-west1-d --image=daily-ubuntu-minimal-2004-focal-v20240405 --image-project=ubuntu-os-cloud-devel
```

4. logged in and ran `sudo snap debug seeding`

```
ubuntu@usable-mullet:~$ sudo snap debug seeding
seeded: true
preseeded: true
image-preseeding: 5.744s
seed-completion: 3.278s
```

5. and just to double check, yes there are snaps

```
ubuntu@usable-mullet:~$ snap list
Name Version Rev Tracking Publisher Notes
core20 20240111 2182 latest/stable canonical✓ base
google-cloud-cli 471.0.0 229 latest/stable/… google-cloud-sdk✓ classic
snapd 2.61.2 21184 latest/stable canonical✓ snapd
```