apparmor features overrides for specific kernels never restored

[Thomas Bechtold]

Recent changes (commit bd1690bd16c70f9631, bug is https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/2015596) added a feature to livecd-rootfs to adjust the apparmor features for preseeding.

But CPC builds do use derivative images. In that case, it's
eg. possible that an image has kernel 5.19 installed (which would
trigger the modification of apparmor features on the filesystem) and a
derivate image (eg. the realtime kernel image) does then install a
5.15 kernel. In that case the modifications from the 5.19 kernel would
still be visible in /sys/kernel/security/apparmor/features/ and the
preseeding optimizations fail.

That's visible by calling "snap debug seeding":

# uname -a
Linux ip-172-31-28-91 5.15.0-1039-realtime #42-Ubuntu SMP PREEMPT_RT Wed May 17 20:46:21 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
# diff -u <(snap debug seeding|yq -P '.preseed-system-key') <(snap debug seeding|yq -P '.seed-restart-system-key')
--- /dev/fd/63 2023-06-22 07:55:00.603927264 +0000
+++ /dev/fd/62 2023-06-22 07:55:00.604927242 +0000
@@ -3,7 +3,6 @@
   - dbus
   - domain
   - file
- - ipc
   - mount
   - namespaces
   - network

[Impact]
For some images (eg. the realtime kernel images) the preseed optimizations do not work. That affects bootspeed

[Test Plan]
* Build images with the fix and boot the image
* make sure the output is:

# snap debug seeding
seeded: true
preseeded: true
image-preseeding: 13.682s
seed-completion: 4.684s

[Where problems could occur]
Snap seed may not be validated correctly and fail CPC automation test, which will block image publication

[Michael Hudson-Doyle (mwhudson)]

This bug still needs a "regression potential" / "where problems might occur" section.

[Łukasz Zemczak (sil2100)]

Hello Thomas, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.24 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

As part of the verification here I'd also like us to keep an eye out for other, non CPC images. This touches layered image generation and possibly preinstalled ones, so I'd like us to make sure we didn't cause regressions in the other parts of the world.

Can someone smoke test some non-CPC images while verifying this? Thanks.

[John Chittum (jchittum)]

due to the overlap between features, I've verified this with a CPC specific image. This is a copy/paste of the comment in https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/2031943 , as the verification steps, as far as getting the deb-source, building an image that uses a different kernel, and ensuring the directory is mounted and the proper features set.

This needs further validation of other cases (i'll put that in a separate comment)

Test case and instructions

1. downloaded deb-source of proposed-livecd-rootfs
    1a. lxc launch ubuntu:j
    1b. apt update && apt install dpkg-dev (to download the deb source)
    1c. enable `proposed` in sources.list
    1d. sudo apt-get source livecd-rootfs=2.765.24
    1e. exit container and lxc file pull source
2. using `ubuntu-bartender` from `ubuntu-old-fashioned`, run a build that requires the 6.2 kernel and does snap preseeding
    2a. in this case, built an project=ubuntu-cpc image-target=gce
        * NOTE: This requires cpc secret sauce to build. Unfortunately, there aren't any images in non-secret-sauce that use the HWE kernel.
3. Check for copying / mounting 6.2 kernel features in the code. log snippet:

+ snap_validate_seed mountpoint_gce
+ local CHROOT_ROOT=mountpoint_gce
+ '[' -e mountpoint_gce/boot/vmlinuz ']'
++ readlink --canonicalize --no-newline mountpoint_gce/boot/vmlinuz
++ grep --extended-regexp --only-matching --max-count 1 '[0-9]+\.[0-9]+'
+ local kern_major_min=6.2
+ '[' -d /usr/share/livecd-rootfs/live-build/apparmor/6.2 ']'
+ umount mountpoint_gce/sys/kernel/security/apparmor/features/
+ mount --bind /usr/share/livecd-rootfs/live-build/apparmor/6.2 mountpoint_gce/sys/kernel/security/apparmor/features/
+ '[' -e mountpoint_gce/var/lib/snapd/seed/seed.yaml ']'
+ snap debug validate-seed mountpoint_gce/var/lib/snapd/seed/seed.yaml
++ realpath mountpoint_gce
+ /usr/lib/snapd/snap-preseed --reset /build/mountpoint_gce
++ realpath mountpoint_gce
+ /usr/lib/snapd/snap-preseed /build/mountpoint_gce
ubuntu classic preseeding
starting to preseed root: /build/mountpoint_gce
using snapd binary: /tmp/snapd-preseed/usr/lib/snapd/snapd (2.59.5)
2023/08/21 21:25:46.286526 main.go:57: running for preseeding
2023/08/21 21:25:46.310274 overlord.go:272: Acquiring state lock file
2023/08/21 21:25:46.310292 overlord.go:277: Acquired state lock file
2023/08/21 21:25:46.354527 daemon.go:247: started snapd/2.59.5 (series 16; classic) ubuntu/22.04 (amd64) linux/5.15.0-1041-aws.
2023/08/21 21:25:46.370960 daemon.go:340: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
2023/08/21 21:25:46.382935 backends.go:58: AppArmor status: apparmor is enabled and all features are available
2023/08/21 21:25:46.889313 helpers.go:146: error trying to compare the snap system key: system-key missing on disk
2023/08/21 21:25:58.334026 daemon.go:428: stopping snapd as requested
2023/08/21 21:25:58.349910 daemon.go:521: gracefully waiting for running hooks
2023/08/21 21:25:58.349930 daemon.go:523: done waiting for running hooks
2023/08/21 21:25:58.350457 overlord.go:516: Released state lock file
unmounting: /tmp/snapd-preseed

4. register the image in GCE
    4a. push resultant buil...

[John Chittum (jchittum)]

cases that still need verified

* something that moves "back and forth" during a build -- CPC has an example of a cloud-image that it's "base" is the HWE, but the variant must use the LTS kernel. That needs to be tested
* something _not_ ubuntu-cpc project. I'm not sure what will need built there, as i'm not sure what other images have hwe kernels / build pre-installed with snap preseeding.

[Jess Jang (jessica-youjeong)]

Tested proposed livecd-rootfs with Realtime image AWS which kernel base is HWE but RT kernel version is 5.15 (LTS)

1. Building image with `ubuntu-bartender` from `ubuntu-old-fashioned`
    1a. project=ubuntu-cpc image-target=ec2-pro-realtime

2. Log file check
+ snap_validate_seed /tmp/tmp.ymGEaZ1Bbc
+ local CHROOT_ROOT=/tmp/tmp.ymGEaZ1Bbc
+ '[' -e /tmp/tmp.ymGEaZ1Bbc/boot/vmlinuz ']'
++ readlink --canonicalize --no-newline /tmp/tmp.ymGEaZ1Bbc/boot/vmlinuz
++ grep --extended-regexp --only-matching --max-count 1 '[0-9]+\.[0-9]+'
+ local kern_major_min=5.15
+ '[' -d /usr/share/livecd-rootfs/live-build/apparmor/5.15 ']'
+ '[' -e /tmp/tmp.ymGEaZ1Bbc/var/lib/snapd/seed/seed.yaml ']'
+ snap debug validate-seed /tmp/tmp.ymGEaZ1Bbc/var/lib/snapd/seed/seed.yaml
++ realpath /tmp/tmp.ymGEaZ1Bbc
+ /usr/lib/snapd/snap-preseed --reset /tmp/tmp.ymGEaZ1Bbc
++ realpath /tmp/tmp.ymGEaZ1Bbc
+ /usr/lib/snapd/snap-preseed /tmp/tmp.ymGEaZ1Bbc
ubuntu classic preseeding
starting to preseed root: /tmp/tmp.ymGEaZ1Bbc
using snapd binary: /tmp/snapd-preseed/usr/lib/snapd/snapd (2.59.5)
2023/08/22 13:36:56.513867 main.go:57: running for preseeding
2023/08/22 13:36:56.541899 overlord.go:272: Acquiring state lock file
2023/08/22 13:36:56.541919 overlord.go:277: Acquired state lock file
2023/08/22 13:36:56.594654 daemon.go:247: started snapd/2.59.5 (series 16; classic) ubuntu/22.04 (amd64) linux/5.15.0-1041-aws.
2023/08/22 13:36:56.610803 daemon.go:340: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
2023/08/22 13:36:56.624055 backends.go:58: AppArmor status: apparmor is enabled and all features are available
2023/08/22 13:36:57.153210 helpers.go:146: error trying to compare the snap system key: system-key missing on disk
2023/08/22 13:37:06.764204 daemon.go:428: stopping snapd as requested
2023/08/22 13:37:06.779818 daemon.go:521: gracefully waiting for running hooks
2023/08/22 13:37:06.779836 daemon.go:523: done waiting for running hooks
2023/08/22 13:37:06.780356 overlord.go:516: Released state lock file
unmounting: /tmp/snapd-preseed

3. register image and check kernel&snap debug seeding result

ubuntu@ip-172-31-26-219:~$ snap debug seeding
seeded: true
preseeded: true
image-preseeding: 8.3s
seed-completion: 3720h29m46.734s
ubuntu@ip-172-31-26-219:~$ uname -a
Linux ip-172-31-26-219 5.15.0-1043-realtime #48-Ubuntu SMP PREEMPT_RT Thu Jul 13 20:54:21 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

This verification shows snap seeds are validated with correct kernel version

[Utkarsh Gupta (utkarsh)]

Hi,

> Can someone smoke test some non-CPC images while verifying this? Thanks.

I've tested a server image built here:
https://launchpad.net/~ubuntu-cdimage/+livefs/ubuntu/jammy/ubuntu-server-live/

Steve kicked off the builds from -proposed on 23-08-2023 and I smoke tested livecd.ubuntu-server.kernel-generic-hwe and livecd.ubuntu-server.initrd-generic-hwe to ensure it works. Of course I didn't do a through test, that is, extensive test but I think a simple smoke-test should suffice here.

Thanks,
Utkarsh

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.765.24

---------------
livecd-rootfs (2.765.24) jammy; urgency=medium

  [ Jess Jang ]
  * fix: bind correct apparmor feature for validating snap seed.
    (LP: #2024639)

  [ John Chittum ]
  * fix: 6.2 kernel missing feature. (LP: #2031943)

livecd-rootfs (2.765.23) jammy; urgency=medium

  [ Loïc Minier ]
  * Initial support for NVIDIA Tegra (LP: #2015644).

livecd-rootfs (2.765.22) jammy; urgency=medium

  * Drop use of --removable flag to grub-install from
    live-build/buildd/hooks/02-disk-image-uefi.binary, to match the cloud
    images (7c760864fdcb278ca37396f06f5e3f297428d63d). This fixes
    bootloader updates in the buildd images, but also fixes compatibility
    with using devtmpfs for losetup.

livecd-rootfs (2.765.21) jammy; urgency=medium

  [ jchittum ]
  * add multiple kernel apparmor feature sets for snap preseeding
    (LP: #2015596).

 -- Utkarsh Gupta <email address hidden> Sat, 19 Aug 2023 03:24:34 +0530

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.