6.2 Kernel Apparmor Feature Mismatch on Jammy

[John Chittum]

Ubuntu Jammy 22.04 rolled the HWE kernel to 6.2. This has run into the same issue as the 5.19 kernel roll, where the generic apparmor feature set being mounted into the build env does not match the 6.2 kernel. The same feature is missing

ipc

Observable by building an image, booting it, and running `snap debug preseeding`. Example output from a GCE image:

sudo snap debug seeding
seeded: true
preseeded: true
image-preseeding: 7.411s
seed-completion: 13.899s
preseed-system-key: {
  "apparmor-features": [
    "caps",
    "dbus",
    "domain",
    "file",
    "mount",
    "namespaces",
    "network",
    "network_v8",
    "policy",
    "ptrace",
    "query",
    "rlimit",
    "signal"
  ],
  "apparmor-parser-features": [
    "cap-audit-read",
    "cap-bpf",
    "include-if-exists",
    "mqueue",
    "qipcrtr-socket",
    "unsafe",
    "xdp"
  ],
  "apparmor-parser-mtime": 1666191120,
  "build-id": "4399dbaff2bd50a4ae62c9324bbc96def18bf9a8",
  "cgroup-version": "2",
  "nfs-home": false,
  "overlay-root": "",
  "seccomp-compiler-version": "0f51c115b973679d7ed64b550a8d9d461f7cd5c6 2.5.4 c3c9b282ef3c8dfcc3124b2aeaef62f56b813bfd21f8806b30a6c9dbc2e6e58d bpf-actlog",
  "seccomp-features": [
    "allow",
    "errno",
    "kill_process",
    "kill_thread",
    "log",
    "trace",
    "trap",
    "user_notif"
  ],
  "version": 10
}
seed-restart-system-key: {
  "apparmor-features": [
    "caps",
    "dbus",
    "domain",
    "file",
    "ipc",
    "mount",
    "namespaces",
    "network",
    "network_v8",
    "policy",
    "ptrace",
    "query",
    "rlimit",
    "signal"
  ],
  "apparmor-parser-features": [
    "cap-audit-read",
    "cap-bpf",
    "include-if-exists",
    "mqueue",
    "qipcrtr-socket",
    "unsafe",
    "xdp"
  ],
  "apparmor-parser-mtime": 1666191120,
  "build-id": "4399dbaff2bd50a4ae62c9324bbc96def18bf9a8",
  "cgroup-version": "2",
  "nfs-home": false,
  "overlay-root": "",
  "seccomp-compiler-version": "0f51c115b973679d7ed64b550a8d9d461f7cd5c6 2.5.4 c3c9b282ef3c8dfcc3124b2aeaef62f56b813bfd21f8806b30a6c9dbc2e6e58d bpf-actlog",
  "seccomp-features": [
    "allow",
    "errno",
    "kill_process",
    "kill_thread",
    "log",
    "trace",
    "trap",
    "user_notif"
  ],
  "version": 10
}

[ Impact ]

 * boot times for all pre-installed images are adversely affected, causing issues for cloud partners

[ Test Plan ]

 * build any cloud image
 * boot
 * run `snap debug preseed`
 * there should not be anything in : seed-restart-system-key

[ Where problems could occur ]

 * if i didn't get the exact match of features, and seed-restart-system-key is still present
    * thankfully, it tells you the mismatch, so we can iterate and fix
 * without this the 6.2 kernel will not be properly pre-seeded and optimized for first boot deployments.

[ Other Info ]

[Łukasz Zemczak (sil2100)]

Hello John, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.24 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[John Chittum (jchittum)]

Verification -- build an image with a 6.2 kernel, and ensure that snap preseed operates as expected.

note: the package in proposed has 2 commits related to snap preseeding -- the one related to this Bug, and https://bugs.launchpad.net/ubuntu/+source/livecd-rootfs/+bug/2024639 . Can't really separate them fully, so i'll comment on both bugs with this specific test case

Test case and instructions

1. downloaded deb-source of proposed-livecd-rootfs
    1a. lxc launch ubuntu:j
    1b. apt update && apt install dpkg-dev (to download the deb source)
    1c. enable `proposed` in sources.list
    1d. sudo apt-get source livecd-rootfs=2.765.24
    1e. exit container and lxc file pull source
2. using `ubuntu-bartender` from `ubuntu-old-fashioned`, run a build that requires the 6.2 kernel and does snap preseeding
    2a. in this case, built an project=ubuntu-cpc image-target=gce
        * NOTE: This requires cpc secret sauce to build. Unfortunately, there aren't any images in non-secret-sauce that use the HWE kernel.
3. Check for copying / mounting 6.2 kernel features in the code. log snippet:

+ snap_validate_seed mountpoint_gce
+ local CHROOT_ROOT=mountpoint_gce
+ '[' -e mountpoint_gce/boot/vmlinuz ']'
++ readlink --canonicalize --no-newline mountpoint_gce/boot/vmlinuz
++ grep --extended-regexp --only-matching --max-count 1 '[0-9]+\.[0-9]+'
+ local kern_major_min=6.2
+ '[' -d /usr/share/livecd-rootfs/live-build/apparmor/6.2 ']'
+ umount mountpoint_gce/sys/kernel/security/apparmor/features/
+ mount --bind /usr/share/livecd-rootfs/live-build/apparmor/6.2 mountpoint_gce/sys/kernel/security/apparmor/features/
+ '[' -e mountpoint_gce/var/lib/snapd/seed/seed.yaml ']'
+ snap debug validate-seed mountpoint_gce/var/lib/snapd/seed/seed.yaml
++ realpath mountpoint_gce
+ /usr/lib/snapd/snap-preseed --reset /build/mountpoint_gce
++ realpath mountpoint_gce
+ /usr/lib/snapd/snap-preseed /build/mountpoint_gce
ubuntu classic preseeding
starting to preseed root: /build/mountpoint_gce
using snapd binary: /tmp/snapd-preseed/usr/lib/snapd/snapd (2.59.5)
2023/08/21 21:25:46.286526 main.go:57: running for preseeding
2023/08/21 21:25:46.310274 overlord.go:272: Acquiring state lock file
2023/08/21 21:25:46.310292 overlord.go:277: Acquired state lock file
2023/08/21 21:25:46.354527 daemon.go:247: started snapd/2.59.5 (series 16; classic) ubuntu/22.04 (amd64) linux/5.15.0-1041-aws.
2023/08/21 21:25:46.370960 daemon.go:340: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
2023/08/21 21:25:46.382935 backends.go:58: AppArmor status: apparmor is enabled and all features are available
2023/08/21 21:25:46.889313 helpers.go:146: error trying to compare the snap system key: system-key missing on disk
2023/08/21 21:25:58.334026 daemon.go:428: stopping snapd as requested
2023/08/21 21:25:58.349910 daemon.go:521: gracefully waiting for running hooks
2023/08/21 21:25:58.349930 daemon.go:523: done waiting for running hooks
2023/08/21 21:25:58.350457 overlord.go:516: Released state lock file
unmounting: /tmp/snapd-preseed

4. register the image in GCE
    4a. push resultant builds tar.gz to a bucket
    4b. register an image by pointing to the file in the buc...

[Jess Jang (jessica-youjeong)]

Same as above John's comment.
Tested proposed livecd-rootfs with Realtime image AWS which kernel version is 5.15

1. Building image with `ubuntu-bartender` from `ubuntu-old-fashioned`
    1a. project=ubuntu-cpc image-target=ec2-pro-realtime

2. Log file check
+ snap_validate_seed /tmp/tmp.ymGEaZ1Bbc
+ local CHROOT_ROOT=/tmp/tmp.ymGEaZ1Bbc
+ '[' -e /tmp/tmp.ymGEaZ1Bbc/boot/vmlinuz ']'
++ readlink --canonicalize --no-newline /tmp/tmp.ymGEaZ1Bbc/boot/vmlinuz
++ grep --extended-regexp --only-matching --max-count 1 '[0-9]+\.[0-9]+'
+ local kern_major_min=5.15
+ '[' -d /usr/share/livecd-rootfs/live-build/apparmor/5.15 ']'
+ '[' -e /tmp/tmp.ymGEaZ1Bbc/var/lib/snapd/seed/seed.yaml ']'
+ snap debug validate-seed /tmp/tmp.ymGEaZ1Bbc/var/lib/snapd/seed/seed.yaml
++ realpath /tmp/tmp.ymGEaZ1Bbc
+ /usr/lib/snapd/snap-preseed --reset /tmp/tmp.ymGEaZ1Bbc
++ realpath /tmp/tmp.ymGEaZ1Bbc
+ /usr/lib/snapd/snap-preseed /tmp/tmp.ymGEaZ1Bbc
ubuntu classic preseeding
starting to preseed root: /tmp/tmp.ymGEaZ1Bbc
using snapd binary: /tmp/snapd-preseed/usr/lib/snapd/snapd (2.59.5)
2023/08/22 13:36:56.513867 main.go:57: running for preseeding
2023/08/22 13:36:56.541899 overlord.go:272: Acquiring state lock file
2023/08/22 13:36:56.541919 overlord.go:277: Acquired state lock file
2023/08/22 13:36:56.594654 daemon.go:247: started snapd/2.59.5 (series 16; classic) ubuntu/22.04 (amd64) linux/5.15.0-1041-aws.
2023/08/22 13:36:56.610803 daemon.go:340: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
2023/08/22 13:36:56.624055 backends.go:58: AppArmor status: apparmor is enabled and all features are available
2023/08/22 13:36:57.153210 helpers.go:146: error trying to compare the snap system key: system-key missing on disk
2023/08/22 13:37:06.764204 daemon.go:428: stopping snapd as requested
2023/08/22 13:37:06.779818 daemon.go:521: gracefully waiting for running hooks
2023/08/22 13:37:06.779836 daemon.go:523: done waiting for running hooks
2023/08/22 13:37:06.780356 overlord.go:516: Released state lock file
unmounting: /tmp/snapd-preseed

3. register image and check kernel&snap debug seeding result

ubuntu@ip-172-31-26-219:~$ snap debug seeding
seeded: true
preseeded: true
image-preseeding: 8.3s
seed-completion: 3720h29m46.734s
ubuntu@ip-172-31-26-219:~$ uname -a
Linux ip-172-31-26-219 5.15.0-1043-realtime #48-Ubuntu SMP PREEMPT_RT Thu Jul 13 20:54:21 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

This verification shows snap seeds are validated with correct kernel version

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.765.24

---------------
livecd-rootfs (2.765.24) jammy; urgency=medium

  [ Jess Jang ]
  * fix: bind correct apparmor feature for validating snap seed.
    (LP: #2024639)

  [ John Chittum ]
  * fix: 6.2 kernel missing feature. (LP: #2031943)

livecd-rootfs (2.765.23) jammy; urgency=medium

  [ Loïc Minier ]
  * Initial support for NVIDIA Tegra (LP: #2015644).

livecd-rootfs (2.765.22) jammy; urgency=medium

  * Drop use of --removable flag to grub-install from
    live-build/buildd/hooks/02-disk-image-uefi.binary, to match the cloud
    images (7c760864fdcb278ca37396f06f5e3f297428d63d). This fixes
    bootloader updates in the buildd images, but also fixes compatibility
    with using devtmpfs for losetup.

livecd-rootfs (2.765.21) jammy; urgency=medium

  [ jchittum ]
  * add multiple kernel apparmor feature sets for snap preseeding
    (LP: #2015596).

 -- Utkarsh Gupta <email address hidden> Sat, 19 Aug 2023 03:24:34 +0530

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Andreas Hasenack (ahasenack)]

Why is this fix-released in jammy, but not in the devel task?