Activity log for bug #2016908

Date Who What changed Old value New value Message
2023-04-18 16:16:38 Francis Ginther bug added bug
2023-04-18 16:16:38 Francis Ginther attachment added scobee-lunar-no-squashfs.log https://bugs.launchpad.net/bugs/2016908/+attachment/5665037/+files/scobee-lunar-no-squashfs.log
2023-04-19 09:47:20 Dimitri John Ledkov maas-images: status New Incomplete
2023-04-19 09:47:26 Dimitri John Ledkov bug added subscriber Dimitri John Ledkov
2023-04-19 12:50:53 Francis Ginther attachment added fili-0319.log https://bugs.launchpad.net/maas-images/+bug/2016908/+attachment/5665216/+files/fili-0319.log
2023-04-19 12:52:21 Francis Ginther attachment added fili-0419.log https://bugs.launchpad.net/maas-images/+bug/2016908/+attachment/5665217/+files/fili-0419.log
2023-04-19 14:28:43 Alexsander de Souza bug added subscriber Alexsander de Souza
2023-04-19 14:50:09 Alexsander de Souza attachment added Working boot log from LXD VM https://bugs.launchpad.net/maas-images/+bug/2016908/+attachment/5665260/+files/boot_jammy.log
2023-04-19 14:50:46 Alexsander de Souza attachment added Non-working boot log from Lunar on LXD VM https://bugs.launchpad.net/maas-images/+bug/2016908/+attachment/5665261/+files/boot_lunar.log
2023-04-19 17:15:09 Francis Ginther attachment added udev-debug-0319.log https://bugs.launchpad.net/maas-images/+bug/2016908/+attachment/5665268/+files/udev-debug-0319.log
2023-04-19 17:15:59 Francis Ginther attachment added udev-debug-0418.log https://bugs.launchpad.net/maas-images/+bug/2016908/+attachment/5665292/+files/udev-debug-0418.log
2023-04-19 22:10:49 Alexsander de Souza bug task added systemd (Ubuntu)
2023-04-19 22:15:20 Alexsander de Souza attachment added lunar-udev-debug.log https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2016908/+attachment/5665324/+files/lunar-udev-debug.log
2023-04-20 18:48:46 Dimitri John Ledkov bug task added linux (Ubuntu)
2023-04-20 19:00:09 Ubuntu Kernel Bot linux (Ubuntu): status New Incomplete
2023-04-20 19:41:35 Dimitri John Ledkov description I'm assuming the image being used for these deploys is 20230417 or 20230417.1 based on the fact that I saw a 6.2 kernel being used which I don't believe was part of the 20230319 serial. I don't have access to the maas server, so I can't directly check any log files. MAAS Version: 3.3.2 Here's where the serial log indicates it can't download the squashfs. The full log is attached as scobee-lunar-no-squashfs.log (there are some other console message intermixed): no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf /run/net6 -*.conf :: root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity =yes shfs :: mount_squash downloading http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0 4/lunar/candidate/squashfs to /root.tmp.img Connecting to 10.229.32.21:5248 (10.229.32.21:5248) wget: can't connect to remote host (10.229.32.21): Network is unreachable :: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp' mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory done. Still gathering logs and info and will update as I go. I'm assuming the image being used for these deploys is 20230417 or 20230417.1 based on the fact that I saw a 6.2 kernel being used which I don't believe was part of the 20230319 serial. I don't have access to the maas server, so I can't directly check any log files. MAAS Version: 3.3.2 Here's where the serial log indicates it can't download the squashfs. The full log is attached as scobee-lunar-no-squashfs.log (there are some other console message intermixed): no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf /run/net6 -*.conf :: root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity =yes shfs :: mount_squash downloading http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0 4/lunar/candidate/squashfs to /root.tmp.img Connecting to 10.229.32.21:5248 (10.229.32.21:5248) wget: can't connect to remote host (10.229.32.21): Network is unreachable :: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp' mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory done. Still gathering logs and info and will update as I go. ---- Kernel Bug / Apparmor reproducer $ wget https://images.maas.io/ephemeral-v3/candidate/lunar/amd64/20230419/ga-23.04/generic/boot-kernel $ wget https://images.maas.io/ephemeral-v3/candidate/lunar/amd64/20230419/ga-23.04/generic/boot-initrd $ qemu-system-x86_64 -nographic -m 2G -kernel ./boot-kernel -initrd ./boot-initrd -append 'console=ttyS0 break=modules apparmor=0' #start the VM .... Starting systemd-udevd version 252.5-2ubuntu3 Spawning shell within the initramfs BusyBox v1.35.0 (Ubuntu 1:1.35.0-4ubuntu1) built-in shell (ash) Enter 'help' for a list of built-in commands. (initramfs) udevadm info --export-db Failed to set death signal: Invalid argument Observe that udevadm fails to setup death signal, with in systemd code is this https://github.com/systemd/systemd/blob/08c2f9c626e0f0052d505b1b7e52f335c0fbfa1d/src/basic/process-util.c#L1252 if (flags & (FORK_DEATHSIG|FORK_DEATHSIG_SIGINT)) if (prctl(PR_SET_PDEATHSIG, (flags & FORK_DEATHSIG_SIGINT) ? SIGINT : SIGTERM) < 0) { log_full_errno(prio, errno, "Failed to set death signal: %m"); _exit(EXIT_FAILURE); } ---- MAAS bug Why is maas setting `apparmor=0` ? Ubuntu shouldn't be used without apparmor. Even for deployment and commisioning.
2023-04-20 19:41:44 Dimitri John Ledkov linux (Ubuntu): status Incomplete Triaged
2023-04-20 19:41:48 Dimitri John Ledkov maas-images: status Incomplete Invalid
2023-04-20 19:41:51 Dimitri John Ledkov systemd (Ubuntu): status New Invalid
2023-04-20 19:41:59 Dimitri John Ledkov bug task added maas
2023-04-20 19:42:25 Dimitri John Ledkov summary Unable to deploy hosts with lunar images after 20230319 - fails to connect and download squashfs udev fails to make prctl() syscall with apparmor=0 (as used by maas by default)
2023-04-20 19:43:05 Dimitri John Ledkov description I'm assuming the image being used for these deploys is 20230417 or 20230417.1 based on the fact that I saw a 6.2 kernel being used which I don't believe was part of the 20230319 serial. I don't have access to the maas server, so I can't directly check any log files. MAAS Version: 3.3.2 Here's where the serial log indicates it can't download the squashfs. The full log is attached as scobee-lunar-no-squashfs.log (there are some other console message intermixed): no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf /run/net6 -*.conf :: root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity =yes shfs :: mount_squash downloading http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0 4/lunar/candidate/squashfs to /root.tmp.img Connecting to 10.229.32.21:5248 (10.229.32.21:5248) wget: can't connect to remote host (10.229.32.21): Network is unreachable :: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp' mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory done. Still gathering logs and info and will update as I go. ---- Kernel Bug / Apparmor reproducer $ wget https://images.maas.io/ephemeral-v3/candidate/lunar/amd64/20230419/ga-23.04/generic/boot-kernel $ wget https://images.maas.io/ephemeral-v3/candidate/lunar/amd64/20230419/ga-23.04/generic/boot-initrd $ qemu-system-x86_64 -nographic -m 2G -kernel ./boot-kernel -initrd ./boot-initrd -append 'console=ttyS0 break=modules apparmor=0' #start the VM .... Starting systemd-udevd version 252.5-2ubuntu3 Spawning shell within the initramfs BusyBox v1.35.0 (Ubuntu 1:1.35.0-4ubuntu1) built-in shell (ash) Enter 'help' for a list of built-in commands. (initramfs) udevadm info --export-db Failed to set death signal: Invalid argument Observe that udevadm fails to setup death signal, with in systemd code is this https://github.com/systemd/systemd/blob/08c2f9c626e0f0052d505b1b7e52f335c0fbfa1d/src/basic/process-util.c#L1252 if (flags & (FORK_DEATHSIG|FORK_DEATHSIG_SIGINT)) if (prctl(PR_SET_PDEATHSIG, (flags & FORK_DEATHSIG_SIGINT) ? SIGINT : SIGTERM) < 0) { log_full_errno(prio, errno, "Failed to set death signal: %m"); _exit(EXIT_FAILURE); } ---- MAAS bug Why is maas setting `apparmor=0` ? Ubuntu shouldn't be used without apparmor. Even for deployment and commisioning. I'm assuming the image being used for these deploys is 20230417 or 20230417.1 based on the fact that I saw a 6.2 kernel being used which I don't believe was part of the 20230319 serial. I don't have access to the maas server, so I can't directly check any log files. MAAS Version: 3.3.2 Here's where the serial log indicates it can't download the squashfs. The full log is attached as scobee-lunar-no-squashfs.log (there are some other console message intermixed): no search or nameservers found in /run/net-BOOTIF.conf /run/net-*.conf /run/net6 -*.conf :: root=squash:http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.04/lunar/candi date/squa[ 206.804704] Btrfs loaded, crc32c=crc32c-generic, zoned=yes, fsverity =yes shfs :: mount_squash downloading http://10.229.32.21:5248/images/ubuntu/arm64/ga-23.0 4/lunar/candidate/squashfs to /root.tmp.img Connecting to 10.229.32.21:5248 (10.229.32.21:5248) wget: can't connect to remote host (10.229.32.21): Network is unreachable :: mount -t squashfs -o loop '/root.tmp.img' '/root.tmp' mount: mounting /root.tmp.img on /root.tmp failed: No such file or directory done. Still gathering logs and info and will update as I go. ---- Kernel Bug / Apparmor reproducer $ wget https://images.maas.io/ephemeral-v3/candidate/lunar/amd64/20230419/ga-23.04/generic/boot-kernel $ wget https://images.maas.io/ephemeral-v3/candidate/lunar/amd64/20230419/ga-23.04/generic/boot-initrd $ qemu-system-x86_64 -nographic -m 2G -kernel ./boot-kernel -initrd ./boot-initrd -append 'console=ttyS0 break=modules apparmor=0' #start the VM .... Starting systemd-udevd version 252.5-2ubuntu3 Spawning shell within the initramfs BusyBox v1.35.0 (Ubuntu 1:1.35.0-4ubuntu1) built-in shell (ash) Enter 'help' for a list of built-in commands. (initramfs) udevadm info --export-db Failed to set death signal: Invalid argument Observe that udevadm fails to setup death signal, with in systemd code is this https://github.com/systemd/systemd/blob/08c2f9c626e0f0052d505b1b7e52f335c0fbfa1d/src/basic/process-util.c#L1252         if (flags & (FORK_DEATHSIG|FORK_DEATHSIG_SIGINT))                 if (prctl(PR_SET_PDEATHSIG, (flags & FORK_DEATHSIG_SIGINT) ? SIGINT : SIGTERM) < 0) {                         log_full_errno(prio, errno, "Failed to set death signal: %m");                         _exit(EXIT_FAILURE);                 } workaround set kernel commandline to `apparmor=1` ---- MAAS bug Why is maas setting `apparmor=0` ? Ubuntu shouldn't be used without apparmor. Even for deployment and commisioning.
2023-04-20 20:15:11 Alexsander de Souza maas: status New Triaged
2023-04-20 20:15:15 Alexsander de Souza maas: importance Undecided High
2023-04-20 20:15:21 Alexsander de Souza maas: milestone 3.4.0
2023-04-20 20:15:34 Alexsander de Souza maas: assignee Alexsander de Souza (alexsander-souza)
2023-04-21 10:11:19 Dimitri John Ledkov bug task added apparmor
2023-04-21 10:11:27 Dimitri John Ledkov bug task added apparmor (Ubuntu)
2023-04-28 08:31:05 John Johansen attachment added 0001-UBUNTU-SAUCE-no-up-Stacking-v38-Fix-prctl-syscall-wi.patch https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/2016908/+attachment/5669359/+files/0001-UBUNTU-SAUCE-no-up-Stacking-v38-Fix-prctl-syscall-wi.patch
2023-04-28 09:17:22 Stefan Bader linux (Ubuntu): importance Undecided Medium
2023-04-28 09:17:33 Stefan Bader nominated for series Ubuntu Lunar
2023-04-28 09:17:33 Stefan Bader bug task added apparmor (Ubuntu Lunar)
2023-04-28 09:17:33 Stefan Bader bug task added linux (Ubuntu Lunar)
2023-04-28 09:17:33 Stefan Bader bug task added systemd (Ubuntu Lunar)
2023-04-28 09:17:55 Stefan Bader linux (Ubuntu Lunar): importance Undecided Medium
2023-04-28 09:17:55 Stefan Bader linux (Ubuntu Lunar): status New In Progress
2023-04-28 09:18:06 Stefan Bader linux (Ubuntu): status Triaged Fix Committed
2023-04-28 12:17:53 Ubuntu Foundations Team Bug Bot tags patch
2023-04-28 12:17:54 Ubuntu Foundations Team Bug Bot bug added subscriber Terry Rudd
2023-04-28 12:18:07 Ubuntu Foundations Team Bug Bot bug added subscriber Ubuntu Review Team
2023-05-12 09:04:58 Stefan Bader linux (Ubuntu Lunar): status In Progress Fix Committed
2023-05-19 13:53:47 Alexsander de Souza maas: status Triaged In Progress
2023-05-19 23:44:36 Alexsander de Souza merge proposal linked https://code.launchpad.net/~alexsander-souza/maas/+git/maas/+merge/443301
2023-05-22 14:27:24 MAAS Lander maas: status In Progress Fix Committed
2023-05-25 07:24:41 Alberto Donato maas: milestone 3.4.0 3.4.0-beta2
2023-06-07 21:42:30 Launchpad Janitor apparmor (Ubuntu): status New Confirmed
2023-06-07 21:42:30 Launchpad Janitor apparmor (Ubuntu Lunar): status New Confirmed
2023-06-07 21:42:30 Launchpad Janitor systemd (Ubuntu Lunar): status New Confirmed
2023-06-08 13:58:29 Nick Rosbrook systemd (Ubuntu Lunar): status Confirmed Invalid
2023-06-08 18:41:33 Ubuntu Kernel Bot tags patch kernel-spammed-jammy-linux-hwe-6.2 patch verification-needed-jammy
2023-06-09 20:55:37 Ubuntu Kernel Bot tags kernel-spammed-jammy-linux-hwe-6.2 patch verification-needed-jammy kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-6.2 patch verification-needed-jammy
2023-06-09 21:09:10 Alexsander de Souza tags kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-6.2 patch verification-needed-jammy kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-6.2 patch verification-done-jammy
2023-06-15 22:21:51 Launchpad Janitor linux (Ubuntu Lunar): status Fix Committed Fix Released
2023-06-15 22:21:51 Launchpad Janitor cve linked 2023-1380
2023-06-15 22:21:51 Launchpad Janitor cve linked 2023-2612
2023-06-15 22:21:51 Launchpad Janitor cve linked 2023-30456
2023-06-15 22:21:51 Launchpad Janitor cve linked 2023-31436
2023-06-15 22:21:51 Launchpad Janitor cve linked 2023-32233
2023-06-22 09:09:26 John Johansen apparmor: status New Fix Committed
2023-06-22 15:45:12 Launchpad Janitor linux (Ubuntu): status Fix Committed Fix Released
2023-07-06 09:51:58 Nobuto Murata bug added subscriber Nobuto Murata
2023-07-19 13:14:47 Ubuntu Kernel Bot tags kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-6.2 patch verification-done-jammy kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-lunar-linux-azure patch verification-done-jammy verification-needed-lunar
2023-08-10 12:10:59 Ubuntu Kernel Bot tags kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-lunar-linux-azure patch verification-done-jammy verification-needed-lunar kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-jammy-linux-oem-6.5-v2 kernel-spammed-lunar-linux-azure patch verification-done-jammy verification-needed-jammy-linux-oem-6.5 verification-needed-lunar
2023-10-12 12:01:17 Dimitri John Ledkov apparmor (Ubuntu): status Confirmed Invalid
2023-10-12 12:01:20 Dimitri John Ledkov apparmor (Ubuntu Lunar): status Confirmed Invalid
2023-10-12 12:01:23 Dimitri John Ledkov maas: status Fix Committed Fix Released
2023-10-12 12:01:25 Dimitri John Ledkov apparmor: status Fix Committed Fix Released
2023-10-24 16:46:42 Ubuntu Kernel Bot tags kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-jammy-linux-oem-6.5-v2 kernel-spammed-lunar-linux-azure patch verification-done-jammy verification-needed-jammy-linux-oem-6.5 verification-needed-lunar kernel-spammed-jammy-linux-azure-6.5-v2 kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-jammy-linux-oem-6.5-v2 kernel-spammed-lunar-linux-azure patch verification-done-jammy verification-needed-jammy-linux-azure-6.5 verification-needed-jammy-linux-oem-6.5 verification-needed-lunar
2023-10-24 18:21:45 Ubuntu Kernel Bot tags kernel-spammed-jammy-linux-azure-6.5-v2 kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-jammy-linux-oem-6.5-v2 kernel-spammed-lunar-linux-azure patch verification-done-jammy verification-needed-jammy-linux-azure-6.5 verification-needed-jammy-linux-oem-6.5 verification-needed-lunar kernel-spammed-jammy-linux-aws-6.5-v2 kernel-spammed-jammy-linux-azure-6.5-v2 kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-jammy-linux-oem-6.5-v2 kernel-spammed-lunar-linux-azure patch verification-done-jammy verification-needed-jammy-linux-aws-6.5 verification-needed-jammy-linux-azure-6.5 verification-needed-jammy-linux-oem-6.5 verification-needed-lunar
2023-10-31 23:45:59 Ubuntu Kernel Bot tags kernel-spammed-jammy-linux-aws-6.5-v2 kernel-spammed-jammy-linux-azure-6.5-v2 kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-jammy-linux-oem-6.5-v2 kernel-spammed-lunar-linux-azure patch verification-done-jammy verification-needed-jammy-linux-aws-6.5 verification-needed-jammy-linux-azure-6.5 verification-needed-jammy-linux-oem-6.5 verification-needed-lunar kernel-spammed-jammy-linux-aws-6.5-v2 kernel-spammed-jammy-linux-azure-6.5-v2 kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-jammy-linux-nvidia-6.5-v2 kernel-spammed-jammy-linux-oem-6.5-v2 kernel-spammed-lunar-linux-azure patch verification-done-jammy verification-needed-jammy-linux-aws-6.5 verification-needed-jammy-linux-azure-6.5 verification-needed-jammy-linux-nvidia-6.5 verification-needed-jammy-linux-oem-6.5 verification-needed-lunar
2023-11-27 12:16:54 Ubuntu Kernel Bot tags kernel-spammed-jammy-linux-aws-6.5-v2 kernel-spammed-jammy-linux-azure-6.5-v2 kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-jammy-linux-nvidia-6.5-v2 kernel-spammed-jammy-linux-oem-6.5-v2 kernel-spammed-lunar-linux-azure patch verification-done-jammy verification-needed-jammy-linux-aws-6.5 verification-needed-jammy-linux-azure-6.5 verification-needed-jammy-linux-nvidia-6.5 verification-needed-jammy-linux-oem-6.5 verification-needed-lunar kernel-spammed-jammy-linux-aws-6.5-v2 kernel-spammed-jammy-linux-azure-6.5-v2 kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-lowlatency-hwe-6.5-v2 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-jammy-linux-nvidia-6.5-v2 kernel-spammed-jammy-linux-oem-6.5-v2 kernel-spammed-lunar-linux-azure patch verification-done-jammy verification-needed-jammy-linux-aws-6.5 verification-needed-jammy-linux-azure-6.5 verification-needed-jammy-linux-lowlatency-hwe-6.5 verification-needed-jammy-linux-nvidia-6.5 verification-needed-jammy-linux-oem-6.5 verification-needed-lunar
2023-11-29 14:13:29 Roxana Nicolescu tags kernel-spammed-jammy-linux-aws-6.5-v2 kernel-spammed-jammy-linux-azure-6.5-v2 kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-lowlatency-hwe-6.5-v2 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-jammy-linux-nvidia-6.5-v2 kernel-spammed-jammy-linux-oem-6.5-v2 kernel-spammed-lunar-linux-azure patch verification-done-jammy verification-needed-jammy-linux-aws-6.5 verification-needed-jammy-linux-azure-6.5 verification-needed-jammy-linux-lowlatency-hwe-6.5 verification-needed-jammy-linux-nvidia-6.5 verification-needed-jammy-linux-oem-6.5 verification-needed-lunar kernel-spammed-jammy-linux-aws-6.5-v2 kernel-spammed-jammy-linux-azure-6.5-v2 kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-lowlatency-hwe-6.5-v2 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-jammy-linux-nvidia-6.5-v2 kernel-spammed-jammy-linux-oem-6.5-v2 kernel-spammed-lunar-linux-azure patch verification-done-jammy verification-done-jammy-linux-lowlatency-hwe-6.5 verification-needed-jammy-linux-aws-6.5 verification-needed-jammy-linux-azure-6.5 verification-needed-jammy-linux-nvidia-6.5 verification-needed-jammy-linux-oem-6.5 verification-needed-lunar
2024-05-14 19:47:46 Ubuntu Kernel Bot tags kernel-spammed-jammy-linux-aws-6.5-v2 kernel-spammed-jammy-linux-azure-6.5-v2 kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-lowlatency-hwe-6.5-v2 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-jammy-linux-nvidia-6.5-v2 kernel-spammed-jammy-linux-oem-6.5-v2 kernel-spammed-lunar-linux-azure patch verification-done-jammy verification-done-jammy-linux-lowlatency-hwe-6.5 verification-needed-jammy-linux-aws-6.5 verification-needed-jammy-linux-azure-6.5 verification-needed-jammy-linux-nvidia-6.5 verification-needed-jammy-linux-oem-6.5 verification-needed-lunar kernel-spammed-jammy-linux-aws-6.5-v2 kernel-spammed-jammy-linux-azure-6.5-v2 kernel-spammed-jammy-linux-hwe-6.2 kernel-spammed-jammy-linux-lowlatency-hwe-6.5-v2 kernel-spammed-jammy-linux-nvidia-6.2 kernel-spammed-jammy-linux-nvidia-6.5-v2 kernel-spammed-jammy-linux-nvidia-6.8-v2 kernel-spammed-jammy-linux-oem-6.5-v2 kernel-spammed-lunar-linux-azure patch verification-done-jammy verification-done-jammy-linux-lowlatency-hwe-6.5 verification-needed-jammy-linux-aws-6.5 verification-needed-jammy-linux-azure-6.5 verification-needed-jammy-linux-nvidia-6.5 verification-needed-jammy-linux-nvidia-6.8 verification-needed-jammy-linux-oem-6.5 verification-needed-lunar