Activity log for bug #2036968

Date Who What changed Old value New value Message
2023-09-21 13:22:35 Philip Roche bug added bug
2023-09-21 13:32:14 Philip Roche bug task added linux (Ubuntu)
2023-09-21 13:37:19 Dimitri John Ledkov linux (Ubuntu): milestone ubuntu-23.10
2023-09-21 13:39:49 Andreas Hasenack bug added subscriber Andreas Hasenack
2023-09-21 13:40:27 Andreas Hasenack description Following a recent change from linux-kvm kernel to linux-generic kernel in the mantic minimized images there is a reproducable bug where a guest VM does not have an IP address assigned as part of cloud-init provisioning. This is easiest to reproduce when emulating arm64 on adm64 host. The bug is a race condition so there could exist fast enough virtualisation on fast enough hardware where this bug is not present but in all my testing I have been able to reproduce. The latest mantic minimized images from http://cloud-images.ubuntu.com/minimal/daily/mantic/ have force initrdless boot and no initrd to fallback to. This but is not present in the non minimized/base images @ http://cloud-images.ubuntu.com/mantic/ as these boot with initrd with the required drivers present for virtio-net. Reproducer ``` wget -O "launch-qcow2-image-qemu-arm64.sh" https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/launch-qcow2-image-qemu-arm64.sh chmod +x ./launch-qcow2-image-qemu-arm64.sh wget https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/livecd.ubuntu-cpc.img ./launch-qcow2-image-qemu-arm64.sh --password passw0rd --image ./livecd.ubuntu-cpc.img ``` You will then be able to log in with user `ubuntu` and password `passw0rd`. You can run `ip a` and see that there is a network interface present (separate to `lo`) but no IP address has been assigned. ``` ubuntu@cloudimg:~$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff ``` This is because when cloud-init is trying to configure network interfaces it doesn't find any so it doesn't configure any. But by the time boot is complete the network interface is present but cloud-init provisioning has already completed. You can verify this by running `sudo cloud-init clean && sudo cloud-init init` You can then see a successfully configured network interface ``` ubuntu@cloudimg:~$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic enp0s1 valid_lft 86391sec preferred_lft 86391sec inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute valid_lft 86393sec preferred_lft 14393sec inet6 fe80::5054:ff:fe12:3456/64 scope link valid_lft forever preferred_lft forever ``` The bug is also reproducable with amd64 guest on adm64 host on older/slower hardware. The suggested fixes while debugging this issue are: * to include `virtio-net` as a built-in in the mantic generic kernel * understand what needs to change in cloud-init so that it can react to late additions of network interfaces I will file a separate bug against cloud-init to address the race condition on emulated guest/older hardware. Following a recent change from linux-kvm kernel to linux-generic kernel in the mantic minimized images there is a reproducable bug where a guest VM does not have an IP address assigned as part of cloud-init provisioning. This is easiest to reproduce when emulating arm64 on adm64 host. The bug is a race condition so there could exist fast enough virtualisation on fast enough hardware where this bug is not present but in all my testing I have been able to reproduce. The latest mantic minimized images from http://cloud-images.ubuntu.com/minimal/daily/mantic/ have force initrdless boot and no initrd to fallback to. This bug is not present in the non minimized/base images @ http://cloud-images.ubuntu.com/mantic/ as these boot with initrd with the required drivers present for virtio-net. Reproducer ``` wget -O "launch-qcow2-image-qemu-arm64.sh" https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/launch-qcow2-image-qemu-arm64.sh chmod +x ./launch-qcow2-image-qemu-arm64.sh wget https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/livecd.ubuntu-cpc.img ./launch-qcow2-image-qemu-arm64.sh --password passw0rd --image ./livecd.ubuntu-cpc.img ``` You will then be able to log in with user `ubuntu` and password `passw0rd`. You can run `ip a` and see that there is a network interface present (separate to `lo`) but no IP address has been assigned. ``` ubuntu@cloudimg:~$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: enp0s1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000     link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff ``` This is because when cloud-init is trying to configure network interfaces it doesn't find any so it doesn't configure any. But by the time boot is complete the network interface is present but cloud-init provisioning has already completed. You can verify this by running `sudo cloud-init clean && sudo cloud-init init` You can then see a successfully configured network interface ``` ubuntu@cloudimg:~$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000     link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff     inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic enp0s1        valid_lft 86391sec preferred_lft 86391sec     inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute        valid_lft 86393sec preferred_lft 14393sec     inet6 fe80::5054:ff:fe12:3456/64 scope link        valid_lft forever preferred_lft forever ``` The bug is also reproducable with amd64 guest on adm64 host on older/slower hardware. The suggested fixes while debugging this issue are: * to include `virtio-net` as a built-in in the mantic generic kernel * understand what needs to change in cloud-init so that it can react to late additions of network interfaces I will file a separate bug against cloud-init to address the race condition on emulated guest/older hardware.
2023-09-21 13:41:19 Andreas Hasenack description Following a recent change from linux-kvm kernel to linux-generic kernel in the mantic minimized images there is a reproducable bug where a guest VM does not have an IP address assigned as part of cloud-init provisioning. This is easiest to reproduce when emulating arm64 on adm64 host. The bug is a race condition so there could exist fast enough virtualisation on fast enough hardware where this bug is not present but in all my testing I have been able to reproduce. The latest mantic minimized images from http://cloud-images.ubuntu.com/minimal/daily/mantic/ have force initrdless boot and no initrd to fallback to. This bug is not present in the non minimized/base images @ http://cloud-images.ubuntu.com/mantic/ as these boot with initrd with the required drivers present for virtio-net. Reproducer ``` wget -O "launch-qcow2-image-qemu-arm64.sh" https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/launch-qcow2-image-qemu-arm64.sh chmod +x ./launch-qcow2-image-qemu-arm64.sh wget https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/livecd.ubuntu-cpc.img ./launch-qcow2-image-qemu-arm64.sh --password passw0rd --image ./livecd.ubuntu-cpc.img ``` You will then be able to log in with user `ubuntu` and password `passw0rd`. You can run `ip a` and see that there is a network interface present (separate to `lo`) but no IP address has been assigned. ``` ubuntu@cloudimg:~$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: enp0s1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000     link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff ``` This is because when cloud-init is trying to configure network interfaces it doesn't find any so it doesn't configure any. But by the time boot is complete the network interface is present but cloud-init provisioning has already completed. You can verify this by running `sudo cloud-init clean && sudo cloud-init init` You can then see a successfully configured network interface ``` ubuntu@cloudimg:~$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000     link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff     inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic enp0s1        valid_lft 86391sec preferred_lft 86391sec     inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute        valid_lft 86393sec preferred_lft 14393sec     inet6 fe80::5054:ff:fe12:3456/64 scope link        valid_lft forever preferred_lft forever ``` The bug is also reproducable with amd64 guest on adm64 host on older/slower hardware. The suggested fixes while debugging this issue are: * to include `virtio-net` as a built-in in the mantic generic kernel * understand what needs to change in cloud-init so that it can react to late additions of network interfaces I will file a separate bug against cloud-init to address the race condition on emulated guest/older hardware. Following a recent change from linux-kvm kernel to linux-generic kernel in the mantic minimized images there is a reproducable bug where a guest VM does not have an IP address assigned as part of cloud-init provisioning. This is easiest to reproduce when emulating arm64 on adm64 host. The bug is a race condition so there could exist fast enough virtualisation on fast enough hardware where this bug is not present but in all my testing I have been able to reproduce. The latest mantic minimized images from http://cloud-images.ubuntu.com/minimal/daily/mantic/ have force initrdless boot and no initrd to fallback to. This bug is not present in the non minimized/base images @ http://cloud-images.ubuntu.com/mantic/ as these boot with initrd with the required drivers present for virtio-net. Reproducer ``` wget -O "launch-qcow2-image-qemu-arm64.sh" https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/launch-qcow2-image-qemu-arm64.sh chmod +x ./launch-qcow2-image-qemu-arm64.sh wget https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/livecd.ubuntu-cpc.img ./launch-qcow2-image-qemu-arm64.sh --password passw0rd --image ./livecd.ubuntu-cpc.img ``` You will then be able to log in with user `ubuntu` and password `passw0rd`. You can run `ip a` and see that there is a network interface present (separate to `lo`) but no IP address has been assigned. ``` ubuntu@cloudimg:~$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: enp0s1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000     link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff ``` This is because when cloud-init-local.service is trying to configure network interfaces it doesn't find any so it doesn't configure any. But by the time boot is complete the network interface is present but cloud-init provisioning has already completed. You can verify this by running `sudo cloud-init clean && sudo cloud-init init` You can then see a successfully configured network interface ``` ubuntu@cloudimg:~$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000     link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff     inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic enp0s1        valid_lft 86391sec preferred_lft 86391sec     inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute        valid_lft 86393sec preferred_lft 14393sec     inet6 fe80::5054:ff:fe12:3456/64 scope link        valid_lft forever preferred_lft forever ``` The bug is also reproducable with amd64 guest on adm64 host on older/slower hardware. The suggested fixes while debugging this issue are: * to include `virtio-net` as a built-in in the mantic generic kernel * understand what needs to change in cloud-init so that it can react to late additions of network interfaces I will file a separate bug against cloud-init to address the race condition on emulated guest/older hardware.
2023-09-21 13:56:37 Andrew Cloke bug added subscriber Andrew Cloke
2023-09-21 14:00:07 Ubuntu Kernel Bot linux (Ubuntu): status New Incomplete
2023-09-21 14:25:09 Philip Roche description Following a recent change from linux-kvm kernel to linux-generic kernel in the mantic minimized images there is a reproducable bug where a guest VM does not have an IP address assigned as part of cloud-init provisioning. This is easiest to reproduce when emulating arm64 on adm64 host. The bug is a race condition so there could exist fast enough virtualisation on fast enough hardware where this bug is not present but in all my testing I have been able to reproduce. The latest mantic minimized images from http://cloud-images.ubuntu.com/minimal/daily/mantic/ have force initrdless boot and no initrd to fallback to. This bug is not present in the non minimized/base images @ http://cloud-images.ubuntu.com/mantic/ as these boot with initrd with the required drivers present for virtio-net. Reproducer ``` wget -O "launch-qcow2-image-qemu-arm64.sh" https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/launch-qcow2-image-qemu-arm64.sh chmod +x ./launch-qcow2-image-qemu-arm64.sh wget https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/livecd.ubuntu-cpc.img ./launch-qcow2-image-qemu-arm64.sh --password passw0rd --image ./livecd.ubuntu-cpc.img ``` You will then be able to log in with user `ubuntu` and password `passw0rd`. You can run `ip a` and see that there is a network interface present (separate to `lo`) but no IP address has been assigned. ``` ubuntu@cloudimg:~$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: enp0s1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000     link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff ``` This is because when cloud-init-local.service is trying to configure network interfaces it doesn't find any so it doesn't configure any. But by the time boot is complete the network interface is present but cloud-init provisioning has already completed. You can verify this by running `sudo cloud-init clean && sudo cloud-init init` You can then see a successfully configured network interface ``` ubuntu@cloudimg:~$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000     link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff     inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic enp0s1        valid_lft 86391sec preferred_lft 86391sec     inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute        valid_lft 86393sec preferred_lft 14393sec     inet6 fe80::5054:ff:fe12:3456/64 scope link        valid_lft forever preferred_lft forever ``` The bug is also reproducable with amd64 guest on adm64 host on older/slower hardware. The suggested fixes while debugging this issue are: * to include `virtio-net` as a built-in in the mantic generic kernel * understand what needs to change in cloud-init so that it can react to late additions of network interfaces I will file a separate bug against cloud-init to address the race condition on emulated guest/older hardware. Following a recent change from linux-kvm kernel to linux-generic kernel in the mantic minimized images, there is a reproducable bug where a guest VM does not have an IP address assigned as part of cloud-init provisioning. This is easiest to reproduce when emulating arm64 on amd64 host. The bug is a race condition, so there could exist fast enough virtualisation on fast enough hardware where this bug is not present but in all my testing I have been able to reproduce. The latest mantic minimized images from http://cloud-images.ubuntu.com/minimal/daily/mantic/ have force initrdless boot and no initrd to fallback to. This but is not present in the non minimized/base images @ http://cloud-images.ubuntu.com/mantic/ as these boot with initrd with the required drivers present for virtio-net. Reproducer ``` wget -O "launch-qcow2-image-qemu-arm64.sh" https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/launch-qcow2-image-qemu-arm64.sh chmod +x ./launch-qcow2-image-qemu-arm64.sh wget https://people.canonical.com/~philroche/20230921-cloud-images-mantic-fail-to-provision/livecd.ubuntu-cpc.img ./launch-qcow2-image-qemu-arm64.sh --password passw0rd --image ./livecd.ubuntu-cpc.img ``` You will then be able to log in with user `ubuntu` and password `passw0rd`. You can run `ip a` and see that there is a network interface present (separate to `lo`) but no IP address has been assigned. ``` ubuntu@cloudimg:~$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: enp0s1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000     link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff ``` This is because when cloud-init is trying to configure network interfaces it doesn't find any so it doesn't configure any. But by the time boot is complete the network interface is present but cloud-init provisioning has already completed. You can verify this by running `sudo cloud-init clean && sudo cloud-init init` You can then see a successfully configured network interface ``` ubuntu@cloudimg:~$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     inet 127.0.0.1/8 scope host lo        valid_lft forever preferred_lft forever     inet6 ::1/128 scope host        valid_lft forever preferred_lft forever 2: enp0s1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000     link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff     inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic enp0s1        valid_lft 86391sec preferred_lft 86391sec     inet6 fec0::5054:ff:fe12:3456/64 scope site dynamic mngtmpaddr noprefixroute        valid_lft 86393sec preferred_lft 14393sec     inet6 fe80::5054:ff:fe12:3456/64 scope link        valid_lft forever preferred_lft forever ``` The bug is also reproducible with amd64 guest on adm64 host on older/slower hardware. The suggested fixes while debugging this issue are: * to include `virtio-net` as a built-in in the mantic generic kernel * understand what needs to change in cloud-init so that it can react to late additions of network interfaces I will file a separate bug against cloud-init to address the race condition on emulated guest/older hardware.
2023-09-21 15:28:20 Philip Roche bug watch added https://github.com/canonical/cloud-init/issues/4451
2023-09-21 21:21:44 Ryan Harper bug added subscriber Ryan Harper
2023-09-22 16:49:42 Frank Heimes bug added subscriber Frank Heimes
2023-09-25 12:32:44 Dimitri John Ledkov linux (Ubuntu): status Incomplete Triaged
2023-10-02 13:16:57 Dimitri John Ledkov linux (Ubuntu): status Triaged Fix Committed
2023-10-02 13:57:11 Dimitri John Ledkov bug task added systemd (Ubuntu)
2023-10-02 13:57:19 Dimitri John Ledkov tags rls-mm-incoming
2023-10-02 13:58:50 Dimitri John Ledkov summary Mantic minimized/minimal cloud images do not receive IP address during provisioning Mantic minimized/minimal cloud images do not receive IP address during provisioning; systemd regression with wait-online
2023-10-02 14:14:28 Nick Rosbrook tags rls-mm-incoming foundations-todo
2023-10-02 14:14:37 Nick Rosbrook tags foundations-todo foundations-todo rls-mm-incoming
2023-10-02 14:18:33 Nick Rosbrook systemd (Ubuntu): status New Triaged
2023-10-02 14:18:42 Nick Rosbrook systemd (Ubuntu): importance Undecided High
2023-10-02 14:18:45 Nick Rosbrook systemd (Ubuntu): assignee Nick Rosbrook (enr0n)
2023-10-05 08:02:36 Launchpad Janitor linux (Ubuntu): status Fix Committed Fix Released
2023-10-05 15:12:33 Julian Andres Klode tags foundations-todo rls-mm-incoming foundations-todo
2023-10-05 15:13:40 Julian Andres Klode systemd (Ubuntu): importance High Medium
2023-10-07 10:53:16 Ubuntu Kernel Bot tags foundations-todo foundations-todo kernel-spammed-jammy-linux-oem-6.5-v2 verification-needed-jammy-linux-oem-6.5
2023-10-24 16:47:51 Ubuntu Kernel Bot tags foundations-todo kernel-spammed-jammy-linux-oem-6.5-v2 verification-needed-jammy-linux-oem-6.5 foundations-todo kernel-spammed-jammy-linux-azure-6.5-v2 kernel-spammed-jammy-linux-oem-6.5-v2 verification-needed-jammy-linux-azure-6.5 verification-needed-jammy-linux-oem-6.5
2023-10-24 18:22:37 Ubuntu Kernel Bot tags foundations-todo kernel-spammed-jammy-linux-azure-6.5-v2 kernel-spammed-jammy-linux-oem-6.5-v2 verification-needed-jammy-linux-azure-6.5 verification-needed-jammy-linux-oem-6.5 foundations-todo kernel-spammed-jammy-linux-aws-6.5-v2 kernel-spammed-jammy-linux-azure-6.5-v2 kernel-spammed-jammy-linux-oem-6.5-v2 verification-needed-jammy-linux-aws-6.5 verification-needed-jammy-linux-azure-6.5 verification-needed-jammy-linux-oem-6.5
2024-05-14 19:47:29 Ubuntu Kernel Bot tags foundations-todo kernel-spammed-jammy-linux-aws-6.5-v2 kernel-spammed-jammy-linux-azure-6.5-v2 kernel-spammed-jammy-linux-oem-6.5-v2 verification-needed-jammy-linux-aws-6.5 verification-needed-jammy-linux-azure-6.5 verification-needed-jammy-linux-oem-6.5 foundations-todo kernel-spammed-jammy-linux-aws-6.5-v2 kernel-spammed-jammy-linux-azure-6.5-v2 kernel-spammed-jammy-linux-nvidia-6.8-v2 kernel-spammed-jammy-linux-oem-6.5-v2 verification-needed-jammy-linux-aws-6.5 verification-needed-jammy-linux-azure-6.5 verification-needed-jammy-linux-nvidia-6.8 verification-needed-jammy-linux-oem-6.5