jammy-server-cloudimg-amd64-disk-kvm.img NoCloud data source not detected with qemu SATA cdrom

[Cole Robinson]

cloud-init running in ubuntu jammy-server-cloudimg-amd64-disk-kvm.img does not detect NoCloud cidata ISO in a qemu VM that uses emulated SATA cdrom drive.

SATA is the default used by qemu's -M q35 machine type, which is the preferred modern config over the default `-M pc` machine type, which uses IDE. Using -M pc and therefore IDE _does_ work.

Using the the non-KVM image also _does_ work (jammy-server-cloudimg-amd64.img)

Debugging this a bit showed that cloud-init `ds-identify --force` correctly finds the ISO once I've logged in, but not at boot up, I think because the sata cdrom is discovered by udev too late. This also affects ubuntu20.04 cloud images similarly: -disk-kvm is busted, other image is fine.

Maybe it's an initrd driver issue, or issue with bits being modularized when they shouldn't, not sure. Here's the loaded driver difference for -M pc and -M q35 with -kvm-disk.img

root@ubuntu:~# diff -rup pc q35
--- pc 2022-02-22 19:25:11.731067208 +0000
+++ q35 2022-02-22 19:26:55.749459099 +0000
@@ -7,7 +7,8 @@ loop 28672 6
 dm_multipath 24576 0
 dm_mod 86016 3 dm_multipath
 kvm_intel 200704 0
-pata_acpi 12288 0
+ahci 36864 0
+libahci 24576 1 ahci
 fuse 106496 1
 configfs 32768 1
 ip_tables 24576 0

Some background: I'm upstream virt-manager/virt-install dev. virt-install provides a `--cloud-init` option where we attach a NoCloud formatted ISO. A user reported this doesn't work out of the box with ubuntu20.04 -kvm-disk.img: https://github.com/virt-manager/virt-manager/issues/359

This isn't virt-install --cloud-init specific, see this report which pointed me towards the -kvm-disk difference: https://stackoverflow.com/questions/68512432/cannot-login-to-vm-using-qemu-nocloud-cloud-init-on-macos-with-ubuntu-cloud-imag

Though virt-install users are kinda more likely to hit this since we increasingly default to q35. If users are just testing with qemu-kvm directly without specifying -machine q35, they won't hit this.

Reproducer with virt-install:
git clone https://github.com/virt-manager/virt-manager
cd virt-manager
wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-disk-kvm.img
./virt-install --osinfo ubuntu20.04 --disk jammy-server-cloudimg-amd64-disk-kvm.img --cloud-init

[Cole Robinson (crobinso)]

The reporter of the virt-install bug also found this expired focal cloud-images bug which sounds like the same root issue: https://bugs.launchpad.net/cloud-images/+bug/1940791

If someone with more launchpad knowledge can route this report to the correct place, please do. Would be nice to get this straightened out before 22.04 is released.

[Christian Ehrhardt  (paelzer)]

Hi Cole, FYI I have yesterday pinged the cloud-init develpers about both to check these cases.
I also added the extra bug task here.
And for bug 1940791 I have mentioned that I think it might have expired without a real conclusion.

[James Falcon (falcojr)]

Thanks for the detailed bug report.

I used your reproducer steps, but I got a slightly different result. I'm still not getting the NoCloud identification, but in my case, I can't force ds-identify to identify it. In order for ds-identify to identify NoCloud, it needs to see a "cidata" filesystem label somewhere (or see the relevant kernel args), but "cidata" is not showing up on the sr0 device:

root@ubuntu:/run/cloud-init# lsblk -o name,mountpoint,label
NAME MOUNTPOINT LABEL
loop0 /snap/core20/1328
loop1 /snap/snapd/14549
loop2 /snap/lxd/22340
sr0
vda
├─vda1 / cloudimg-rootfs
├─vda14
└─vda15 /boot/efi UEFI

Have you seen this as well, or have I missed a step in trying to reproduce this?

[Cole Robinson (crobinso)]

Thanks for testing James. But that's different from what I'm seeing.

True full reproducer for me:

Host: Fedora 35

git clone https://github.com/virt-manager/virt-manager
cd virt-manager
wget https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64-disk-kvm.img
# change image root password to 'root'
virt-customize -a jammy-server-cloudimg-amd64-disk-kvm.img --root-password password:root
./virt-install --osinfo ubuntu20.04 --disk jammy-server-cloudimg-amd64-disk-kvm.img --cloud-init

<console spew>

root@ubuntu:~# cat /run/cloud-init/ds-identify.log | grep FS_LABELS
FS_LABELS=UEFI,UEFI,cloudimg-rootfs
root@ubuntu:~# DI_LOG=stderr /usr/lib/cloud-init/ds-identify --force | grep FS_LABELS
FS_LABELS=cidata,UEFI,UEFI,cloudimg-rootfs

ds-identify.log is from cloud-init boot up, FS_LABELS shows that cidata wasn't detected.
But when re-running with --force, once logged in, the label is showing up.

If that's not reproducing for you I can come probably wrangle up a straight qemu reproducer too

[James Falcon (falcojr)]

Chad Smith was able to reproduce this while investigating #1940791, and this appears to be the same root cause as #1940791, so I'm going to mark this one as a duplicate.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.