KVM Containers Get Same Metadata ISOs

[Joseph Phillips]

This example shows that 2 KVMs on the same host have the same host name:
https://pastebin.canonical.com/p/fvcvjz69Bb/

The KVM container manager generates cloud-init user-data into a file here:
https://github.com/juju/juju/blob/f73e98eeb6971ade1da020e6e58448469fc0ad00/container/kvm/kvm.go#L178

In the same example, the directories have user-data files with the *correct* hostnames, so the cause of this behaviour is not yet known.

[Dmitrii Shcherbakov (dmitriis)]

It looks like the target data store ISOs are identical for different KVMs (although they need to be different):

sha256sum juju-21e577-7-kvm-*iso
8842a839db4779840c8fa276e62a2814b32051d061484d0adf0ff5e7ba849290 juju-21e577-7-kvm-0-ds.iso
8842a839db4779840c8fa276e62a2814b32051d061484d0adf0ff5e7ba849290 juju-21e577-7-kvm-1-ds.iso

The user-data generated is different though as it should be under /var/lib/juju/containers/*/user-data:

root@control-9:/var/lib/juju/kvm/guests# grep -RiP hostname /var/lib/juju/containers/juju-21e577-7-kvm-*/
/var/lib/juju/containers/juju-21e577-7-kvm-0/user-data:hostname: juju-21e577-7-kvm-0
/var/lib/juju/containers/juju-21e577-7-kvm-1/user-data:hostname: juju-21e577-7-kvm-1

[Dmitrii Shcherbakov (dmitriis)]

Previous reproductions:

https://github.com/juju/juju/pull/9898#issuecomment-474345689 (with https://github.com/juju/juju/pull/9898)
https://bugs.launchpad.net/juju/+bug/1816170/comments/14 (with https://github.com/juju/juju/pull/9912)

[Tim Penhey (thumper)]

@Dmitrii, I'm confused.

> It looks like the target data store ISOs are identical for different KVMs
> (although they need to be different):

Why?

It is my understanding that the ISOs that have been downloaded are based on the cloud images for the various OS series we are using. If the two KVM machines are the same series, like bionic, then yes the ISOs will be the same for both instances, as they are both starting with the bionic cloud image.

Juju delivers the customisation using cloud-init. This is what is giving the machines their identity in the Juju model.

So... why do you say that the ISOs need to be different?

[Dmitrii Shcherbakov (dmitriis)]

Tim,

There are several moving parts here:

1) Ubuntu cloud image (used as a base image for every VM in qcow2 format);
2) VM-specific images created as qcow2 snapshots of that cloud image (qcow2);
3) the data store ISO which contains only meta-data for the target instance (https://cloudinit.readthedocs.io/en/latest/topics/datasources/configdrive.html) - this is generated via genisoimage (https://github.com/juju/juju/blob/juju-2.5.2/container/kvm/wrappedcmds.go#L367-L375).

When a VM is created a new image is created (2) as a snapshot of the base image via `qemu-img create` (1) - this works well after Joseph's fixes.

(3) is passed as a separate storage device to a VM and has metadata relevant to a specific VM that Juju generates (cloudinit-userdata, mostly runcmds for agent installation etc.):

pgrep -af qemu | grep -ioP 'drive file=.*?\s'
drive file=/var/lib/juju/kvm/guests/juju-d703a2-6-kvm-1.qcow,format=qcow2,if=none,id=drive-virtio-disk0
drive file=/var/lib/juju/kvm/guests/juju-d703a2-6-kvm-1-ds.iso,format=raw,if=none,id=drive-virtio-disk1
drive file=/var/lib/juju/kvm/guests/juju-d703a2-6-kvm-0.qcow,format=qcow2,if=none,id=drive-virtio-disk0
drive file=/var/lib/juju/kvm/guests/juju-d703a2-6-kvm-0-ds.iso,format=raw,if=none,id=drive-virtio-disk1

So those ISOs with metadata have to have different contents as they hold different metadata for different VMs.

In my case, a hostname is only a part of the metadata that was different. netplan config was also related to kvm-1 while it should have been for kvm-0.

https://pastebin.canonical.com/p/fvcvjz69Bb/

As a result, netplan config could not be applied as the VM simply had different MAC addresses.

[Dmitrii Shcherbakov (dmitriis)]

The strange part about this bug is that the two ISOs are the same in the end but the metadata prepared by Juju from which these ISOs are supposed to be generated is different:

root@control-9:/var/lib/juju/kvm/guests# grep -RiP hostname /var/lib/juju/containers/juju-21e577-7-kvm-*/
/var/lib/juju/containers/juju-21e577-7-kvm-0/user-data:hostname: juju-21e577-7-kvm-0
/var/lib/juju/containers/juju-21e577-7-kvm-1/user-data:hostname: juju-21e577-7-kvm-1

Given that the machine that got the wrong ISO definitely got correct interfaces in an XML definition of the VM, I suspect that CreateMachine ran successfully at least once:
https://github.com/juju/juju/blob/juju-2.5.2/container/kvm/wrappedcmds.go#L167-L213

Something seems to have overridden the metadata ISO for kvm-0 with contents of the ISO for kvm-1.

I can redeploy and check crtime vs mtime for the kvm-0 ISO to have a more definitive proof of that.

[Joseph Phillips (manadart)]

Looks like the issue is here:
https://github.com/juju/juju/blob/f73e98eeb6971ade1da020e6e58448469fc0ad00/container/kvm/wrappedcmds.go#L355

The working directory is changed so that genisoimage is run with user-data, meta-data, and network-config as relative paths.

According to https://github.com/golang/go/issues/27658 this is not Goroutine-safe. Dmitrii's observations are consistent with a race here.

[Joseph Phillips (manadart)]

Proposed fix: https://github.com/juju/juju/pull/9922