Cloud-Init fails to deal with SWAP and Ephemeral if virtio-scsi is enabled

[Thiago Martins]

Hello!

 In my OpenStack environment, everything works great! Including Cloud-Init.

 However, I can not use virtio-blk anymore (i.e., /dev/vdX devices), because it is too slow.

 I changed to virtio-scsi (i.e., /dev/sdX devices) and the IOPS is more than double!

 But, after this change in Glance images, Cloud-Init is failing to make the SWAP and to mount the Ephemeral volumes.

 * Here is a normal log (with virtio-blk):

 http://paste.ubuntu.com/25127354/

 So, after changing my Ubuntu image properties on Glance, like this:

 glance image-update --property hw_scsi_model=virtio-scsi --property hw_disk_bus=scsi $UBUNTU_IMAGE_ID

 Cloud Init is now failing to make the SWAP, also Ephemeral is not mounted!

 * Here is the error log (with virtio-scsi):

 http://paste.ubuntu.com/25127989/

 Image used in Glance:

 http://uec-images.ubuntu.com/releases/16.04/release/ubuntu-16.04-server-cloudimg-amd64-disk1.img

 If I bring back virtio-blk, cloud-init works again (same Image, same Flavor).

 But, I can't use virtio-blk because it is very slow in my system! The virtio-scsi rocks!

Cheers!
Thiago

[Scott Moser (smoser)]

Can you please collect the config drive that was used ?

at very least we want to get the metadata off it. but just as easy to grab the whole thing.

[Thiago Martins (martinx)]

This is the config driver used on this tests.

[Scott Moser (smoser)]

Thiago, thanks for attaching the config drive.

So here is the net of the problem... (Description as much for me and other
cloud-init developers as for you).

On openstack, to determine block device mappings (mapping a string like "ephemeral0" or "swap" to a real block device like /dev/sdb or /dev/sda) cloud-init utilizes the ec2 metadata information present alongside the openstack metadata. That is because openstack doesn't provide any such data.

The ec2 metadata in <version>/ec2/meta-data.json says:
 "block-device-mapping": {"ami": "sda", "root": "/dev/sda", "swap": "/dev/sdc", "ephemeral0": "/dev/sdb"}

which is clearly wrong.

If that information were not present at all, then cloud-init would attempt
to look the info up by a LABEL on the filesystem (which is less than ideal
but would probably work in this case).

That code is at
  https://git.launchpad.net/~cloud-init-dev/cloud-init/tree/cloudinit/sources/helpers/openstack.py?h=master#n129

[Thiago Martins (martinx)]

Hmm... So, do you think that this is an OpenStack Nova bug instead?

Do you know if there is a good workaround?

Also, why are you saying that this:

-
 "block-device-mapping": {"ami": "sda", "root": "/dev/sda", "swap": "/dev/sdc", "ephemeral0": "/dev/sdb"}
-

Is clearly wrong?

It looks okay to me...

How you think that it must looks like?

[Scott Moser (smoser)]

I said 'clearly wrong' because the deivce is probably not called '/dev/sdb'. I guess it might be but it could just as well be 'vdb'.

[Thiago Martins (martinx)]

Hey Scott,

 I completely forgot to update this bug report.

 This is still a problem with Ubuntu 18.04!

 So, the devices are RIGHT, they're supposed to be called "sda/sdb/sdc" and NOT "vda/vdb/vdc", since I'm using VirtIO-SCSI instead.

 Then, this is a Cloud-Init bug, right?

Cheers!
Thiago

[Scott Moser (smoser)]

Could you attach the output of
$ sudo blkid --cache=/dev/null
and also attach the output of running:
 $ cloud-init collect-logs

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2957