Cannot boot from volume with 2 devices

[Gaurav Gupta]

More details on:
https://answers.launchpad.net/nova/+question/176938

Summary:
-------------
Say I had 2 disks, disk1 and disk2 (represented by 2 volumes). disk1 has the root-file-system and disk2 has some data. I boot an instances using the boot-from-volumes extension, and specify the 2 disks such as disk1 should be attached to /dev/vda and disk2 to /dev/vdb. When the instance is launched it fails to boot, because it tries to find the root-filesystem on disk2 instead.

The underlying problem is with virsh/libvirt. Boot fails because in the libvirt.xml file created by Openstack, disk2 (/dev/vdb) is listed before disk1 (/dev/vda). So, what happens is that the hypervisor attaches disk2 first (since its listed first in the XML). Therefore when these disks are attached on the guest, disk2 appears as /dev/vda and disk1 as /dev/vdb. Later the kernel tries to find the root filesystem on '/dev/vda' (because that's what is selected as the root) and it fails for obvious reason. I think it's a virsh bug. It should be smart about it and attach the devices in the right order.

[Gaurav Gupta (gagupta)]

Vish suggested:
"Sounds like we can work around this pretty easily by sorting the disks before we pass them into the xml template."

[Isaku Yamahata (yamahata)]

The order in XML isn't a part of libvirt API. It's just how the current libvirt is implemented and qemu-kvm works.
So sorting is not long term solution. Only acceptable as short term work around.
For example, if you specify /dev/vda and /dev/vdc (skipping /dev/vdb), sorting doesn't work.

How guest OS associates device names with devices is highly OS/distribution dependent,
thus libvirt can't know all the cases.

So long term solution is
- use disk label where possible (This is already suggested by Scott)
- pass the information of the relation to the guest OS from nova-compute(or libvirt) somehow

If guest BIOS(seabios) is involuved for OS booting, the information also needs to be passed to BIOS.

[Gaurav Gupta (gagupta)]

Is there any more thought about this bug. Can we prioritise this in Essex.

[Thierry Carrez (ttx)]

Adding openstack-volumes team to cc:

[Vish Ishaya (vishvananda)]

I still think sorting is a good short term solution to get booting to work again. Beyond that we need better support for naming virtio disks. We may have to do some guest agent / udev magic to get the names right because it doesn't appear that udev has any way to tell which device name it should pick for attached volumes.

[Isaku Yamahata (yamahata)]

Hmm I didn't say precisely my opinion.
I'm not objecting against sorting as short term work around. Rather I think it should be committed at first because it would take long time to attack long term solution. It needs to fix not only nova but also other component like kvm, libvirt.
The reason why xen case(AWS) works is such information is passed to drivers in kernel via xenstore. On the other hand KVM lacks such framework yet.

When short term work around is committed,
please put on the comment on the code that it's work around,
and please don't mark this bug fixed, keep this bug open until the long term solution is committed.

[Nikola Đipanov (ndipanov)]

I believe that with the new block device syntax introduced in https://blueprints.launchpad.net/nova/+spec/improve-block-device-handling now allows you to sidestep these issues as it allows for explicitly specifying the root device when booting. So for example booting with two volumes one of which is the root device can now be done like so:

$ nova boot --boot_volume $BOOT_VOL_ID --block_device id=$OTHER_VOL_ID,source=volume,dest=volume --flavor=$FLAVOR_ID testvm

I am marking this bug as invalid, but the reporter or anyone else should feel free to provide more information and re-open should they feel the above does not address the problem fully.