VM's disk configuration not persisted after hard-reboot

[Simon]

While running current trunk code, I started a VM to boot from an image but creating a volume. The image and volume are both on ceph. The initial VM libvirt.xml file is correct - the devices/disk portion is pointing to ceph.

After hard-rebooting the VM, however, the libvirt.xml file for some reason pointing to a non-existent qcow file on the file system ../nova/instances/<uuid>/disk.

[Simon (xchenum)]

I think the problem is that the source_type of the volume is "image" and the destination_type is "volume", in this case.

Unfortunately, class DriverVolumeBlockDevice only allows both source and dest to be "volume".

https://github.com/openstack/nova/blob/master/nova/virt/block_device.py

[Blane Bramble (blane)]

I think I have also hit this problem when using Ceph storage and boot from image (create new volume). Everything works fine until either I do a hard reboot or evacuate from a server. diff of the created file vs the original working one is:

31,32c31,32
< <driver type="raw" cache="none"/>
< <source protocol="rbd" name="volumes/instance-00000003_disk">
---
> <driver name="qemu" type="raw" cache="none"/>
> <source protocol="rbd" name="volumes/volume-9e1a7835-b780-495c-a88a-4558be784dde">
40a41
> <serial>9e1a7835-b780-495c-a88a-4558be784dde</serial>

So the correct rbd volumes path is overwritten with a non-existant instance name one.

[Blane Bramble (blane)]

Simon seems to be on the right track - manually changing the source_type in nova.block_device_mapping in the DB fixes the issue. Is this a case of the class needing to handle different source and destination types or should the source type be set to volume when the volume is created?

[Simon (xchenum)]

I tried to fix this issue by changing the source_type to source_type_list, including both 'image' and 'volume', but this somehow fails the creation process. Must be impacting something else :-(

[Blane Bramble (blane)]

Hi Simon, see the work-around in this ticket https://bugs.launchpad.net/nova/+bug/1245719

[Simon (xchenum)]

Doesn't work.. Can't create VMs at all with this tweak..

[Blane Bramble (blane)]

Hi Simon, do you get a different error in the log? Also, what does the livirt.xml look like after a hard reboot when using the patch?

[Simon (xchenum)]

Emm. I forgot to capture the error, but basically I cannot create new VMs when applying this tweak. The log shows something like volume None cannot be attached.

[Simon (xchenum)]

Here you go:

2013-11-07 02:05:14.044 ERROR nova.compute.manager [req-a2a723f1-226b-4439-a1c2-5c60b630ee9e xxx xxx] [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] Error: Volume None could not be found.
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] Traceback (most recent call last):
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] File "/opt/stack/nova/nova/compute/manager.py", line 1021, in _build_instance
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] context, instance, bdms)
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] File "/opt/stack/nova/nova/compute/manager.py", line 1422, in _prep_block_device
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] instance=instance)
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] File "/opt/stack/nova/nova/compute/manager.py", line 1400, in _prep_block_device
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] self._await_block_device_map_created) +
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] File "/opt/stack/nova/nova/virt/block_device.py", line 291, in attach_block_devices
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] block_device_mapping)
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] File "/opt/stack/nova/nova/virt/block_device.py", line 155, in attach
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] volume = volume_api.get(context, self.volume_id)
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] File "/opt/stack/nova/nova/volume/cinder.py", line 176, in wrapper
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] res = method(self, ctx, volume_id, *args, **kwargs)
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] File "/opt/stack/nova/nova/volume/cinder.py", line 211, in get
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] item = cinderclient(context).volumes.get(volume_id)
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] File "/opt/stack/python-cinderclient/cinderclient/v1/volumes.py", line 196, in get
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] return self._get("/volumes/%s" % volume_id, "volume")
2013-11-07 02:05:14.044 19193 TRACE nova.compute.manager [instance: 861cf9aa-7088-4de1-b7c9-2044486244a8] File "/opt/stack/python-cinderclient/cinderclient/base.py", line 145, in _get
2013-...

[Blane Bramble (blane)]

Odd. Could you post the libvirt.xml after the hard reboot attempt?

[Simon (xchenum)]

I meant new VMs cannot be created. So this is not about hard reboot anymore..

[Blane Bramble (blane)]

Ahh, OK - what method are you selecting as the Instance Boot Source?

[Simon (xchenum)]

boot from image, create a new volume