live migration does not work after volume migration

[eugene]

Hi,

$subj

[#]> rpm -qi openstack-nova-common.noarch
Name : openstack-nova-common
Epoch : 1
Version : 15.0.0
Release : 1.el7

STR:
-configure 2x compute node with cinder storage and lvm back-ends;
-crate bootable volume from image;
-create instance and use bootable volume;
-migrate volume from current node to another
-try to perform live migration

Actual Result:
-live migration fails:
-------------------------
2017-07-21 11:33:00.554 4552 ERROR nova.virt.libvirt.driver [req-792a37bf-3a2e-4976-84ad-cc308eb1ffbf - - - - -] [instance: 5dd0b6ab-8743-42a2-af7a-38f1fb10dedf] Live Migration failure: missing source information for device vda
2017-07-21 11:33:00.599 4552 ERROR nova.virt.libvirt.driver [req-792a37bf-3a2e-4976-84ad-cc308eb1ffbf - - - - -] [instance: 5dd0b6ab-8743-42a2-af7a-38f1fb10dedf] Migration operation has aborted

Jul 21 11:33:00 compute-02 nova-compute: Traceback (most recent call last):
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
Jul 21 11:33:00 compute-02 nova-compute: timer()
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
Jul 21 11:33:00 compute-02 nova-compute: cb(*args, **kw)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 168, in _do_send
Jul 21 11:33:00 compute-02 nova-compute: waiter.switch(result)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
Jul 21 11:33:00 compute-02 nova-compute: result = function(*args, **kwargs)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/nova/utils.py", line 1087, in context_wrapper
Jul 21 11:33:00 compute-02 nova-compute: return func(*args, **kwargs)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6150, in _live_migration_operation
Jul 21 11:33:00 compute-02 nova-compute: instance=instance)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jul 21 11:33:00 compute-02 nova-compute: self.force_reraise()
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jul 21 11:33:00 compute-02 nova-compute: six.reraise(self.type_, self.value, self.tb)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6143, in _live_migration_operation
Jul 21 11:33:00 compute-02 nova-compute: bandwidth=CONF.libvirt.live_migration_bandwidth)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 623, in migrate
Jul 21 11:33:00 compute-02 nova-compute: destination, params=params, flags=flags)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
Jul 21 11:33:00 compute-02 nova-compute: result = proxy_call(self._autowrap, f, *args, **kwargs)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
Jul 21 11:33:00 compute-02 nova-compute: rv = execute(f, *args, **kwargs)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
Jul 21 11:33:00 compute-02 nova-compute: six.reraise(c, e, tb)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
Jul 21 11:33:00 compute-02 nova-compute: rv = meth(*args, **kwargs)
Jul 21 11:33:00 compute-02 nova-compute: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1939, in migrateToURI3
Jul 21 11:33:00 compute-02 nova-compute: if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
Jul 21 11:33:00 compute-02 nova-compute: libvirtError: missing source information for device vda
-------------------------

Here is disk description when instance was created
--------------------------
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/disk/by-path/ip-10.30.30.101:3260-iscsi-iqn.2010-10.org.openstack:volume-fcc11b41-514e-4a91-a29e-4632c4f42648-lun-0'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <serial>4c0b28d5-fc0b-4907-b563-b47b521bc945</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
--------------------------

Here is disk description during migration:
--------------------------
   <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/disk/by-path/ip-10.30.30.101:3260-iscsi-iqn.2010-10.org.openstack:volume-fcc11b41-514e-4a91-a29e-4632c4f42648-lun-0'/>
      <backingStore/>
      <mirror type='file' file='/dev/disk/by-path/ip-10.30.30.100:3260-iscsi-iqn.2010-10.org.openstack:volume-7604d08d-e987-4c8d-9a9b-56f3258d5b60-lun-0' format='raw' job='copy'>
        <format type='raw'/>
        <source file='/dev/disk/by-path/ip-10.30.30.100:3260-iscsi-iqn.2010-10.org.openstack:volume-7604d08d-e987-4c8d-9a9b-56f3258d5b60-lun-0'/>
      </mirror>
      <target dev='vda' bus='virtio'/>
      <serial>4c0b28d5-fc0b-4907-b563-b47b521bc945</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
--------------------------

Here is disk description after the migration:
--------------------------
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/dev/disk/by-path/ip-10.30.30.100:3260-iscsi-iqn.2010-10.org.openstack:volume-7604d08d-e987-4c8d-9a9b-56f3258d5b60-lun-0'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <serial>4c0b28d5-fc0b-4907-b563-b47b521bc945</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
--------------------------

Here is disk description after instance hard reboot:
--------------------------
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/disk/by-path/ip-10.30.30.100:3260-iscsi-iqn.2010-10.org.openstack:volume-7604d08d-e987-4c8d-9a9b-56f3258d5b60-lun-0'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <serial>4c0b28d5-fc0b-4907-b563-b47b521bc945</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
--------------------------

The issue is caused by the fact that dist type is changed from 'block' to 'file' after volume migration. Please note that after hard reboot instance the xml description is fixed and live migration works.

[Kashyap Chamarthy (kashyapc)]

So, as per this report, it seems like 'block' as the disk type post-migration is not being retained.

Here's a related (merged) patch that changed the disk type from 'file' to 'block' for block devices:

  https://review.openstack.org/#/c/465205/: Use VIR_DOMAIN_BLOCK_REBASE_COPY_DEV when rebasing

[Sylvain Bauza (sylvain-bauza)]

Looks related to https://bugs.launchpad.net/nova/+bug/1691195 whose the fix was backported to Newton and Ocata.

https://review.openstack.org/#/c/465205/

[Sylvain Bauza (sylvain-bauza)]

Just saw you were on 15.0.0, could you please upgrade your compute nodes to the latest stable Ocata release (at least >= 15.0.6) so you could tell us if the above bugfix (which was backported to ocata-15.0.6) fixes your problem ?

[eugene (mamut)]

>so you could tell us if the above bugfix (which was backported to ocata-15.0.6) fixes your problem?

Works like a charm after updating the packeges. Thank you!

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]