gate-tempest-dsvm-multinode-live-migration fails with "Live Migration failure: Operation not supported: Selecting disks to migrate is not implemented for tunnelled migration"

Bug #1589591 reported by Matt Riedemann
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
High
Timofey Durakov

Bug Description

Seeing that here:

http://logs.openstack.org/04/325204/2/experimental/gate-tempest-dsvm-multinode-live-migration/2b050a0/logs/screen-n-cpu.txt.gz?level=TRACE#_2016-06-06_16_07_04_867

I'm not sure if this is a job config issue or a bug in the code.

2016-06-06 16:07:04.867 20822 ERROR nova.virt.libvirt.driver [req-1e0f7bf7-e455-4a10-8b43-3a5311e11207 tempest-LiveBlockMigrationTestJSON-1933661062 tempest-LiveBlockMigrationTestJSON-1933661062] [instance: 1a447113-59d4-48f5-9ee8-3f7120e1a8fc] Live Migration failure: Operation not supported: Selecting disks to migrate is not implemented for tunnelled migration
2016-06-06 16:07:04.868 20822 DEBUG nova.virt.libvirt.driver [req-1e0f7bf7-e455-4a10-8b43-3a5311e11207 tempest-LiveBlockMigrationTestJSON-1933661062 tempest-LiveBlockMigrationTestJSON-1933661062] [instance: 1a447113-59d4-48f5-9ee8-3f7120e1a8fc] Migration operation thread notification thread_finished /opt/stack/new/nova/nova/virt/libvirt/driver.py:6230
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
    timer()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 168, in _do_send
    waiter.switch(result)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/opt/stack/new/nova/nova/utils.py", line 1053, in context_wrapper
    return func(*args, **kwargs)
  File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5759, in _live_migration_operation
    instance=instance)
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
    self.force_reraise()
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5755, in _live_migration_operation
    bandwidth=CONF.libvirt.live_migration_bandwidth)
  File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 525, in migrate
    destination, params=params, flags=flags)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
    result = proxy_call(self._autowrap, f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
    rv = execute(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
    six.reraise(c, e, tb)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
    rv = meth(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1833, in migrateToURI3
    if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
libvirtError: Operation not supported: Selecting disks to migrate is not implemented for tunnelled migration

It's most likely a problem with the job because CONF.libvirt.live_migration_tunnelled is not set, so it uses:

    cfg.StrOpt('block_migration_flag',
               default='VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, '
                       'VIR_MIGRATE_LIVE, VIR_MIGRATE_TUNNELLED, '
                       'VIR_MIGRATE_NON_SHARED_INC',
               help='Migration flags to be set for block migration',
               deprecated_for_removal=True,
               deprecated_reason='The correct block migration flags can be '
                                 'inferred from the new '
                                 'live_migration_tunnelled config option. '
                                 'block_migration_flag will be removed to '
                                 'avoid potential misconfiguration.'),

Which includes the VIR_MIGRATE_TUNNELLED flag, which apparently fails if you pass device_names to migrateToURI3.

Revision history for this message
Matt Riedemann (mriedem) wrote :

Looks like this is a duplicate of bug 1441054.

Changed in nova:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Matt Riedemann (mriedem) wrote :
Revision history for this message
Matt Riedemann (mriedem) wrote :

The driver checks in if it can perform live migration from the source host:

https://github.com/openstack/nova/blob/f5c9ebd56075f8eb04f9f0e683f85bacdcd68c38/nova/virt/libvirt/driver.py#L5360

That checks to see if there are attached block device mappings and libvirt<1.2.17 it won't work. But it also points out it won't work in tunneled mode, which is the bug we're hitting here.

Revision history for this message
Matt Riedemann (mriedem) wrote :

And I'm wondering if we should add the tunneled flag check to https://github.com/openstack/nova/blob/f5c9ebd56075f8eb04f9f0e683f85bacdcd68c38/nova/virt/libvirt/driver.py#L5360 and fail if it's setup to use tunneling, or if we should just avoid the selective block migration.

Revision history for this message
Matt Riedemann (mriedem) wrote :

tdurakov has a potential fix here: https://review.openstack.org/#/c/326111/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/326258

Changed in nova:
assignee: nobody → Timofey Durakov (tdurakov)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Timofey Durakov (<email address hidden>) on branch: master
Review: https://review.openstack.org/326258
Reason: there is patch on review already https://review.openstack.org/#/c/310707

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.