Nova fails live migrations on dedicated interface due to wrong type of migrate_uri

Bug #1638625 reported by Pawel Koniszewski
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Pawel Koniszewski

Bug Description

Nova fails all live migrations on dedicated interface due to conversion of string to unicode (in Python 2.7).

My environment:
* nova trunk, commit 40f9b0ad16d3a5fae184bbe6d4a49cf792967089
* QEMU 2.6
* Libvirt 1.3.4

1. Set live_migration_inbound_addr to IP address of existing interface
2. Make sure that live_migration_tunnelled is set to False
3. Restart nova-compute
4. Try to live migrate any instance
5. Outcome:

2016-11-02 12:26:04.272 DEBUG nova.virt.libvirt.driver [req-f3cc94e2-4b45-4236-8d1b-888343ae6885 admin demo] [instance: 09b7c15c-7a5c-4e86-8c27-96d2edc5094c] Migration operation thread notification from (pid=8533) thread_finished /opt/stack/nova/nova/virt/libvirt/driver.py:6325
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
    timer()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 168, in _do_send
    waiter.switch(result)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/opt/stack/nova/nova/utils.py", line 1066, in context_wrapper
    return func(*args, **kwargs)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5926, in _live_migration_operation
    instance=instance)
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5922, in _live_migration_operation
    bandwidth=CONF.libvirt.live_migration_bandwidth)
  File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 571, in migrate
    destination, params=params, flags=flags)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
    result = proxy_call(self._autowrap, f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
    rv = execute(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
    six.reraise(c, e, tb)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
    rv = meth(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1915, in migrateToURI3
    ret = libvirtmod.virDomainMigrateToURI3(self._o, dconnuri, params, flags)
TypeError: Unknown type of "migrate_uri" field

This happens to me everytime when trying to live migrate any VM over dedicated interface using nova trunk.

tags: added: live-migration
Changed in nova:
status: New → Confirmed
importance: Undecided → High
Changed in nova:
assignee: nobody → Timofey Durakov (tdurakov)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/398956

Changed in nova:
assignee: Timofey Durakov (tdurakov) → Pawel Koniszewski (pawel-koniszewski)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/398956
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=90f5a727e98a59c648c0dc4222050857d59edab2
Submitter: Jenkins
Branch: master

commit 90f5a727e98a59c648c0dc4222050857d59edab2
Author: Pawel Koniszewski <email address hidden>
Date: Tue Nov 29 17:13:37 2016 +0100

    Convert live migration uri back to string

    If live_migration_inbound_addr is set in nova.conf, value of the option
    will be saved under target_connect_addr in migrate_data. Unfortunately,
    the type of value saved in target_connect_addr is unicode, therefore
    when creating URI for live migration over dedicated interface we are
    ending up with unicode type which is not acceptable by libvirt. To fix
    the issue let's convert URI explictly back to string.

    Change-Id: I651cc7a45fe06a2b6d81e007c4faf62947ee3203
    Closes-Bug: #1638625

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/404810

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/404810
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a098e72c1977ceaff087a0d071e930940878feee
Submitter: Jenkins
Branch: stable/newton

commit a098e72c1977ceaff087a0d071e930940878feee
Author: Pawel Koniszewski <email address hidden>
Date: Tue Nov 29 17:13:37 2016 +0100

    Convert live migration uri back to string

    If live_migration_inbound_addr is set in nova.conf, value of the option
    will be saved under target_connect_addr in migrate_data. Unfortunately,
    the type of value saved in target_connect_addr is unicode, therefore
    when creating URI for live migration over dedicated interface we are
    ending up with unicode type which is not acceptable by libvirt. To fix
    the issue let's convert URI explictly back to string.

    Change-Id: I651cc7a45fe06a2b6d81e007c4faf62947ee3203
    Closes-Bug: #1638625
    (cherry picked from commit 90f5a727e98a59c648c0dc4222050857d59edab2)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.0.0b2

This issue was fixed in the openstack/nova 15.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.0.3

This issue was fixed in the openstack/nova 14.0.3 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.