Libvirt driver broken for non-disk-image backends

Bug #1555287 reported by Dan Smith on 2016-03-09
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Matthew Booth
Kilo
High
Tristan Cacqueray
Liberty
High
Tristan Cacqueray

Bug Description

Recently the ceph job (and any other configuration that doesn't use disk image as the backend storage) started failing like this:

2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/exception.py", line 110, in wrapped
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher payload)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/exception.py", line 89, in wrapped
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 359, in decorated_function
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 328, in decorated_function
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 409, in decorated_function
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 316, in decorated_function
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher migration.instance_uuid, exc_info=True)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 293, in decorated_function
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 387, in decorated_function
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 375, in decorated_function
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 3876, in resize_instance
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher timeout, retry_interval)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7237, in migrate_disk_and_power_off
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher shared_storage)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7230, in migrate_disk_and_power_off
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher host=dest, on_execute=on_execute,
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher UnboundLocalError: local variable 'on_execute' referenced before assignment

Changed in nova:
assignee: nobody → Dan Smith (danms)
status: Confirmed → In Progress
Matt Riedemann (mriedem) on 2016-03-09
tags: added: ceph mitaka-rc-potential

Reviewed: https://review.openstack.org/290715
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ccd6195e67c735aa216bea51d9dccfed631dc814
Submitter: Jenkins
Branch: master

commit ccd6195e67c735aa216bea51d9dccfed631dc814
Author: Matthew Booth <email address hidden>
Date: Wed Mar 9 17:27:03 2016 +0000

    Fix processing of libvirt disk.info in non-disk-image cases

    In Idfc16f54049aaeab31ac1c1d8d79a129acc9fb87 a change was made
    that caused non-disk-image backends to fall over because of an
    undefined variable because they skipped processing of the disk.info
    file. This adds a check for that case to make sure we don't run
    that path in the non-disk-image backend case.

    Closes-Bug: #1555287

    Change-Id: I02f8a5f0e29816336e500a8fe8dcc9ece15968e9

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/290847
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2d7166db25c2d6f5147f4da2c6067895420d5995
Submitter: Jenkins
Branch: stable/kilo

commit 2d7166db25c2d6f5147f4da2c6067895420d5995
Author: Matthew Booth <email address hidden>
Date: Wed Mar 9 17:27:03 2016 +0000

    Fix processing of libvirt disk.info in non-disk-image cases

    In Idfc16f54049aaeab31ac1c1d8d79a129acc9fb87 a change was made
    that caused non-disk-image backends to fall over because of an
    undefined variable because they skipped processing of the disk.info
    file. This adds a check for that case to make sure we don't run
    that path in the non-disk-image backend case.

    Closes-Bug: #1555287

    Change-Id: I02f8a5f0e29816336e500a8fe8dcc9ece15968e9

tags: added: in-stable-kilo

Reviewed: https://review.openstack.org/290843
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e8dc5c876cbade81a0a2991a8c100bc9afc0e42a
Submitter: Jenkins
Branch: stable/liberty

commit e8dc5c876cbade81a0a2991a8c100bc9afc0e42a
Author: Matthew Booth <email address hidden>
Date: Wed Mar 9 17:27:03 2016 +0000

    Fix processing of libvirt disk.info in non-disk-image cases

    In Idfc16f54049aaeab31ac1c1d8d79a129acc9fb87 a change was made
    that caused non-disk-image backends to fall over because of an
    undefined variable because they skipped processing of the disk.info
    file. This adds a check for that case to make sure we don't run
    that path in the non-disk-image backend case.

    Closes-Bug: #1555287

    Change-Id: I02f8a5f0e29816336e500a8fe8dcc9ece15968e9

tags: added: in-stable-liberty
Matt Riedemann (mriedem) on 2016-03-10
Changed in nova:
assignee: Dan Smith (danms) → Matthew Booth (mbooth-9)

This issue was fixed in the openstack/nova 13.0.0.0rc1 release candidate.

Matt Riedemann (mriedem) on 2016-03-21
tags: removed: mitaka-rc-potential

This issue was fixed in the openstack/nova 12.0.3 release.

This issue was fixed in the openstack/nova 2015.1.4 release.

This issue was fixed in the openstack/nova 2015.1.4 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers