OpenStack Compute (nova)

Libvirt driver broken for non-disk-image backends

Bug #1555287 reported by Dan Smith on 2016-03-09

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
OpenStack Compute (nova)	Fix Released	High	Matthew Booth	OpenStack Compute (nova) mitaka-rc1 "RC1"
Kilo	Fix Released	High	Tristan Cacqueray	OpenStack Compute (nova) 2015.1.4
Liberty	Fix Released	High	Tristan Cacqueray

Bug Description

Recently the ceph job (and any other configuration that doesn't use disk image as the backend storage) started failing like this:

2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/exception.py", line 110, in wrapped
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher payload)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/exception.py", line 89, in wrapped
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 359, in decorated_function
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 328, in decorated_function
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 409, in decorated_function
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 316, in decorated_function
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher migration.instance_uuid, exc_info=True)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 293, in decorated_function
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 387, in decorated_function
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 375, in decorated_function
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/compute/manager.py", line 3876, in resize_instance
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher timeout, retry_interval)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7237, in migrate_disk_and_power_off
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher shared_storage)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7230, in migrate_disk_and_power_off
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher host=dest, on_execute=on_execute,
2016-03-09 14:47:29.102 17597 ERROR oslo_messaging.rpc.dispatcher UnboundLocalError: local variable 'on_execute' referenced before assignment

Tags:

OpenStack Infra (hudson-openstack) on 2016-03-09

Changed in nova:
assignee:	nobody → Dan Smith (danms)
status:	Confirmed → In Progress

Matt Riedemann (mriedem) on 2016-03-09

tags:

added: ceph mitaka-rc-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-09: Fix proposed to nova (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/290843

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-09: Fix proposed to nova (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/290847

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-09: Fix merged to nova (master)

Reviewed: https://review.openstack.org/290715
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ccd6195e67c735aa216bea51d9dccfed631dc814
Submitter: Jenkins
Branch: master

commit ccd6195e67c735aa216bea51d9dccfed631dc814
Author: Matthew Booth <email address hidden>
Date: Wed Mar 9 17:27:03 2016 +0000

Fix processing of libvirt disk.info in non-disk-image cases

    In Idfc16f54049aaeab31ac1c1d8d79a129acc9fb87 a change was made
    that caused non-disk-image backends to fall over because of an
    undefined variable because they skipped processing of the disk.info
    file. This adds a check for that case to make sure we don't run
    that path in the non-disk-image backend case.

Closes-Bug: #1555287

Change-Id: I02f8a5f0e29816336e500a8fe8dcc9ece15968e9

Changed in nova:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-10: Fix merged to nova (stable/kilo)

Reviewed: https://review.openstack.org/290847
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2d7166db25c2d6f5147f4da2c6067895420d5995
Submitter: Jenkins
Branch: stable/kilo

commit 2d7166db25c2d6f5147f4da2c6067895420d5995
Author: Matthew Booth <email address hidden>
Date: Wed Mar 9 17:27:03 2016 +0000

Fix processing of libvirt disk.info in non-disk-image cases

Closes-Bug: #1555287

Change-Id: I02f8a5f0e29816336e500a8fe8dcc9ece15968e9

tags:

added: in-stable-kilo

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-10: Fix merged to nova (stable/liberty)

Reviewed: https://review.openstack.org/290843
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e8dc5c876cbade81a0a2991a8c100bc9afc0e42a
Submitter: Jenkins
Branch: stable/liberty

commit e8dc5c876cbade81a0a2991a8c100bc9afc0e42a
Author: Matthew Booth <email address hidden>
Date: Wed Mar 9 17:27:03 2016 +0000

Fix processing of libvirt disk.info in non-disk-image cases

Closes-Bug: #1555287

Change-Id: I02f8a5f0e29816336e500a8fe8dcc9ece15968e9

tags:

added: in-stable-liberty

Matt Riedemann (mriedem) on 2016-03-10

Changed in nova:
assignee:	Dan Smith (danms) → Matthew Booth (mbooth-9)

Revision history for this message

Thierry Carrez (ttx) wrote on 2016-03-16: Fix included in openstack/nova 13.0.0.0rc1

This issue was fixed in the openstack/nova 13.0.0.0rc1 release candidate.

Matt Riedemann (mriedem) on 2016-03-21

tags:

removed: mitaka-rc-potential

Revision history for this message

Davanum Srinivas (DIMS) (dims-v) wrote on 2016-04-14: Fix included in openstack/nova 12.0.3

This issue was fixed in the openstack/nova 12.0.3 release.

Revision history for this message

Doug Hellmann (doug-hellmann) wrote on 2016-05-10: Fix included in openstack/nova 2015.1.4

This issue was fixed in the openstack/nova 2015.1.4 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-10:

This issue was fixed in the openstack/nova 2015.1.4 release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.