powervm error'd local volume cleanup fails

Bug #1180955 reported by Mathew Odden
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Matt Riedemann

Bug Description

During spawn, if a local volume for the backing block device fails to be created, the cleanup operation for the volume can fail with a "UnboundLocalError: local variable 'disk_name' referenced before assignment" error.

The fix is to assign to disk_name earlier and check for None before running cleanup.

Logs:

2013-05-16 13:40:34.521 ERROR nova.virt.powervm.blockdev [req-5bf9faeb-d14e-45d9-afc5-6c9e90e82257 4e06f68d356744f1a4e832e3c154c1d9 b3b27607effe4cc6a284ae4278d67a2a] Could not create logical volume. No space left on any volume group.
2013-05-16 13:40:34.522 ERROR nova.virt.powervm.blockdev [req-5bf9faeb-d14e-45d9-afc5-6c9e90e82257 4e06f68d356744f1a4e832e3c154c1d9 b3b27607effe4cc6a284ae4278d67a2a] Error while creating logical volume from image. Will attempt cleanup.
2013-05-16 13:40:34.523 ERROR nova.virt.powervm.blockdev [req-5bf9faeb-d14e-45d9-afc5-6c9e90e82257 4e06f68d356744f1a4e832e3c154c1d9 b3b27607effe4cc6a284ae4278d67a2a] Error while attempting cleanup of failed deploy to logical volume.
2013-05-16 13:40:34.523 18061 TRACE nova.virt.powervm.blockdev Traceback (most recent call last):
2013-05-16 13:40:34.523 18061 TRACE nova.virt.powervm.blockdev File "/usr/lib/python2.6/site-packages/nova/virt/powervm/blockdev.py", line 309, in create_volume_from_image
2013-05-16 13:40:34.523 18061 TRACE nova.virt.powervm.blockdev self.delete_volume(disk_name)
2013-05-16 13:40:34.523 18061 TRACE nova.virt.powervm.blockdev UnboundLocalError: local variable 'disk_name' referenced before assignment
2013-05-16 13:40:34.523 18061 TRACE nova.virt.powervm.blockdev
2013-05-16 13:40:34.525 ERROR nova.virt.powervm.operator [req-5bf9faeb-d14e-45d9-afc5-6c9e90e82257 4e06f68d356744f1a4e832e3c154c1d9 b3b27607effe4cc6a284ae4278d67a2a] PowerVM image creation failed: No space left on any volume group
2013-05-16 13:40:34.525 18061 TRACE nova.virt.powervm.operator Traceback (most recent call last):
2013-05-16 13:40:34.525 18061 TRACE nova.virt.powervm.operator File "/usr/lib/python2.6/site-packages/nova/virt/powervm/operator.py", line 345, in _create_image
2013-05-16 13:40:34.525 18061 TRACE nova.virt.powervm.operator context, instance, image_id)
2013-05-16 13:40:34.525 18061 TRACE nova.virt.powervm.operator File "/usr/lib/python2.6/site-packages/nova/virt/powervm/blockdev.py", line 313, in create_volume_from_image
2013-05-16 13:40:34.525 18061 TRACE nova.virt.powervm.operator LOG.exception(msg)
2013-05-16 13:40:34.525 18061 TRACE nova.virt.powervm.operator File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-05-16 13:40:34.525 18061 TRACE nova.virt.powervm.operator self.gen.next()
2013-05-16 13:40:34.525 18061 TRACE nova.virt.powervm.operator File "/usr/lib/python2.6/site-packages/nova/virt/powervm/blockdev.py", line 299, in create_volume_from_image
2013-05-16 13:40:34.525 18061 TRACE nova.virt.powervm.operator disk_name = self._create_logical_volume(size)
2013-05-16 13:40:34.525 18061 TRACE nova.virt.powervm.operator File "/usr/lib/python2.6/site-packages/nova/virt/powervm/blockdev.py", line 450, in _create_logical_volume
2013-05-16 13:40:34.525 18061 TRACE nova.virt.powervm.operator raise exception.PowerVMNoSpaceLeftOnVolumeGroup()
2013-05-16 13:40:34.525 18061 TRACE nova.virt.powervm.operator PowerVMNoSpaceLeftOnVolumeGroup: No space left on any volume group
2013-05-16 13:40:34.525 18061 TRACE nova.virt.powervm.operator

Tags: powervm
Mathew Odden (locke105)
Changed in nova:
assignee: nobody → Mathew Odden (locke105)
Matt Riedemann (mriedem)
Changed in nova:
assignee: Mathew Odden (locke105) → Matt Riedemann (mriedem)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/29991

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/29991
Committed: http://github.com/openstack/nova/commit/6ea7df2cbcdca6c9be0c44d2436e24549c499297
Submitter: Jenkins
Branch: master

commit 6ea7df2cbcdca6c9be0c44d2436e24549c499297
Author: Matt Riedemann <email address hidden>
Date: Tue May 21 13:33:19 2013 -0700

    Fix UnboundLocalError in powervm lvm cleanup code

    During spawn, if a local volume for the backing block device fails to
    be created, the cleanup operation for the volume can fail with a
    "UnboundLocalError: local variable 'disk_name' referenced before
    assignment" error.

    The fix is to assign to disk_name earlier and check for None before
    running cleanup.

    This patch also adds tests to cover the failure case.

    Fixes bug 1180955

    Change-Id: I0f32df1586d7bd87eccb674fec2a9597b1ce351d

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → havana-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: havana-1 → 2013.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.