can not delete an instance if the instance's rescue volume is not found

Bug #1404791 reported by Eli Qiao
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
Alexis Lee

Bug Description

can not delete an instance if the instance's rescue lvm can not be found.

how to reproduce:

1. configure images_type lvm for libvirt dirver.
[libvirt]
images_type = lvm <----
images_volume_group = stack-volumes-lvmdriver-1 <-- lvm used

2. rescue the instance, will generate uuid.rescue lvm
3. unrescue the instance, the rescue lvm can not be deleted due to bug https://bugs.launchpad.net/nova/+bug/1385480
4. delete the uuid.rescue lvm manually.
5. delete the instance(failed and set to error state)

below is the call trace of nova-compute driver.

2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher self._cleanup_lvm(instance)
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 944, in _cleanup_lvm
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher lvm.remove_volumes(disks)
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/lvm.py", line 272, in remove_volumes
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher clear_volume(path)
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/lvm.py", line 250, in clear_volume
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher volume_size = get_volume_size(path)
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/lvm.py", line 66, in decorated_function
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher return function(path)
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/lvm.py", line 197, in get_volume_size
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher run_as_root=True)
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/utils.py", line 53, in execute
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher return utils.execute(*args, **kwargs)
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/utils.py", line 164, in execute
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher return processutils.execute(*cmd, **kwargs)
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 224, in execute
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher cmd=sanitized_cmd)
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher Command: sudo nova-rootwrap /etc/nova/rootwrap.conf blockdev --getsize64 /dev/stack-volumes-lvmdriver-1/b09687ee-f525-4edc-aaf4-1272562d46fd_disk.rescue
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher Exit code: 1
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher Stdout: u''
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher Stderr: u'blockdev: cannot open /dev/stack-volumes-lvmdriver-1/b09687ee-f525-4edc-aaf4-1272562d46fd_disk.rescue: No such file or directory\n'
2014-12-22 13:18:29.691 TRACE oslo.messaging.rpc.dispatcher

Eli Qiao (taget-9)
Changed in nova:
assignee: nobody → Eli Qiao (taget-9)
Revision history for this message
Eli Qiao (taget-9) wrote :

beside , in this case the error instance can not be deleted any more.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/143374

Changed in nova:
status: New → In Progress
Changed in nova:
importance: Undecided → Low
Changed in nova:
assignee: Eli Qiao (taget-9) → Alexis Lee (alexisl)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/143374
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=11fb3936a1ad01af2a96a4eb4c00d1b158bd6f40
Submitter: Jenkins
Branch: master

commit 11fb3936a1ad01af2a96a4eb4c00d1b158bd6f40
Author: Eli Qiao <email address hidden>
Date: Mon Feb 16 11:23:42 2015 +0800

    Libvirt: Allow missing volumes during delete

    When deleting an instance, libvirt driver will clear all the volumes
    of the instance. Ignore exception when clearing a missing volume
    to avoid delete instance error.

    Closes-Bug: 1404791
    Change-Id: I8a663e8f0396bf2303642610ae1001c330159b75

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → kilo-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: kilo-3 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.