cannot delete nova instance if volume is not active

Bug #1482066 reported by Eli Qiao
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Eli Qiao
Liberty
Fix Released
High
Matt Riedemann

Bug Description

I booted from an encrypted volume, when try to delete it, it failed

'Device ip-10.238.157.47:3260-iscsi-iqn.2010-10.org.openstack:volume-d167a0ac-fab5-484a-865d-667f1583c2ab-lun-1 is not active.\n'

| fault | {"message": "Unexpected error while running command. |
| | Command: sudo nova-rootwrap /etc/nova/rootwrap.conf cryptsetup luksClose ip-10.238.157.47:3260-iscsi-iqn.2010-10.org.openstack:volume-d167a0ac-fab5-484a-865d-667f1583c2ab-lun-1 |
| | Exit code: 4 |
| | Stdout: u'' |
| | Stderr: u'Dev", "code": 500, "details": " File \"/opt/stack/nova/nova/compute/manager.py\", line 351, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File \"/opt/stack/nova/nova/compute/manager.py\", line 2370, in terminate_instance |
| | do_terminate_instance(instance, bdms) |
| | File \"/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py\", line 252, in inner |
| | return f(*args, **kwargs) |
| | File \"/opt/stack/nova/nova/compute/manager.py\", line 2368, in do_terminate_instance |
| | self._set_instance_error_state(context, instance) |
| | File \"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py\", line 119, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/opt/stack/nova/nova/compute/manager.py\", line 2358, in do_terminate_instance |
| | self._delete_instance(context, instance, bdms, quotas) |
| | File \"/opt/stack/nova/nova/hooks.py\", line 149, in inner |
| | rv = f(*args, **kwargs) |
| | File \"/opt/stack/nova/nova/compute/manager.py\", line 2337, in _delete_instance |
| | quotas.rollback() |
| | File \"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py\", line 119, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/opt/stack/nova/nova/compute/manager.py\", line 2307, in _delete_instance |
| | self._shutdown_instance(context, instance, bdms) |
| | File \"/opt/stack/nova/nova/compute/manager.py\", line 2234, in _shutdown_instance |
| | requested_networks) |
| | File \"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py\", line 119, in __exit__ |
| | six.reraise(self.type_, self.value, self.tb) |
| | File \"/opt/stack/nova/nova/compute/manager.py\", line 2223, in _shutdown_instance |
| | block_device_info) |
| | File \"/opt/stack/nova/nova/virt/libvirt/driver.py\", line 768, in destroy |
| | destroy_disks, migrate_data) |
| | File \"/opt/stack/nova/nova/virt/libvirt/driver.py\", line 844, in cleanup |
| | encryptor.detach_volume(**encryption) |
| | File \"/opt/stack/nova/nova/volume/encryptors/cryptsetup.py\", line 101, in detach_volume |
| | self._close_volume(**kwargs) |
| | File \"/opt/stack/nova/nova/volume/encryptors/luks.py\", line 128, in _close_volume |
| | attempts=3) |
| | File \"/opt/stack/nova/nova/utils.py\", line 229, in execute |
| | return processutils.execute(*cmd, **kwargs) |
| | File \"/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py\", line 275, in execute |
| | cmd=sanitized_cmd) |
| | ", "created": "2015-08-06T03:30:52Z"} |

Eli Qiao (taget-9)
Changed in nova:
assignee: nobody → Eli Qiao (taget-9)
jichenjc (jichenjc)
Changed in nova:
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/215252

Changed in nova:
status: Confirmed → In Progress
Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → High
tags: added: volumes
tags: added: liberty-rc-potential
Revision history for this message
Matt Riedemann (mriedem) wrote :

I don't think this is a regression in liberty, so marking as liberty-backport-potential so it's not blocking the liberty rc1.

tags: added: liberty-backport-potential
removed: liberty-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/215252
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=11e20dd6f91d2417e45b44d8c9008b07ebdcec28
Submitter: Jenkins
Branch: master

commit 11e20dd6f91d2417e45b44d8c9008b07ebdcec28
Author: Eli Qiao <email address hidden>
Date: Tue Oct 13 14:08:31 2015 +0800

    Ignore errorcode=4 when executing `cryptsetup remove` command

    If an attached encrypted volume is failed to detach from the instance
    when deleting that instance, user can not delete that instance at all.

    This patch adds 4 in check_exit_code when executing `cryptsetup remove`
    command to eat that exception.

    PS: exit_code = 4 indicate ENODEV error which means no device(also includes
    the crypt device inactive).

    Closes-Bug: #1482066
    Change-Id: I12e2a52068850528a4bd68486344b74eb9b82c88

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/nova 13.0.0.0b1

This issue was fixed in the openstack/nova 13.0.0.0b1 development milestone.

Changed in nova:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/288480

Matt Riedemann (mriedem)
tags: removed: liberty-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/liberty)

Reviewed: https://review.openstack.org/288480
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=20c3335478de9495c55ce63a3a21fab97de1642f
Submitter: Jenkins
Branch: stable/liberty

commit 20c3335478de9495c55ce63a3a21fab97de1642f
Author: Eli Qiao <email address hidden>
Date: Tue Oct 13 14:08:31 2015 +0800

    Ignore errorcode=4 when executing `cryptsetup remove` command

    If an attached encrypted volume is failed to detach from the instance
    when deleting that instance, user can not delete that instance at all.

    This patch adds 4 in check_exit_code when executing `cryptsetup remove`
    command to eat that exception.

    PS: exit_code = 4 indicate ENODEV error which means no device(also includes
    the crypt device inactive).

    Closes-Bug: #1482066
    Change-Id: I12e2a52068850528a4bd68486344b74eb9b82c88
    (cherry picked from commit 11e20dd6f91d2417e45b44d8c9008b07ebdcec28)

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote : Fix included in openstack/nova 12.0.3

This issue was fixed in the openstack/nova 12.0.3 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.