can't terminate instance with attached volumes

[dan kim]

When I try to terminate an instance with attached volumes, I've got following exceptions.

2011-11-08 08:58:51,553 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2003-10.com.lefthandnetworks:lhn:533:volume-00000003 -p 192.168.1.10:3260 --op update -n node.startup -v manual from (pid=27971) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-11-08 08:58:51,576 DEBUG nova.utils [-] Result was 255 from (pid=27971) execute /usr/lib/python2.7/dist-packages/nova/utils.py:184
2011-11-08 08:58:51,584 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 115, in wrapped
(nova.rpc): TRACE: return f(*args, **kw)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 122, in decorated_function
(nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 601, in terminate_instance
(nova.rpc): TRACE: self._delete_instance(context, instance_id)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 578, in _delete_instance
(nova.rpc): TRACE: self._shutdown_instance(context, instance_id, 'Terminating', True)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 564, in _shutdown_instance
(nova.rpc): TRACE: self.driver.destroy(instance, network_info, block_device_info, cleanup)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 348, in destroy
(nova.rpc): TRACE: mountpoint)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 371, in volume_driver_method
(nova.rpc): TRACE: return method(connection_info, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 147, in disconnect_volume
(nova.rpc): TRACE: self._iscsiadm_update(iscsi_properties, "node.startup", "manual")
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 82, in _iscsiadm_update
(nova.rpc): TRACE: return self._run_iscsiadm(iscsi_properties, iscsi_command)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 74, in _run_iscsiadm
(nova.rpc): TRACE: *iscsi_command, run_as_root=True)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 192, in execute
(nova.rpc): TRACE: cmd=' '.join(cmd))
(nova.rpc): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova.rpc): TRACE: Command: sudo iscsiadm -m node -T iqn.2003-10.com.lefthandnetworks:lhn:533:volume-00000003 -p 192.168.1.10:3260 --op update -n node.startup -v manual
(nova.rpc): TRACE: Exit code: 255
(nova.rpc): TRACE: Stdout: ''
(nova.rpc): TRACE: Stderr: 'iscsiadm: no records found!\n'

----

I've checked codes, and it looks the cause is that nova-compute tries to disconnect volume twice.
first one runs when detaching volumes, second one runs when destroy instance.
So second try makes the excepion because there's no iscsi device, it already removed.

Please check this.

Thanks.

[Dean Troyer (dtroyer)]

The problem is not recognizing the exit code 255 as valid on the second call to disconnect_volume.

Proposed Icffeb0fe82 to not check exit codes on logout/delete.

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/2326
Committed: http://github.com/openstack/nova/commit/ed1b801a2cc3691cdab6ca2ae95f7299a964d00f
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit ed1b801a2cc3691cdab6ca2ae95f7299a964d00f
Author: Dean Troyer <email address hidden>
Date: Tue Dec 13 19:05:21 2011 -0600

    Fixes bug 887402

    Change utils.execute to accept a list or a single integer in
    check_exit_code.

    In libvirt.disconnect_volume() return codes 0 and 255 are both valid
    for logout/delete, where 255 is returned if the volume is already
    disconnected.

    Change-Id: Icffeb0fe8269a02d95ac6ed180ba0bb9f458a6ed

[Devin Carlen (devcamcar)]

This is a critical issue that will need to be backported to essex2 immediately.

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/2346
Committed: http://github.com/openstack/nova/commit/31a7924ecfae0b9c9fea0edc344f0e3ca2fe78a5
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 31a7924ecfae0b9c9fea0edc344f0e3ca2fe78a5
Author: Vishvananda Ishaya <email address hidden>
Date: Wed Dec 14 14:03:04 2011 -0800

    Refactors handling of detach volume

     * removes unnecessary flags in detach_volume call
     * stops double detach reported in bug 887402
     * moves volume.API() into init

    Change-Id: I65332cabedf2edb88acb48b3293cba291d440238

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/2365
Committed: http://github.com/openstack/nova/commit/44955a1937ae25514bfe4ee5ccb561642681ab1c
Submitter: Jenkins
Branch: milestone-proposed

 tag in-milestone-proposed
 done

commit 44955a1937ae25514bfe4ee5ccb561642681ab1c
Author: Dean Troyer <email address hidden>
Date: Tue Dec 13 19:05:21 2011 -0600

    Fixes bug 887402

    Change utils.execute to accept a list or a single integer in
    check_exit_code.

    In libvirt.disconnect_volume() return codes 0 and 255 are both valid
    for logout/delete, where 255 is returned if the volume is already
    disconnected.

    Also includes a fix by Dan Prince for a regression introduced by
    the above fix (bug 904560)

    Change-Id: Icffeb0fe8269a02d95ac6ed180ba0bb9f458a6ed

[Mark McLoughlin (markmc)]

Is this applicable to Diablo? Anyone care to backport it?