OpenStack Compute (Nova)

can't terminate instance with attached volumes

Reported by dan kim on 2011-11-08
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Dean Troyer

Bug Description

When I try to terminate an instance with attached volumes, I've got following exceptions.

2011-11-08 08:58:51,553 DEBUG nova.utils [-] Running cmd (subprocess): sudo iscsiadm -m node -T iqn.2003-10.com.lefthandnetworks:lhn:533:volume-00000003 -p 192.168.1.10:3260 --op update -n node.startup -v manual from (pid=27971) execute /usr/lib/python2.7/dist-packages/nova/utils.py:168
2011-11-08 08:58:51,576 DEBUG nova.utils [-] Result was 255 from (pid=27971) execute /usr/lib/python2.7/dist-packages/nova/utils.py:184
2011-11-08 08:58:51,584 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 115, in wrapped
(nova.rpc): TRACE: return f(*args, **kw)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 122, in decorated_function
(nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 601, in terminate_instance
(nova.rpc): TRACE: self._delete_instance(context, instance_id)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 578, in _delete_instance
(nova.rpc): TRACE: self._shutdown_instance(context, instance_id, 'Terminating', True)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 564, in _shutdown_instance
(nova.rpc): TRACE: self.driver.destroy(instance, network_info, block_device_info, cleanup)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 348, in destroy
(nova.rpc): TRACE: mountpoint)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 371, in volume_driver_method
(nova.rpc): TRACE: return method(connection_info, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 147, in disconnect_volume
(nova.rpc): TRACE: self._iscsiadm_update(iscsi_properties, "node.startup", "manual")
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 82, in _iscsiadm_update
(nova.rpc): TRACE: return self._run_iscsiadm(iscsi_properties, iscsi_command)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/volume.py", line 74, in _run_iscsiadm
(nova.rpc): TRACE: *iscsi_command, run_as_root=True)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 192, in execute
(nova.rpc): TRACE: cmd=' '.join(cmd))
(nova.rpc): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova.rpc): TRACE: Command: sudo iscsiadm -m node -T iqn.2003-10.com.lefthandnetworks:lhn:533:volume-00000003 -p 192.168.1.10:3260 --op update -n node.startup -v manual
(nova.rpc): TRACE: Exit code: 255
(nova.rpc): TRACE: Stdout: ''
(nova.rpc): TRACE: Stderr: 'iscsiadm: no records found!\n'

----

I've checked codes, and it looks the cause is that nova-compute tries to disconnect volume twice.
first one runs when detaching volumes, second one runs when destroy instance.
So second try makes the excepion because there's no iscsi device, it already removed.

Please check this.

Thanks.

dan kim (dan-kim) on 2011-11-08
description: updated
Thierry Carrez (ttx) on 2011-11-08
Changed in nova:
importance: Undecided → High
status: New → Confirmed
Dean Troyer (dtroyer) wrote :

The problem is not recognizing the exit code 255 as valid on the second call to disconnect_volume.

Proposed Icffeb0fe82 to not check exit codes on logout/delete.

Changed in nova:
assignee: nobody → Dean Troyer (dtroyer)
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/2326
Committed: http://github.com/openstack/nova/commit/ed1b801a2cc3691cdab6ca2ae95f7299a964d00f
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit ed1b801a2cc3691cdab6ca2ae95f7299a964d00f
Author: Dean Troyer <email address hidden>
Date: Tue Dec 13 19:05:21 2011 -0600

    Fixes bug 887402

    Change utils.execute to accept a list or a single integer in
    check_exit_code.

    In libvirt.disconnect_volume() return codes 0 and 255 are both valid
    for logout/delete, where 255 is returned if the volume is already
    disconnected.

    Change-Id: Icffeb0fe8269a02d95ac6ed180ba0bb9f458a6ed

Changed in nova:
status: In Progress → Fix Committed
Devin Carlen (devcamcar) wrote :

This is a critical issue that will need to be backported to essex2 immediately.

Reviewed: https://review.openstack.org/2346
Committed: http://github.com/openstack/nova/commit/31a7924ecfae0b9c9fea0edc344f0e3ca2fe78a5
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 31a7924ecfae0b9c9fea0edc344f0e3ca2fe78a5
Author: Vishvananda Ishaya <email address hidden>
Date: Wed Dec 14 14:03:04 2011 -0800

    Refactors handling of detach volume

     * removes unnecessary flags in detach_volume call
     * stops double detach reported in bug 887402
     * moves volume.API() into init

    Change-Id: I65332cabedf2edb88acb48b3293cba291d440238

Thierry Carrez (ttx) on 2011-12-15
Changed in nova:
milestone: none → essex-2

Reviewed: https://review.openstack.org/2365
Committed: http://github.com/openstack/nova/commit/44955a1937ae25514bfe4ee5ccb561642681ab1c
Submitter: Jenkins
Branch: milestone-proposed

 tag in-milestone-proposed
 done

commit 44955a1937ae25514bfe4ee5ccb561642681ab1c
Author: Dean Troyer <email address hidden>
Date: Tue Dec 13 19:05:21 2011 -0600

    Fixes bug 887402

    Change utils.execute to accept a list or a single integer in
    check_exit_code.

    In libvirt.disconnect_volume() return codes 0 and 255 are both valid
    for logout/delete, where 255 is returned if the volume is already
    disconnected.

    Also includes a fix by Dan Prince for a regression introduced by
    the above fix (bug 904560)

    Change-Id: Icffeb0fe8269a02d95ac6ed180ba0bb9f458a6ed

Thierry Carrez (ttx) on 2011-12-15
Changed in nova:
status: Fix Committed → Fix Released
Mark McLoughlin (markmc) wrote :

Is this applicable to Diablo? Anyone care to backport it?

Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-2 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers