OpenStack Compute (nova)

Instance stuck in ACTIVE/deleting if detach volume failure

Bug #1172695 reported by MotoKen
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Won't Fix
Undecided
Unassigned

Bug Description

When deleting an Instance with attached volume, it stuck in ACTIVE/deleting status if detach volume failure.

To reproduce:
1. boot an instance and attach a volume.
2. stop cinder-api
3. delete the instance
4. instance stuck in ACTIVE/deleting

Traceback:
2013-04-25 20:13:58.348 ERROR nova.openstack.common.rpc.amqp [req-14043ff3-79d0-4c6c-9394-9ac8f53c84d5 1458120d63784bb49858437c275dc59e 7dd8607e7d294efc845ccf68f8d282b7] Exception during message handling
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1383, in terminate_instance
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp do_terminate_instance(instance, bdms)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 242, in inner
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1375, in do_terminate_instance
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp reservations=reservations)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 85, in inner
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp rv = f(*args, **kwargs)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1338, in _delete_instance
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp project_id=project_id)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1311, in _delete_instance
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp self._shutdown_instance(context, instance, bdms)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1259, in _shutdown_instance
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp volume = self.volume_api.get(context, bdm['volume_id'])
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 193, in get
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp self._reraise_translated_volume_exception(volume_id)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 190, in get
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp item = cinderclient(context).volumes.get(volume_id)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinderclient/v1/volumes.py", line 164, in get
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp return self._get("/volumes/%s" % volume_id, "volume")
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 141, in _get
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp resp, body = self.api.client.get(url)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 189, in get
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp return self._cs_request(url, 'GET', **kwargs)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 157, in _cs_request
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp **kwargs)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 127, in request
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp **kwargs)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/requests/api.py", line 40, in request
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp return s.request(method=method, url=url, **kwargs)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 229, in request
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp r.send(prefetch=prefetch)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/requests/models.py", line 605, in send
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp raise ConnectionError(e)
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8776): Max retries exceeded with url: /v1/7dd8607e7d294efc845ccf68f8d282b7/volumes/71e67805-8380-4971-952f-9091997b3a6b
2013-04-25 20:13:58.348 18464 TRACE nova.openstack.common.rpc.amqp

When exception is raised during detaching volume then the instance stuck in ACTIVE(deleting). The instance should probably go to ERROR.

Liang Wang (hiliang56)
Changed in nova:
assignee: nobody → Liang Wang (hiliang56)
status: New → In Progress
Liang Wang (hiliang56)
Changed in nova:
status: In Progress → New
Liang Wang (hiliang56)
Changed in nova:
assignee: Liang Wang (hiliang56) → nobody
Rohit Karajgi (rohitk)
Changed in nova:
assignee: nobody → Rohit Karajgi (rohitkarajgi)
Michael Still (mikal)
Changed in nova:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
haruka tanizawa (h-tanizawa) wrote :

This stuck can be reproduced in master.
In my case, it is ERROR/deleting.
In either case, stuck and can't be deleted with delete
command is big problem.

Revision history for this message
Yoshiaki Kosaka (yoshiaki-kousaka) wrote :

Hi MotoKen,

I'm curious about this error.
I couldn't reproduce on my environment.
Do you know there is any specific situation to face it ?

Thanks.

Revision history for this message
haruka tanizawa (h-tanizawa) wrote :

I could not reproduce this problem anymore.
I guess this is fixed by this bug report https://bugs.launchpad.net/nova/+bug/1332198 .
Motoken(or any other ATC), if you see same as me, it would be nice to be closed :)

Revision history for this message
MotoKen (motokentsai) wrote :

The instance will become ERROR now in my Icehouse environment.

Revision history for this message
melanie witt (melwitt) wrote :

I'm going to mark this as Won't Fix as the submitter has confirmed this is no longer an issue in icehouse (thanks MotoKen!).

Changed in nova:
assignee: Rohit Karajgi (rohitk) → nobody
importance: Medium → Undecided
status: Confirmed → Won't Fix
Revision history for this message
Takashi Natsume (natsume-takashi) wrote :

This bug has been fixed by the following bug report.

Failure to power off a VM during delete leads to it going back to Active(None)
https://bugs.launchpad.net/nova/+bug/1254122

Cannot delete an instance that failed a previous delete
https://bugs.launchpad.net/nova/+bug/1329559

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.