run_instance should not traceback if instance is deleted

Bug #1154292 reported by Dan Smith
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Dan Smith

Bug Description

Right now, if an instance is deleted before we grab the lock to go run it, we harmlessly but loudly traceback on the failure:

2013-03-12 18:57:11.487 DEBUG nova.compute.manager [req-c65d021f-ef0e-42f0-957d-f98f0d2df5ed ListServersNegativeTest549863-user ListServersNegativeTest549863-tenant] [instance: 6799974f-2cad-4cf1-8b67-20e421180b0f] Instance has been destroyed from under us while trying to set it to ERROR _set_instance_error_state /opt/stack/new/nova/nova/compute/manager.py:388
2013-03-12 18:57:11.723 ERROR nova.openstack.common.rpc.amqp [req-c65d021f-ef0e-42f0-957d-f98f0d2df5ed ListServersNegativeTest549863-user ListServersNegativeTest549863-tenant] Exception during message handling
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/exception.py", line 117, in wrapped
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/exception.py", line 94, in wrapped
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 210, in decorated_function
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp pass
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 196, in decorated_function
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 261, in decorated_function
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 225, in decorated_function
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 1168, in run_instance
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp do_run_instance()
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/lockutils.py", line 242, in inner
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 1167, in do_run_instance
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp admin_password, is_first_time, node, instance)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 826, in _run_instance
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp self._set_instance_error_state(context, instance['uuid'])
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 762, in _run_instance
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp self._start_building(context, instance)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 994, in _start_building
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp None))
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 372, in _instance_update
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp **kwargs)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/conductor/api.py", line 380, in instance_update
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp updates, 'conductor')
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/conductor/rpcapi.py", line 107, in instance_update
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp version='1.38')
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/proxy.py", line 80, in call
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp return rpc.call(context, self._get_topic(topic), msg, timeout)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/__init__.py", line 140, in call
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp return _get_impl().call(CONF, context, topic, msg, timeout)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/impl_kombu.py", line 798, in call
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp rpc_amqp.get_connection_pool(conf, Connection))
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/amqp.py", line 613, in call
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp rv = list(rv)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/amqp.py", line 562, in __iter__
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp raise result
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp InstanceNotFound_Remote: Instance 6799974f-2cad-4cf1-8b67-20e421180b0f could not be found.
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/common.py", line 407, in catch_client_exception
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp return func(*args, **kwargs)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/conductor/manager.py", line 96, in instance_update
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp context, instance_uuid, updates)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/db/api.py", line 686, in instance_update_and_get_original
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp rv = IMPL.instance_update_and_get_original(context, instance_uuid, values)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 112, in wrapper
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1767, in instance_update_and_get_original
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp copy_old_instance=True)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1802, in _instance_update
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp session=session)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 112, in wrapper
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 1492, in _instance_get_by_uuid
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp raise exception.InstanceNotFound(instance_id=uuid)
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp InstanceNotFound: Instance 6799974f-2cad-4cf1-8b67-20e421180b0f could not be found.
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp
2013-03-12 18:57:11.723 10906 TRACE nova.openstack.common.rpc.amqp

Tags: stacktrace
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/24243

Changed in nova:
status: Confirmed → In Progress
Sean Dague (sdague)
tags: added: stacktrace
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/24243
Committed: http://github.com/openstack/nova/commit/4814280ce0af71ad38155d07eefea7c762935fa7
Submitter: Jenkins
Branch: master

commit 4814280ce0af71ad38155d07eefea7c762935fa7
Author: Dan Smith <email address hidden>
Date: Tue Mar 12 16:39:19 2013 -0400

    Make run_instance() bail quietly if instance has been deleted

    Right now, we make a big mess in the logs if someone deletes an
    instance before we get a chance to run it. This cleans that up
    to a single warning message.

    Fixes bug 1154292

    Change-Id: I2de0c2993d6a16d9482878eee4eef10bfd78c3e6

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-rc1 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.