wrap_instance_fault doesn't work for some method

Bug #1067254 reported by Tiantian Gao
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Tiantian Gao

Bug Description

there is always an exception in some methods like:
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 167, in decorated_function
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 202, in decorated_function
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp kwargs['instance']['uuid'], e, sys.exc_info())
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp KeyError: 'instance'

the exception can be raised by some action like 'stop_instance', 'snapshot_instance', '_rotate_backups' etc.
 wrap_instance_fault will catch these exception and log into database, But I found it raised a exception caused by can not find 'instance' kwarg.
This can lead to lose exception information in database, and incorrect exception in logfile.

the completely log like:
========================
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 275, in _process_data
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 181, in decorated_function
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp pass
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 167, in decorated_function
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 202, in decorated_function
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp kwargs['instance']['uuid'], e, sys.exc_info())
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 196, in decorated_function
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 944, in stop_instance
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp final_state=vm_states.STOPPED)
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 181, in decorated_function
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp pass
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 167, in decorated_function
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 202, in decorated_function
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp kwargs['instance']['uuid'], e, sys.exc_info())
2012-10-16 14:35:39 TRACE nova.openstack.common.rpc.amqp KeyError: 'instance'
---------------------
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 275, in _process_data
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 181, in decorated_function
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp pass
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 167, in decorated_function
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 202, in decorated_function
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp kwargs['instance']['uuid'], e, sys.exc_info())
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 196, in decorated_function
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 1178, in snapshot_instance
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp self._rotate_backups(context, instance, backup_type, rotation)
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py", line 202, in decorated_function
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp kwargs['instance']['uuid'], e, sys.exc_info())
2012-10-16 13:37:15 TRACE nova.openstack.common.rpc.amqp KeyError: 'instance'

Tiantian Gao (gtt116)
Changed in nova:
assignee: nobody → TianTian Gao (gtt116)
status: New → In Progress
Revision history for this message
Michael Still (mikal) wrote :

This should be a one line change to nova/compute/manager.py

Changed in nova:
status: In Progress → Triaged
importance: Undecided → Medium
tags: added: low-hanging-fruit
Revision history for this message
Tiantian Gao (gtt116) wrote :

The reason is the argument 'instance' not in "kwargs", it is in "args". So I think we can update kwargs with args.
Any other good idea?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/14496

Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/14496
Committed: http://github.com/openstack/nova/commit/4ade2ad67a3862aa1e6d59e05fad6806baea7a62
Submitter: Jenkins
Branch: master

commit 4ade2ad67a3862aa1e6d59e05fad6806baea7a62
Author: TianTian Gao <gtt116@126.com>
Date: Wed Oct 17 10:03:24 2012 +0800

    update kwargs with args in wrap_instance_fault

    fix bug: 1067254

    If argument 'instance' is in args rather than kwargs,
    we will get a KeyError exception which will cover up
    the real exception. So, we update kwargs with the
    values from args first. then, we can get 'instance'
    from kwargs easily.

    Change-Id: Ife89f9e9629e4b581d96289434e9ececced17915

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-1 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.