Ironic

Unexpected exceptions in nova driver destroy() causes traceback

Bug #1297549 reported by Adam Gandelman on 2014-03-25

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Ironic	Fix Released	Medium	Adam Gandelman	Ironic 2014.1 "icehouse"

Bug Description

The virt drivers destroy() function has some seemingly temporary special casing for 'InstanceDeployFailure' exceptions, which can cause an AttributeError if the API request throws an unexpected exception:

014-03-25 02:20:32.723 4773 TRACE nova.compute.manager [instance: b4f40561-310e-48ef-bf92-be98176ecf00]
2014-03-25 02:20:34.785 4773 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: 'HTTPInternalServerError' object has no attribute '__name__'
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher return func(*args, **kwargs)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher payload)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 280, in decorated_function
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher pass
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 266, in decorated_function
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 333, in decorated_function
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher function(self, context, *args, **kwargs)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 309, in decorated_function
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 296, in decorated_function
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2073, in run_instance
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher do_run_instance()
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2072, in do_run_instance
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher legacy_bdm_in_spec)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1205, in _run_instance
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher notify("error", fault=e) # notify that build failed
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1189, in _run_instance
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher instance, image_meta, legacy_bdm_in_spec)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1353, in _build_instance
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher filter_properties, bdms, legacy_bdm_in_spec)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1399, in _reschedule_or_error
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher self._log_original_error(exc_info, instance_uuid)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1394, in _reschedule_or_error
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher bdms, requested_networks)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2123, in _shutdown_instance
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher requested_networks)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2113, in _shutdown_instance
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher block_device_info)
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/ironic/nova/virt/ironic/driver.py", line 503, in destroy
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher if e.__name__ == 'InstanceDeployFailure':
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher AttributeError: 'HTTPInternalServerError' object has no attribute '__name__'
2014-03-25 02:20:34.785 4773 TRACE oslo.messaging.rpc.dispatcher
2014-03-25 02:20:34.845 4773 ERROR oslo.messaging._drivers.common [-] Returning exception 'HTTPInternalServerError' object has no attribute '__name__' to caller

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-03-25: Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/82948

Changed in ironic:
assignee:	nobody → Adam Gandelman (gandelman-a)
status:	New → In Progress

Revision history for this message

Adam Gandelman (gandelman-a) wrote on 2014-03-25:

Proposed a band-aid, not sure if it would be better to fix properly as per the comments in code, or to defer a more thorough fix till Juno:

        except Exception as e:
            # if the node is already in a deprovisioned state, continue
            # This should be fixed in Ironic.
            # TODO(deva): This exception should be added to python-ironicclient
            # and matched directly, rather than via __name__.
            if e.__name__ == 'InstanceDeployFailure':
                pass

aeva black (tenbrae) on 2014-03-26

Changed in ironic:
importance:	Undecided → Medium

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-03-27: Fix merged to ironic (master)

Reviewed: https://review.openstack.org/82948
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=bb05b189d175c0e590074646d661981a40c0bb28
Submitter: Jenkins
Branch: master

commit bb05b189d175c0e590074646d661981a40c0bb28
Author: Adam Gandelman <email address hidden>
Date: Tue Mar 25 16:43:21 2014 -0700

Raise unexpected exceptions during destroy()

    Do not match exception by __name__ When an API request throws an
    unexpected exception. Avoids nova-compute AttributeError traceback
    and provides better debug info on the nova side.

Change-Id: I2886df814e8641717c57f77626f92917d1ff744c
Closes-bug: #1297549

Changed in ironic:
status:	In Progress → Fix Committed

Thierry Carrez (ttx) on 2014-04-01

Changed in ironic:
milestone:	none → icehouse-rc1
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2014-04-17

Changed in ironic:
milestone:	icehouse-rc1 → 2014.1

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.