AttributeError: 'unicode' object has no attribute 'tb_frame'

[clayg]

Saw this in CI -> http://logs.openstack.org/11/98511/2/check/check-grenade-dsvm/5d60567/logs/new/screen-n-cpu.txt.gz?level=ERROR

2014-06-06 23:00:29.533 ERROR oslo.messaging._drivers.common [req-4ae02289-de3e-4700-968c-7611980e0346 ServerActionsTestXML-67143133 ServerActionsTestXML-1032052660] Returning exception 'unicode' object has no attribute 'tb_frame'
Traceback (most recent call last):

  File "/opt/stack/new/nova/nova/conductor/manager.py", line 602, in _object_dispatch
    return getattr(target, method)(context, *args, **kwargs)

  File "/opt/stack/new/nova/nova/objects/instance_action.py", line 119, in wrapper
    kwargs['exc_tb'] = ''.join(traceback.format_tb(exc_tb))

  File "/usr/lib/python2.7/traceback.py", line 76, in format_tb
    return format_list(extract_tb(tb, limit))

  File "/usr/lib/python2.7/traceback.py", line 95, in extract_tb
    f = tb.tb_frame

AttributeError: 'unicode' object has no attribute 'tb_frame'
 to caller
2014-06-06 23:00:29.534 ERROR oslo.messaging._drivers.common [req-4ae02289-de3e-4700-968c-7611980e0346 ServerActionsTestXML-67143133 ServerActionsTestXML-1032052660] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/opt/stack/new/nova/nova/exception.py", line 88, in wrapped\n payload)\n', ' File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 82, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/new/nova/nova/exception.py", line 71, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/opt/stack/new/nova/nova/compute/manager.py", line 286, in decorated_function\n pass\n', ' File "/opt/stack/new/nova/nova/openstack/common/excutils.py", line 82, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/new/nova/nova/compute/manager.py", line 272, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/new/nova/nova/compute/manager.py", line 336, in decorated_function\n function(self, context, *args, **kwargs)\n', ' File "/opt/stack/new/nova/nova/compute/utils.py", line 437, in __exit__\n exc_tb=exc_tb, want_result=False)\n', ' File "/opt/stack/new/nova/nova/objects/instance_action.py", line 121, in wrapper\n return fn.__get__(None, cls)(*args, **kwargs)\n', ' File "/opt/stack/new/nova/nova/objects/base.py", line 144, in wrapper\n args, kwargs)\n', ' File "/opt/stack/new/nova/nova/conductor/rpcapi.py", line 355, in object_class_action\n objver=objver, args=args, kwargs=kwargs)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 150, in call\n wait_for_reply=True, timeout=timeout)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 89, in _send\n timeout=timeout)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 386, in send\n return self._send(target, ctxt, message, wait_for_reply, timeout)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 379, in _send\n raise result\n', 'AttributeError: \'unicode\' object has no attribute \'tb_frame\'\nTraceback (most recent call last):\n\n File "/opt/stack/new/nova/nova/conductor/manager.py", line 602, in _object_dispatch\n return getattr(target, method)(context, *args, **kwargs)\n\n File "/opt/stack/new/nova/nova/objects/instance_action.py", line 119, in wrapper\n kwargs[\'exc_tb\'] = \'\'.join(traceback.format_tb(exc_tb))\n\n File "/usr/lib/python2.7/traceback.py", line 76, in format_tb\n return format_list(extract_tb(tb, limit))\n\n File "/usr/lib/python2.7/traceback.py", line 95, in extract_tb\n f = tb.tb_frame\n\nAttributeError: \'unicode\' object has no attribute \'tb_frame\'\n\n']
2014-06-06 23:01:15.378 ERROR nova.virt.libvirt.driver [req-a0da2719-553b-4654-8c45-215b49ce1d3f ServerActionsTestJSON-1623844969 ServerActionsTestJSON-1569101731] An error occurred while trying to launch a defined domain with xml: <domain type='qemu'>

I feel like this is the remote end of the error? Or maybe it's an oslo bug, idk.

[Matt Riedemann (mriedem)]

message:"Exception during message handling: 'unicode' object has no attribute 'tb_frame'" AND tags:"screen-n-cpu.txt"

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRXhjZXB0aW9uIGR1cmluZyBtZXNzYWdlIGhhbmRsaW5nOiAndW5pY29kZScgb2JqZWN0IGhhcyBubyBhdHRyaWJ1dGUgJ3RiX2ZyYW1lJ1wiIEFORCB0YWdzOlwic2NyZWVuLW4tY3B1LnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDAyNDA5NDIwMzczfQ==

[Matt Riedemann (mriedem)]

According to logstash this really started spiking on 6/6. This is probably what introduced the problem:

https://review.openstack.org/93017

[Matt Riedemann (mriedem)]

This is the bug:

if not isinstance(exc_tb, str) and exc_tb is not None:

exc_tb is a unicode object, not str.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/99119

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/99119
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8f8b6e656a6ef06bf0b99068a07a7d194783fc9b
Submitter: Jenkins
Branch: master

commit 8f8b6e656a6ef06bf0b99068a07a7d194783fc9b
Author: Matt Riedemann <email address hidden>
Date: Tue Jun 10 08:07:03 2014 -0700

    Handle string types for InstanceActionEvent exc_tb serialization

    Commit 59a6cf233b538d6666740de4796fce25ed8265aa added code to handle
    serializing non-str exception traceback objects, but didn't account for
    unicode. This change uses six.string_types to handle str and unicode
    objects for the exc_tb argument.

    Adds a new unit test for the unicode case and firms up two existing
    tests for how traceback.format_tb is mocked (or shouldn't be in the case
    of exc_tb being a str).

    Closes-Bug: #1327476

    Change-Id: Icc10b62a3f65610c50e86c0b366c2b70b1a0932d