2014-07-07 02:10:08.727 10283 ERROR oslo.messaging.rpc.dispatcher [req-54c68afe-91a8-4a99-86e8-785c0abf7688 ] Exception during message handling: Circular reference detected
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher payload)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 336, in decorated_function
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher function(self, context, *args, **kwargs)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/compute/utils.py", line 437, in __exit__
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher exc_tb=exc_tb)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/objects/base.py", line 142, in wrapper
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher args, kwargs)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 355, in object_class_action
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher objver=objver, args=args, kwargs=kwargs)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher wait_for_reply=True, timeout=timeout)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher timeout=timeout)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 385, in _send
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher msg = rpc_common.serialize_msg(msg)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/_drivers/common.py", line 462, in serialize_msg
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/venvs/nova/local/lib/python2.7/site-packages/oslo/messaging/openstack/common/jsonutils.py", line 164, in dumps
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher return json.dumps(value, default=default, **kwargs)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/json/__init__.py", line 250, in dumps
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher sort_keys=sort_keys, **kw).encode(obj)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher chunks = self.iterencode(o, _one_shot=True)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher return _iterencode(o, 0)
2014-07-07 02:10:08.727 10283 TRACE oslo.messaging.rpc.dispatcher ValueError: Circular reference detected
I instrumented the error: roles\' : [u\'_member_\', u\'admin\'], f9cb26e669bf626 d20c\', request_ id\': u\'req- e559c418- f387-4b2e- b7e2-27d4809e7b 4e\', service_ catalog\ ': [],
\'objmethod \': \'event_ finish_ with_failure\ ',
\'args\ ': (\'e2019ea2- bd2c-479a- 8908-75abbeb483 3c\', \'compute_ terminate_ instance\ '),
\'objname\ ': \'InstanceActio nEvent\ ',
\'kwargs\ ': {\'exc_val\': NovaException( u"Error destroying the instance on node 9e003bf8- 9890-4566- 8307-d4a250efd9 01. Provision state still \'deleting\'.",),
\'exc_ tb\': <traceback object at 0x7f24f3ce40e0>}}, b4cb161d5961a70 6163\', user\': u\'d2caebd29b6f 448e8fac0abd24b 25810\' , user_id\ ': u\'d2caebd29b6f 448e8fac0abd24b 25810\' , project_ name\': u\'admin\', read_deleted\ ': u\'no\', d89d8e5285fc4d1 f8d5ae5a5a7b5fe 22\', auth_token\ ': u\'MIIP. ....... nrq\', tenant\ ': u\'1767f7b0fe0d 4ca39f4f9caaf88 4b630\' , instance_ lock_checked\ ': False, is_admin\ ': True, project_ id\': u\'1767f7b0fe0d 4ca39f4f9caaf88 4b630\' , timestamp\ ': \'2014- 07-07T05: 41:50.904570\ ', user_name\ ': u\'admin\', class_action\ ', remote_ address\ ': u\'192. 168.122. 1\'}'
value is
'{\'_context_
\'_msg_id\': \'4dfe58bf8be54
\'_context_
\'_context_
\'args\': {\'objver\': \'1.0\',
\'_unique_id\': \'91ad34be58b74
\'_context_
\'_context_
\'_context_
\'_context_
\'_reply_q\': \'reply_
\'_context_
\'_context_
\'_context_
\'_context_
\'version\': \'2.0\',
\'_context_
\'_context_
\'_context_
\'method\': \'object_
\'_context_
Its not obvious to me where the circular reference is - but I strongly suspect its in the exc_tb.