This bug happen again in kilo-3.

oslo.log at commit: b9aa6120d843f09c07b20862dc0aa0958ca2aa58 ( Sat Mar 21 14:18:28 2015 +0000 )
oslo.messaging at commit: ec1fb8c76026b05d71979e2ecd1a97e97e081a7c ( Sat Mar 21 00:17:01 2015 +0000 )
nova at commit: 59356a4d80d1a9db41cc0bbc56735fd7cd742be6 ( Fri Mar 20 02:19:40 2015 +0000 )


Adding below content in nova.conf

[DEFAULT]
+ publish_errors=True
+ notification_driver=messaging

When exception happen, logging file show below content:

2015-03-24 07:33:39.752 13656 ERROR oslo_messaging.notify._impl_messaging [-] Could not send notification to notifications. Payload={'priority': 'ERROR', '_unique_id': '60d89ad0d0e24944acdb9938f768fb18', 'event_type': 'error_notification', 'timestamp': u'2015-03-24 07:33:39.044089', 'publisher_id': 'error.publisher', 'payload': {'error': u'Could not send notification to %(topic)s. Payload=%(message)s'}, 'message_id': u'ec5d699d-9f2d-4ec0-bf87-d95ad113c7d0'}
2015-03-24 07:33:39.752 13656 TRACE oslo_messaging.notify._impl_messaging Traceback (most recent call last):
2015-03-24 07:33:39.752 13656 TRACE oslo_messaging.notify._impl_messaging   File "/opt/stack/oslo.messaging/oslo_messaging/notify/_impl_messaging.py", line 48, in notify
2015-03-24 07:33:39.752 13656 TRACE oslo_messaging.notify._impl_messaging     retry=retry)
2015-03-24 07:33:39.752 13656 TRACE oslo_messaging.notify._impl_messaging   File "/opt/stack/oslo.messaging/oslo_messaging/transport.py", line 97, in _send_notification
2015-03-24 07:33:39.752 13656 TRACE oslo_messaging.notify._impl_messaging     retry=retry)
2015-03-24 07:33:39.752 13656 TRACE oslo_messaging.notify._impl_messaging   File "/opt/stack/oslo.messaging/oslo_messaging/_drivers/amqpdriver.py", line 354, in send_notification
2015-03-24 07:33:39.752 13656 TRACE oslo_messaging.notify._impl_messaging     envelope=(version == 2.0), notify=True, retry=retry)
2015-03-24 07:33:39.752 13656 TRACE oslo_messaging.notify._impl_messaging   File "/opt/stack/oslo.messaging/oslo_messaging/_drivers/amqpdriver.py", line 315, in _send
2015-03-24 07:33:39.752 13656 TRACE oslo_messaging.notify._impl_messaging     rpc_amqp.pack_context(msg, context)
2015-03-24 07:33:39.752 13656 TRACE oslo_messaging.notify._impl_messaging   File "/opt/stack/oslo.messaging/oslo_messaging/_drivers/amqp.py", line 225, in pack_context
2015-03-24 07:33:39.752 13656 TRACE oslo_messaging.notify._impl_messaging     context_d = six.iteritems(context.to_dict())
2015-03-24 07:33:39.752 13656 TRACE oslo_messaging.notify._impl_messaging   File "/usr/local/lib/python2.7/dist-packages/six.py", line 576, in iteritems
2015-03-24 07:33:39.752 13656 TRACE oslo_messaging.notify._impl_messaging     return iter(d.iteritems(**kw))
2015-03-24 07:33:39.752 13656 TRACE oslo_messaging.notify._impl_messaging AttributeError: 'NoneType' object has no attribute 'iteritems'
2015-03-24 07:33:39.752 13656 TRACE oslo_messaging.notify._impl_messaging 
2015-03-24 07:33:39.758 13656 DEBUG oslo_messaging._drivers.amqp [-] UNIQUE_ID is c95728d0c835488eb07fdd289aac67e2. _add_unique_id /opt/stack/oslo.messaging/oslo_messaging/_drivers/amqp.py:264


2015-03-24 07:35:06.303 15641 ERROR oslo_messaging.notify.notifier [-] Problem 'maximum recursion depth exceeded while calling a Python object' attempting to send to notification system. Payloa
d={'error': u'error calling %r: %s'}
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier Traceback (most recent call last):
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/opt/stack/oslo.messaging/oslo_messaging/notify/notifier.py", line 177, in do_notify
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     ext.obj.notify(ctxt, msg, priority, retry or self.retry)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/opt/stack/oslo.messaging/oslo_messaging/notify/_impl_messaging.py", line 52, in notify
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     dict(topic=topic, message=message))
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 1185, in exception
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self.error(msg, *args, **kwargs)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 1178, in error
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self._log(ERROR, msg, args, **kwargs)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 1271, in _log
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self.handle(record)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 1281, in handle
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self.callHandlers(record)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 1321, in callHandlers
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     hdlr.handle(record)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 749, in handle
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self.emit(record)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/opt/stack/oslo.messaging/oslo_messaging/notify/log_handler.py", line 39, in emit
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     dict(error=record.msg))
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/opt/stack/oslo.messaging/oslo_messaging/notify/notifier.py", line 256, in error
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self._notify(ctxt, event_type, payload, 'ERROR')
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/opt/stack/oslo.messaging/oslo_messaging/notify/notifier.py", line 184, in _notify
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self._driver_mgr.map(do_notify)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 224, in map
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self._invoke_one_plugin(response.append, func, e, args, kwds)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 260, in _invoke_one_plugin
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     LOG.error('error calling %r: %s', e.name, err)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 1178, in error
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self._log(ERROR, msg, args, **kwargs)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 1271, in _log
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self.handle(record)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 1281, in handle
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self.callHandlers(record)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 1321, in callHandlers
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     hdlr.handle(record)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 749, in handle
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self.emit(record)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/opt/stack/oslo.messaging/oslo_messaging/notify/log_handler.py", line 39, in emit
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     dict(error=record.msg))
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/opt/stack/oslo.messaging/oslo_messaging/notify/notifier.py", line 256, in error
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self._notify(ctxt, event_type, payload, 'ERROR')
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/opt/stack/oslo.messaging/oslo_messaging/notify/notifier.py", line 184, in _notify
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self._driver_mgr.map(do_notify)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 224, in map
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self._invoke_one_plugin(response.append, func, e, args, kwds)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 260, in _invoke_one_plugin
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     LOG.error('error calling %r: %s', e.name, err)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 1178, in error
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self._log(ERROR, msg, args, **kwargs)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 1271, in _log
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self.handle(record)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 1281, in handle
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self.callHandlers(record)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 1321, in callHandlers
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     hdlr.handle(record)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 749, in handle
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self.emit(record)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/opt/stack/oslo.messaging/oslo_messaging/notify/log_handler.py", line 39, in emit
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     dict(error=record.msg))
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/opt/stack/oslo.messaging/oslo_messaging/notify/notifier.py", line 256, in error
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self._notify(ctxt, event_type, payload, 'ERROR')
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/opt/stack/oslo.messaging/oslo_messaging/notify/notifier.py", line 184, in _notify
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self._driver_mgr.map(do_notify)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 224, in map
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self._invoke_one_plugin(response.append, func, e, args, kwds)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 260, in _invoke_one_plugin
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     LOG.error('error calling %r: %s', e.name, err)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 1178, in error
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     self._log(ERROR, msg, args, **kwargs)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 1270, in _log
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     record = self.makeRecord(self.name, level, fn, lno, msg, args, exc_info, func, extra)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier   File "/usr/lib/python2.7/logging/__init__.py", line 1244, in makeRecord
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier     rv = LogRecord(name, level, fn, lno, msg, args, exc_info, func)
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier RuntimeError: maximum recursion depth exceeded while calling a Python object
2015-03-24 07:35:06.303 15641 TRACE oslo_messaging.notify.notifier