Notifications do not work: AttributeError: 'RequestContext' object has no attribute 'iteritems'

[Julien Danjou]

When enabling notification with notification_driver = messaging, I get the following:

2014-02-03 14:20:41.152 ERROR oslo.messaging.notify._impl_messaging [-] Could not send notification to notifications. Payload={'priority': 'INFO', '_unique_id': 'da748b32fd144c25adc45ba5b393339d', 'event_type': 'compute.instance.create.end', 'timestamp': '2014-02-03 14:20:41.151419', 'publisher_id': 'compute.devstack', 'payload': {'node': u'devstack', 'state_description': '', 'ramdisk_id': u'37ad58df-c587-4bed-9062-9428ca14eaf0', 'created_at': '2014-02-03 14:20:33+00:00', 'access_ip_v6': None, 'disk_gb': 0, 'availability_zone': u'nova', 'terminated_at': '', 'ephemeral_gb': 0, 'instance_type_id': 6, 'instance_flavor_id': '42', 'image_name': u'cirros-0.3.1-x86_64-uec', 'host': u'devstack', 'fixed_ips': [FixedIP({'version': 4, 'floating_ips': [], 'label': u'private', 'meta': {}, 'address': u'10.0.0.2', 'type': u'fixed'})], 'user_id': u'6bcbc8f54d65473c9a0c4a55f64fb580', 'message': u'Success', 'deleted_at': '', 'reservation_id': u'r-jycyyveh', 'image_ref_url': u'http://162.209.87.220:9292/images/7b8d712a-fb31-43b8-8a05-a74d70fd8a11', 'memory_mb': 64, 'root_gb': 0, 'display_name': u'dwq', 'instance_type': 'm1.nano', 'tenant_id': u'cda1741ff4ef47f48fb3d9d76e302add', 'access_ip_v4': None, 'hostname': u'dwq', 'vcpus': 1, 'instance_id': '272c2ec6-bb98-4e84-9377-84c63c7a9ce9', 'kernel_id': u'5d1a6130-0e6a-4155-9c05-0174a654da68', 'state': u'active', 'image_meta': {u'kernel_id': u'5d1a6130-0e6a-4155-9c05-0174a654da68', u'container_format': u'ami', u'min_ram': u'0', u'ramdisk_id': u'37ad58df-c587-4bed-9062-9428ca14eaf0', u'disk_format': u'ami', u'min_disk': u'0', u'base_image_ref': u'7b8d712a-fb31-43b8-8a05-a74d70fd8a11'}, 'architecture': None, 'os_type': None, 'launched_at': '2014-02-03T14:20:41.070490', 'metadata': {}}, 'message_id': '03b2985a-6bcd-44ff-8303-29618d3c2b01'}
2014-02-03 14:20:41.152 TRACE oslo.messaging.notify._impl_messaging Traceback (most recent call last):
2014-02-03 14:20:41.152 TRACE oslo.messaging.notify._impl_messaging File "/opt/stack/oslo.messaging/oslo/messaging/notify/_impl_messaging.py", line 47, in notify
2014-02-03 14:20:41.152 TRACE oslo.messaging.notify._impl_messaging version=self.version)
2014-02-03 14:20:41.152 TRACE oslo.messaging.notify._impl_messaging File "/opt/stack/oslo.messaging/oslo/messaging/transport.py", line 93, in _send_notification
2014-02-03 14:20:41.152 TRACE oslo.messaging.notify._impl_messaging self._driver.send_notification(target, ctxt, message, version)
2014-02-03 14:20:41.152 TRACE oslo.messaging.notify._impl_messaging File "/opt/stack/oslo.messaging/oslo/messaging/_drivers/amqpdriver.py", line 393, in send_notification
2014-02-03 14:20:41.152 TRACE oslo.messaging.notify._impl_messaging return self._send(target, ctxt, message, envelope=(version == 2.0))
2014-02-03 14:20:41.152 TRACE oslo.messaging.notify._impl_messaging File "/opt/stack/oslo.messaging/oslo/messaging/_drivers/amqpdriver.py", line 362, in _send
2014-02-03 14:20:41.152 TRACE oslo.messaging.notify._impl_messaging rpc_amqp.pack_context(msg, context)
2014-02-03 14:20:41.152 TRACE oslo.messaging.notify._impl_messaging File "/opt/stack/oslo.messaging/oslo/messaging/_drivers/amqp.py", line 299, in pack_context
2014-02-03 14:20:41.152 TRACE oslo.messaging.notify._impl_messaging context_d = six.iteritems(context.to_dict())
2014-02-03 14:20:41.152 TRACE oslo.messaging.notify._impl_messaging File "/usr/local/lib/python2.7/dist-packages/six.py", line 484, in iteritems
2014-02-03 14:20:41.152 TRACE oslo.messaging.notify._impl_messaging return iter(getattr(d, _iteritems)(**kw))
2014-02-03 14:20:41.152 TRACE oslo.messaging.notify._impl_messaging AttributeError: 'RequestContext' object has no attribute 'iteritems'
2014-02-03 14:20:41.152 TRACE oslo.messaging.notify._impl_messaging

[Julien Danjou (jdanjou)]

Patch at https://review.openstack.org/#/c/70759/

[Mark McLoughlin (markmc)]

See my comment in the review - it sounds to me like nova.rpc.RequestContextSerializer is getting bypassed somehow

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/71532

[Mark McLoughlin (markmc)]

Julien figured out this was a bug in Nova

We still want to make it easier in oslo.messaging to catch this issue, though

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/71532
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=05f688e49fbf543a6c69c0f186279c6732118470
Submitter: Jenkins
Branch: master

commit 05f688e49fbf543a6c69c0f186279c6732118470
Author: Julien Danjou <email address hidden>
Date: Thu Feb 6 15:14:51 2014 +0100

    nova: use RequestContextSerializer for notifications

    RequestContext should be serialized when sent via oslo.messaging. The
    serializer is correctly used for the general RPC mechanism, but has been
    forgotten in the notifier. This patch fixes that.

    Change-Id: I56fa8022e34c0e80835e3bde940fda99ed0f9ba8
    Closes-Bug: #1275771

[Eoghan Glynn (eglynn)]

Impacts on ceilometer alarm notification also since the oslo-messaging switch-over:

2014-05-07 11:38:28.365 25696 DEBUG oslo.messaging._drivers.amqp [-] UNIQUE_ID is 68d046edf6614bab854170f81aec72f1. _add_unique_id /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py:251
2014-05-07 11:38:28.366 25696 ERROR ceilometer.alarm.evaluator [-] alarm state update failed
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator Traceback (most recent call last):
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator File "/opt/stack/ceilometer/ceilometer/alarm/evaluator/__init__.py", line 79, in _refresh
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator self.notifier.notify(alarm, previous, reason, reason_data)
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator File "/opt/stack/ceilometer/ceilometer/alarm/rpc.py", line 66, in notify
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator 'reason_data': reason_data})
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 325, in cast
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator self.prepare().cast(ctxt, method, **kwargs)
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 132, in cast
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator self.transport._send(self.target, ctxt, msg)
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator timeout=timeout)
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 386, in send
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 356, in _send
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator rpc_amqp.pack_context(msg, context)
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py", line 212, in pack_context
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator context_d = six.iteritems(context.to_dict())
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator File "/usr/lib/python2.7/site-packages/six.py", line 498, in iteritems
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator return iter(getattr(d, _iteritems)(**kw))
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator AttributeError: 'RequestContext' object has no attribute 'iteritems'
2014-05-07 11:38:28.366 25696 TRACE ceilometer.alarm.evaluator

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/92576

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/92576
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=5df4cb53ae36ddb2a2495cc7aa9a1f3072133930
Submitter: Jenkins
Branch: master

commit 5df4cb53ae36ddb2a2495cc7aa9a1f3072133930
Author: Mehdi Abaakouk <email address hidden>
Date: Wed May 7 14:42:49 2014 +0200

    oslo.messaging context must be a dict

    oslo.messaging assume the context is a dict not a RequestContext
    This patch ensure this.

    Closes-bug: #1275771

    Change-Id: Iae58d40f171d1d5780b74b747c26a74cca05b459

[Eoghan Glynn (eglynn)]

Reverting the ceilometer patch:

  https://review.openstack.org/92720

due to it causing an unintended side-effect with the notification agent (LP 1317290), pending further investigation.

[Eoghan Glynn (eglynn)]

Resetting status of ceilometer aspect back to 'in progress' since the revert landed:

  https://github.com/openstack/ceilometer/commit/36269031

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/93022

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/93022
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=6f9e46ba5c4ed9a04e4f12460be5edc0b9c0602b
Submitter: Jenkins
Branch: master

commit 6f9e46ba5c4ed9a04e4f12460be5edc0b9c0602b
Author: Mehdi Abaakouk <email address hidden>
Date: Wed May 7 14:42:49 2014 +0200

    oslo.messaging context must be a dict

    oslo.messaging assumes the context is a dict not a RequestContext
    and it assumes the payload in json serializable.

    This patch ensures this.

    Also it removes oslo.messaging mock on some tests and use real oslo.messaging
    library with the fake driver.

    Change-Id: Ie3c6083bbc4ec83de28e42bb10e7c50c7e135070
    Closes-bug: #1275771
    Closes-bug: #1317290