Trove fails to send notification event using oslo.messaging

Bug #1430586 reported by Rohit Jaiswal
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
Fix Released
High
Craig Vyvial

Bug Description

Created a trove instance using the trove create CLI, the trove guest comes up fine.

To enable notification publishing on the oslo bus, the following setting was added to /etc/trove/trove-taskmanager.conf:

notification_driver=messaging
notification_topics=['notifications', ]

The trove task manager service was then restarted. This was done before creating the trove instance.

Following is the error reported in the devstack tr-tmgr screen logs at the very end of trove instance creation:

2015-03-10 23:43:45.943 ERROR oslo_messaging.notify._impl_messaging [req-2b4d17f5-9e64-4e2d-9625-8fb15bd49d3c radmin trove] Could not send notification to ]. Payload={'priority': 'INFO', '_unique_id': 'deec83a5bf3741cfba39c22608e566e8', 'event_type': 'trove.instance.create', 'timestamp': u'2015-03-10 23:43:45.941363', 'publisher_id': '0.0.0.0', 'payload': {'state_description': 'BUILD', 'availability_zone': u'nova', 'instance_type_id': u'2', 'instance_size': 2048, 'nova_volume_id': u'906729b3-b4f3-4635-83f1-10231a76d1b8', 'instance_name': u'foo', 'user_id': u'7423f11194754748bc07f998949d137e', 'name': u'foo', 'volume_size': 2L, 'tenant_id': u'4b0cbdab977842d99e71ae2d81565bee', 'region': 'LOCAL_DEV', 'launched_at': '2015-03-10T23:35:36Z', 'instance_id': u'b93c15fd-f713-462e-8e64-c9e4f6629bdc', 'instance_type': u'm1.small', 'state': 'BUILD', 'nova_instance_id': u'c228be0f-6939-4f1a-843e-d725486c0947', 'service_id': '2f3ff068-2bfb-4f70-9a9d-a6bb65bc084b', 'created_at': '2015-03-10T23:35:36Z'}, 'message_id': u'bf7608ad-2333-4d82-afe9-fbb444d693c0'}
2015-03-10 23:43:45.943 TRACE oslo_messaging.notify._impl_messaging Traceback (most recent call last):
2015-03-10 23:43:45.943 TRACE oslo_messaging.notify._impl_messaging File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/notify/_impl_messaging.py", line 48, in notify
2015-03-10 23:43:45.943 TRACE oslo_messaging.notify._impl_messaging retry=retry)
2015-03-10 23:43:45.943 TRACE oslo_messaging.notify._impl_messaging File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send_notification
2015-03-10 23:43:45.943 TRACE oslo_messaging.notify._impl_messaging retry=retry)
2015-03-10 23:43:45.943 TRACE oslo_messaging.notify._impl_messaging File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 353, in send_notification
2015-03-10 23:43:45.943 TRACE oslo_messaging.notify._impl_messaging envelope=(version == 2.0), notify=True, retry=retry)
2015-03-10 23:43:45.943 TRACE oslo_messaging.notify._impl_messaging File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 314, in _send
2015-03-10 23:43:45.943 TRACE oslo_messaging.notify._impl_messaging rpc_amqp.pack_context(msg, context)
2015-03-10 23:43:45.943 TRACE oslo_messaging.notify._impl_messaging File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py", line 187, in pack_context
2015-03-10 23:43:45.943 TRACE oslo_messaging.notify._impl_messaging context_d = six.iteritems(context.to_dict())
2015-03-10 23:43:45.943 TRACE oslo_messaging.notify._impl_messaging File "/usr/local/lib/python2.7/dist-packages/six.py", line 576, in iteritems
2015-03-10 23:43:45.943 TRACE oslo_messaging.notify._impl_messaging return iter(d.iteritems(**kw))
2015-03-10 23:43:45.943 TRACE oslo_messaging.notify._impl_messaging AttributeError: 'TroveContext' object has no attribute 'iteritems'

trove list at this point shows the instance as ACTIVE.

The same error is found when doing other operations like trove resize-instance and trove resize-volume..

description: updated
description: updated
Changed in trove:
status: New → Triaged
importance: Undecided → High
milestone: none → kilo-3
Changed in trove:
assignee: nobody → Nikhil Manchanda (slicknik)
description: updated
Changed in trove:
milestone: kilo-3 → kilo-rc1
Changed in trove:
status: Triaged → In Progress
Changed in trove:
milestone: kilo-rc1 → liberty-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to trove (master)

Fix proposed to branch: master
Review: https://review.openstack.org/190368

Changed in trove:
assignee: Nikhil Manchanda (slicknik) → Craig Vyvial (cp16net)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to trove (master)

Reviewed: https://review.openstack.org/190368
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=b174d9db8c231cdb96087d223bf05a3f16f6da3f
Submitter: Jenkins
Branch: master

commit b174d9db8c231cdb96087d223bf05a3f16f6da3f
Author: Craig Vyvial <email address hidden>
Date: Wed Jun 10 16:36:12 2015 -0500

    Notification serialization of context

    Notifications with oslo_messaging was throwing an error that the
    TroveContext was not serialized properly.

    Change-Id: I651db0b72dbf4cb860b2df37ac36ddadc9e328d2
    Closes-Bug: #1430586

Changed in trove:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in trove:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in trove:
milestone: liberty-1 → 4.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.