Get Notification for API version 1.1 returns KeyError: 'progress_details'

Bug #1819422 reported by Tushar Patil
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
masakari
Fix Released
High
Shilpa Devharakar

Bug Description

After creating a notification, if you immediately try to get details of the notification before masakari-engine actually updates the progress details of the first task of the recovery workflow then it give an error KeyError: progress_details.'

$curl -g -i -X GET http://<IP>/instance-ha/v1/notifications/6bf9efd2-8786-4cfe-834e-815a63217938 -H "OpenStack-API-Version: instance-ha 1.1" -H "X-Auth-Token: <AUTH_TOKEN>

{"masakariFault": {"message": "Unexpected API Error. Please report this at https://bugs.launchpad.net/masakari/ and attach the Masakari API log if possible.\n<class 'MasakariException_Remote'>", "code": 500}}

Masakari-engine Error logs:
===================================
masakari-engine[28480]: 2019-03-11 13:02:02.942 ERROR masakari.engine.manager [req-cf7e0134-6df1-42f4-83d6-70b4af4fc2bd admin admin] (u'Failed to fetch notification recovery workflow details for %s', u'a5c486b7-969f-4994-bf4a-d24907b0bea3'): KeyError: 'progress_details'
masakari-engine[28480]: 2019-03-11 13:02:02.942 TRACE masakari.engine.manager Traceback (most recent call last):
masakari-engine[28480]: 2019-03-11 13:02:02.942 TRACE masakari.engine.manager File "/opt/stack/masakari/masakari/engine/manager.py", line 348, in get_notification_recovery_workflow_details
masakari-engine[28480]: 2019-03-11 13:02:02.942 TRACE masakari.engine.manager context, recovery_method, notification))
masakari-engine[28480]: 2019-03-11 13:02:02.942 TRACE masakari.engine.manager File "/opt/stack/masakari/masakari/engine/drivers/taskflow/driver.py", line 302, in get_notification_recovery_workflow_details
masakari-engine[28480]: 2019-03-11 13:02:02.942 TRACE masakari.engine.manager value.meta['progress_details']['details']
masakari-engine[28480]: 2019-03-11 13:02:02.942 TRACE masakari.engine.manager KeyError: 'progress_details'
masakari-engine[28480]: 2019-03-11 13:02:02.942 TRACE masakari.engine.manager
masakari-engine[28480]: 2019-03-11 13:02:02.943 ERROR oslo_messaging.rpc.server [req-cf7e0134-6df1-42f4-83d6-70b4af4fc2bd admin admin] Exception during message handling: MasakariException: (u'Failed to fetch notification recovery workflow details for %s', u'a5c486b7-969f-4994-bf4a-d24907b0bea3')
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server result = func(ctxt, **new_args)
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server File "/opt/stack/masakari/masakari/engine/manager.py", line 354, in get_notification_recovery_workflow_details
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server raise exception.MasakariException(msg)
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server MasakariException: (u'Failed to fetch notification recovery workflow details for %s', u'a5c486b7-969f-4994-bf4a-d24907b0bea3')
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server

Note: openstacksdk 0.26.0 is not yet released so if you want to reproduce this issue, you will need to apply below patch for openstacksdk project : https://review.openstack.org/#/c/640673/

Changed in masakari:
assignee: nobody → Shilpa Devharakar (shilpasd)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to masakari (master)

Fix proposed to branch: master
Review: https://review.openstack.org/642462

Changed in masakari:
status: New → In Progress
Changed in masakari:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to masakari (master)

Reviewed: https://review.openstack.org/642462
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=f2343ef09cf2cdd54a2239c12a28ba90104a2bab
Submitter: Zuul
Branch: master

commit f2343ef09cf2cdd54a2239c12a28ba90104a2bab
Author: shilpa <email address hidden>
Date: Mon Mar 11 14:22:59 2019 +0530

    Handle KeyError: 'progress_details'

    If user tries to get notification details before masakari-engine taskflow
    driver add progress details of the first task of the recovery workflow,
    it's giving Keyerror: 'progress_details'.

    Fixed the issue by checking progress_details is available in the meta.
    If it's not present, then get notification details will return empty list
    of recovery_workflow_details.

    Change-Id: I79af240d7715718b424695253ee452dc9552607d
    Closes-Bug: #1819422

Changed in masakari:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/masakari 7.0.0.0rc1

This issue was fixed in the openstack/masakari 7.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.