Get Notification for API version 1.1 returns KeyError: 'progress_details'

Bug #1819422 reported by Tushar Patil on 2019-03-11
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
masakari
High
Shilpa Devharakar

Bug Description

After creating a notification, if you immediately try to get details of the notification before masakari-engine actually updates the progress details of the first task of the recovery workflow then it give an error KeyError: progress_details.'

$curl -g -i -X GET http://<IP>/instance-ha/v1/notifications/6bf9efd2-8786-4cfe-834e-815a63217938 -H "OpenStack-API-Version: instance-ha 1.1" -H "X-Auth-Token: <AUTH_TOKEN>

{"masakariFault": {"message": "Unexpected API Error. Please report this at https://bugs.launchpad.net/masakari/ and attach the Masakari API log if possible.\n<class 'MasakariException_Remote'>", "code": 500}}

Masakari-engine Error logs:
===================================
masakari-engine[28480]: 2019-03-11 13:02:02.942 ERROR masakari.engine.manager [req-cf7e0134-6df1-42f4-83d6-70b4af4fc2bd admin admin] (u'Failed to fetch notification recovery workflow details for %s', u'a5c486b7-969f-4994-bf4a-d24907b0bea3'): KeyError: 'progress_details'
masakari-engine[28480]: 2019-03-11 13:02:02.942 TRACE masakari.engine.manager Traceback (most recent call last):
masakari-engine[28480]: 2019-03-11 13:02:02.942 TRACE masakari.engine.manager File "/opt/stack/masakari/masakari/engine/manager.py", line 348, in get_notification_recovery_workflow_details
masakari-engine[28480]: 2019-03-11 13:02:02.942 TRACE masakari.engine.manager context, recovery_method, notification))
masakari-engine[28480]: 2019-03-11 13:02:02.942 TRACE masakari.engine.manager File "/opt/stack/masakari/masakari/engine/drivers/taskflow/driver.py", line 302, in get_notification_recovery_workflow_details
masakari-engine[28480]: 2019-03-11 13:02:02.942 TRACE masakari.engine.manager value.meta['progress_details']['details']
masakari-engine[28480]: 2019-03-11 13:02:02.942 TRACE masakari.engine.manager KeyError: 'progress_details'
masakari-engine[28480]: 2019-03-11 13:02:02.942 TRACE masakari.engine.manager
masakari-engine[28480]: 2019-03-11 13:02:02.943 ERROR oslo_messaging.rpc.server [req-cf7e0134-6df1-42f4-83d6-70b4af4fc2bd admin admin] Exception during message handling: MasakariException: (u'Failed to fetch notification recovery workflow details for %s', u'a5c486b7-969f-4994-bf4a-d24907b0bea3')
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server result = func(ctxt, **new_args)
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server File "/opt/stack/masakari/masakari/engine/manager.py", line 354, in get_notification_recovery_workflow_details
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server raise exception.MasakariException(msg)
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server MasakariException: (u'Failed to fetch notification recovery workflow details for %s', u'a5c486b7-969f-4994-bf4a-d24907b0bea3')
masakari-engine[28480]: 2019-03-11 13:02:02.943 TRACE oslo_messaging.rpc.server

Note: openstacksdk 0.26.0 is not yet released so if you want to reproduce this issue, you will need to apply below patch for openstacksdk project : https://review.openstack.org/#/c/640673/

Changed in masakari:
assignee: nobody → Shilpa Devharakar (shilpasd)

Fix proposed to branch: master
Review: https://review.openstack.org/642462

Changed in masakari:
status: New → In Progress
Changed in masakari:
importance: Undecided → High

Reviewed: https://review.openstack.org/642462
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=f2343ef09cf2cdd54a2239c12a28ba90104a2bab
Submitter: Zuul
Branch: master

commit f2343ef09cf2cdd54a2239c12a28ba90104a2bab
Author: shilpa <email address hidden>
Date: Mon Mar 11 14:22:59 2019 +0530

    Handle KeyError: 'progress_details'

    If user tries to get notification details before masakari-engine taskflow
    driver add progress details of the first task of the recovery workflow,
    it's giving Keyerror: 'progress_details'.

    Fixed the issue by checking progress_details is available in the meta.
    If it's not present, then get notification details will return empty list
    of recovery_workflow_details.

    Change-Id: I79af240d7715718b424695253ee452dc9552607d
    Closes-Bug: #1819422

Changed in masakari:
status: In Progress → Fix Released

This issue was fixed in the openstack/masakari 7.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers