overcloud deploy fails on a KeyError: 'plans'

Bug #1802592 reported by Alex Schultz
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Alex Schultz

Bug Description

I've seen this a hand full of times so it seems to be a race condition. The latest was a failure in the gate where the overcloud deploy fails because of a keyerror in the workflows code:

http://logs.openstack.org/40/615140/14/gate/tripleo-ci-centos-7-containers-multinode/132bf36/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-09_20_33_49

2018-11-09 20:33:49 | No stack found, will be doing a stack create
2018-11-09 20:34:05 | Exception occured while running the command
2018-11-09 20:34:05 | Traceback (most recent call last):
2018-11-09 20:34:05 | File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 28, in run
2018-11-09 20:34:05 | super(Command, self).run(parsed_args)
2018-11-09 20:34:05 | File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
2018-11-09 20:34:05 | return super(Command, self).run(parsed_args)
2018-11-09 20:34:05 | File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
2018-11-09 20:34:05 | return_code = self.take_action(parsed_args) or 0
2018-11-09 20:34:05 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 889, in take_action
2018-11-09 20:34:05 | self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
2018-11-09 20:34:05 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 361, in _deploy_tripleo_heat_templates_tmpdir
2018-11-09 20:34:05 | new_tht_root, tht_root)
2018-11-09 20:34:05 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 373, in _deploy_tripleo_heat_templates
2018-11-09 20:34:05 | plans = plan_management.list_deployment_plans(self.clients)
2018-11-09 20:34:05 | File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/plan_management.py", line 142, in list_deployment_plans
2018-11-09 20:34:05 | return payload['plans']
2018-11-09 20:34:05 | KeyError: 'plans'
2018-11-09 20:34:05 | 'plans'

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (master)

Fix proposed to branch: master
Review: https://review.openstack.org/617035

Changed in tripleo:
assignee: nobody → Alex Schultz (alex-schultz)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (master)

Reviewed: https://review.openstack.org/617035
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=82ace31be5fcd5e28a8d253cda3b221d8dabf140
Submitter: Zuul
Branch: master

commit 82ace31be5fcd5e28a8d253cda3b221d8dabf140
Author: Alex Schultz <email address hidden>
Date: Fri Nov 9 14:18:08 2018 -0700

    Only return plans if available from the message

    There appears to be a race condition in the list_deployment_plans such
    that if another message is recieved before the plan results, we attempt
    to return the plans payload which may not exist. This change adds a
    check to ensure the payload contains plans before trying to return the
    values.

    Change-Id: I58da6f699fe8257e3ddb89cd3516889944c4ad92
    Closes-Bug: #1802592

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-tripleoclient (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/617351

tags: added: rocky-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-tripleoclient (stable/rocky)

Reviewed: https://review.openstack.org/617351
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=5047cf5cd04b51c0daab1ff68987706eac1a1ceb
Submitter: Zuul
Branch: stable/rocky

commit 5047cf5cd04b51c0daab1ff68987706eac1a1ceb
Author: Alex Schultz <email address hidden>
Date: Fri Nov 9 14:18:08 2018 -0700

    Only return plans if available from the message

    There appears to be a race condition in the list_deployment_plans such
    that if another message is recieved before the plan results, we attempt
    to return the plans payload which may not exist. This change adds a
    check to ensure the payload contains plans before trying to return the
    values.

    Change-Id: I58da6f699fe8257e3ddb89cd3516889944c4ad92
    Closes-Bug: #1802592
    (cherry picked from commit 82ace31be5fcd5e28a8d253cda3b221d8dabf140)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient 11.2.0

This issue was fixed in the openstack/python-tripleoclient 11.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/python-tripleoclient 10.6.1

This issue was fixed in the openstack/python-tripleoclient 10.6.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.