Overcloud deployment failing in promotion jobs at overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution

[yatin]

Overcloud deployment failing with Error:- APIException: resources.WorkflowTasks_Step2_Execution: Invalid input for field/attribute wf_ex.

Logs:- https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/7b3ca5a/undercloud/home/jenkins/failed_deployment_list.log.gz
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/ce9d237/undercloud/home/jenkins/failed_deployment_list.log.gz
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/ce9d237/undercloud/var/log/mistral/api.log.gz

[Ronelle Landy (rlandy)]

Note: error in ... resource_type: OS::Mistral::ExternalResource

[John Fulton (jfulton-org)]

- During a deployment Heat tried to start a mistral workflow to install ceph [2018-02-15 02:59:16.223 30322] [1]
- /usr/lib/python2.7/site-packages/mistral/api/controllers/v2/workflow.py:88 fetched the workflow [2018-02-15 02:59:17.078 30322]
- /usr/lib/python2.7/site-packages/wsme/api.py:222, as called by Mistral, could not decode the JSON that it received from Heat [2018-02-15 02:59:17.097 30322]

I tried to reproduce by copying/pasting that JSON into a Python REPL but json.dumps(x['params']) had no issue parsing it [2]. I'm going to look more closely at the files above.

[1] timestamps refer to https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/ce9d237/undercloud/var/log/mistral/api.log.gz

[2]
In [36]: x.keys()
Out[36]: ['input', 'workflow_name', 'params', 'description', 'workflow_namespace']

In [37]: json.dumps(x['params'])
...
ctlplane_node_ips\\": [\\"192.168.24.17\\"], \\"nova_api_ctlplane_node_ips\\": [\\"192.168.24.17\\"]}, \\"blacklisted_ip_addresses\\": []}}"'

In [38]:

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/545247

[Thomas Herve (therve)]

https://review.openstack.org/545247 should work around the issue in Heat. This was introduced by https://review.openstack.org/#/c/488531/

[Zane Bitter (zaneb)]

Also added python-mistralclient as affected, since https://review.openstack.org/#/c/488531/ will likely break many consumers of its API.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/545371

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545247
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=fd3b5a5d8601d48cf963f4d4b5eb7630a8ec878f
Submitter: Zuul
Branch: master

commit fd3b5a5d8601d48cf963f4d4b5eb7630a8ec878f
Author: Thomas Herve <email address hidden>
Date: Fri Feb 16 10:24:22 2018 +0100

    Pass mistral execution argument by name

    A recent released change added a positional argument where we don't
    expect it. It works around it.

    Closes-Bug: #1749645
    Change-Id: I4e9f4bb726c3461d17b2e22679377e262cd13462

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545371
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=7ec0a70b2001962d616ba3cdddb7dc8f6466696b
Submitter: Zuul
Branch: stable/queens

commit 7ec0a70b2001962d616ba3cdddb7dc8f6466696b
Author: Thomas Herve <email address hidden>
Date: Fri Feb 16 10:24:22 2018 +0100

    Pass mistral execution argument by name

    A recent released change added a positional argument where we don't
    expect it. It works around it.

    Closes-Bug: #1749645
    Change-Id: I4e9f4bb726c3461d17b2e22679377e262cd13462

[Alan Pevec (apevec)]

openstack-heat-api-10.0.0-0.20180218090758.3b894d2 was used in latest promotion attempt[1] and it still failed OC deploy[2]
2018-02-19 02:49:26 | 2018-02-19 02:49:18Z [overcloud.AllNodesDeploySteps.WorkflowTasks_Step2_Execution]: CREATE_FAILED resources.WorkflowTasks_Step2_Execution: ERROR

[1] https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-queens/8b4282d/rpm-qa.txt.gz

[2] https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-queens/8b4282d/undercloud/home/jenkins/overcloud_deploy.log.txt.gz

[yatin (yatinkarel)]

apevec, looks like it's a new bug(in queens, master is not facing it) https://bugs.launchpad.net/tripleo/+bug/1750311

[Alan Pevec (apevec)]

ok, let's keep this one closed and continue in the new bug 1750311

[John Fulton (jfulton-org)]

AFAICT there aren't plans to change mistral client based on brief conversation I had in IRC with one of the mistral client cores.

Seems like https://review.openstack.org/#/c/488531 introduced this problem and https://review.openstack.org/545247 worked around it.

Now we have https://bugs.launchpad.net/tripleo/+bug/1750311 but it seems to have a different root cause and to be something the the intergartion squad can work on. My plan is:

- Call this bug closed as per https://review.openstack.org/545247
- Move on to bug https://bugs.launchpad.net/tripleo/+bug/1750311

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 10.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 11.0.0.0b1 development milestone.