overcloud deployment failing with msg: 'argument parameters is of type <class ''str''> and we were unable to convert to dict: unable to evaluate string as dictionary'.

[Sandeep Yadav]

periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-clients-master jobs are failing at overcloud deploy steps with msg: 'argument parameters is of type <class ''str''> and we were unable to convert to dict: unable to evaluate string as dictionary'.

ERROR Snippet
=============

https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-clients-master/e129acd/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

~~~
2020-04-07 07:23:30 | TASK [Update parameters] *******************************************************
2020-04-07 07:23:30 | Tuesday 07 April 2020 07:23:30 +0000 (0:00:00.064) 0:00:00.283 *********
2020-04-07 07:23:32 | fatal: [localhost]: FAILED! => changed=false
2020-04-07 07:23:32 | msg: 'argument parameters is of type <class ''str''> and we were unable to convert to dict: unable to evaluate string as dictionary'
2020-04-07 07:23:32 |
2020-04-07 07:23:32 | NO MORE HOSTS LEFT *************************************************************
2020-04-07 07:23:32 |
2020-04-07 07:23:32 | PLAY RECAP *********************************************************************
2020-04-07 07:23:32 | localhost : ok=1 changed=0 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0
2020-04-07 07:23:32 | Tuesday 07 April 2020 07:23:32 +0000 (0:00:02.022) 0:00:02.305 *********
2020-04-07 07:23:32 | ===============================================================================
2020-04-07 07:23:32 | Update parameters ------------------------------------------------------- 2.02s
2020-04-07 07:23:32 | Check for required inputs ----------------------------------------------- 0.07s
2020-04-07 07:23:32 | Set parameters fact ----------------------------------------------------- 0.06s
2020-04-07 07:23:33 | Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-update-params.yaml, Run Status: failed, Return Code: 2
2020-04-07 07:23:33 | Exception occured while running the command
2020-04-07 07:23:33 | Traceback (most recent call last):
2020-04-07 07:23:33 | File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
2020-04-07 07:23:33 | super(Command, self).run(parsed_args)
2020-04-07 07:23:33 | File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
2020-04-07 07:23:33 | return super(Command, self).run(parsed_args)
2020-04-07 07:23:33 | File "/usr/lib/python3.6/site-packages/cliff/command.py", line 187, in run
2020-04-07 07:23:33 | return_code = self.take_action(parsed_args) or 0
2020-04-07 07:23:33 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 994, in take_action
2020-04-07 07:23:33 | self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
2020-04-07 07:23:33 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 382, in _deploy_tripleo_heat_templates_tmpdir
2020-04-07 07:23:33 | new_tht_root, tht_root)
2020-04-07 07:23:33 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 488, in _deploy_tripleo_heat_templates
2020-04-07 07:23:33 | deployment_options=deployment_options)
2020-04-07 07:23:33 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 507, in _try_overcloud_deploy_with_compat_yaml
2020-04-07 07:23:33 | deployment_options=deployment_options)
2020-04-07 07:23:33 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 221, in _heat_deploy
2020-04-07 07:23:33 | stack_name, env, moved_files, tht_root)
2020-04-07 07:23:33 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 312, in _process_and_upload_environment
2020-04-07 07:23:33 | "parameters": params
2020-04-07 07:23:33 | File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 640, in run_ansible_playbook
2020-04-07 07:23:33 | raise RuntimeError(err_msg)
2020-04-07 07:23:33 | RuntimeError: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-update-params.yaml, Run Status: failed, Return Code: 2
2020-04-07 07:23:33 | Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-update-params.yaml, Run Status: failed, Return Code: 2
2020-04-07 07:23:33 | + status_code=1
2020-04-07 07:23:33 | + openstack stack list
2020-04-07 07:23:33 | + grep -q overcloud
2020-04-07 07:23:36 | + echo 'overcloud deployment not started. Check the deploy configurations'
2020-04-07 07:23:36 | overcloud deployment not started. Check the deploy configurations
2020-04-07 07:23:36 | + exit 1
~~~

[Sandeep Yadav (sandeepyadav93)]

Hello,

This seems like an issue in cli-update-params.yaml[0] which was introduced last week.

Last good run did not call "Update Parameters" task where deployment is currently failing. My suspect is - it was getting called earlier by mistral earlier and not by ansible.

When "update parameters mistral workflows" was removed [3] 3 days earlier, rdo check job also failed with same issue[4] :-

~~~
2020-04-03 17:34:17 | TASK [Update parameters] *******************************************************
2020-04-03 17:34:17 | Friday 03 April 2020 17:34:17 +0000 (0:00:00.060) 0:00:00.249 **********
2020-04-03 17:34:19 | fatal: [localhost]: FAILED! => changed=false
2020-04-03 17:34:19 | msg: 'argument parameters is of type <class ''str''> and we were unable to convert to dict: unable to evaluate string as dictionary'
2020-04-03 17:34:19 |
~~~

[0] https://review.opendev.org/#/c/716282/

[1] https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/playbooks/cli-update-params.yaml

[2] https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-clients-master/92747c6/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[3] https://review.opendev.org/#/c/716286/

[4] https://logserver.rdoproject.org/86/716286/4/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-vexxhost/4d3fc63/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/718416

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sagi Shnaidman (<email address hidden>) on branch: master
Review: https://review.opendev.org/718416

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sagi Shnaidman (<email address hidden>) on branch: master
Review: https://review.opendev.org/717865

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/718759
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=a947b57094f2fd5da6d54e1c98b2aa947bb37914
Submitter: Zuul
Branch: master

commit a947b57094f2fd5da6d54e1c98b2aa947bb37914
Author: Kevin Carter <email address hidden>
Date: Thu Apr 9 12:18:20 2020 -0500

    Add the ability to load vars from files

    This change will provide the ability to load extra vars from files, instead
    of having to pass options through the CLI parser. By loading vars from file
    we can ensure options are made more safe and better handled, especially in
    cases when a given option may be massive, as is the case with
    `parameter_defaults`.

    > A new argument has been added to the ansible playbook runner which will
      allow us to pass options into the method that will be stored in an
      extravars file, which is then dynamically loaded by ansible-runner.
      Information on extravars files can be seen here: [0].
    > A test has been added to exercise the new extravars file capability.

    [0]: https://ansible-runner.readthedocs.io/en/latest/intro.html#env-extravars

    Closes-Bug: #1871338
    Change-Id: I9675e587abf3f07e91319a40620a8f4c67fbf97b
    Signed-off-by: Kevin Carter <email address hidden>