centos-8 multinode overcloud deploy failed with heat_resource_tree_params = heat_resource_tree['parameters'] KeyError: 'parameters'

[chandan kumar]

http://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-containers-multinode

It is currently failing in check and gates usptream jobs while overcloud deploy.
https://b4a29afad567bb929800-43762991937ccde5eef92e3d2b5c183e.ssl.cf2.rackcdn.com/711273/1/gate/tripleo-ci-centos-8-containers-multinode/55df439/logs/undercloud/home/zuul/overcloud_deploy.log
2020-03-05 11:34:51 | ++ CLOUDPROMPT_ENABLED=1
2020-03-05 11:34:51 | + openstack overcloud deploy --override-ansible-cfg /home/zuul/custom_ansible.cfg --templates /usr/share/openstack-tripleo-heat-templates --libvirt-type qemu --timeout 90 -e /home/zuul/cloud-names.yaml -e /home/zuul/hostnamemap.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-environment.yaml --overcloud-ssh-user zuul --deployed-server -e /home/zuul/containers-prepare-parameter.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml -e /usr/share/openstack-tripleo-heat-templates/ci/environments/multinode-containers.yaml -e /home/zuul/overcloud_network_params.yaml -e /home/zuul/overcloud_storage_params.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml -e /home/zuul/src/opendev.org/openstack/tripleo-ci/test-environments/worker-config.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/debug.yaml --validation-errors-nonfatal --roles-file /home/zuul/overcloud_roles.yaml -e /home/zuul/overcloud-topology-config.yaml -e /home/zuul/config-download.yaml --disable-validations --verbose -e /home/zuul/overcloud-selinux-config.yaml
2020-03-05 11:34:52 | START with options: overcloud deploy --override-ansible-cfg /home/zuul/custom_ansible.cfg --templates /usr/share/openstack-tripleo-heat-templates --libvirt-type qemu --timeout 90 -e /home/zuul/cloud-names.yaml -e /home/zuul/hostnamemap.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-environment.yaml --overcloud-ssh-user zuul --deployed-server -e /home/zuul/containers-prepare-parameter.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml -e /usr/share/openstack-tripleo-heat-templates/ci/environments/multinode-containers.yaml -e /home/zuul/overcloud_network_params.yaml -e /home/zuul/overcloud_storage_params.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml -e /home/zuul/src/opendev.org/openstack/tripleo-ci/test-environments/worker-config.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/debug.yaml --validation-errors-nonfatal --roles-file /home/zuul/overcloud_roles.yaml -e /home/zuul/overcloud-topology-config.yaml -e /home/zuul/config-download.yaml --disable-validations --verbose -e /home/zuul/overcloud-selinux-config.yaml
2020-03-05 11:34:53 | command: overcloud deploy -> tripleoclient.v1.overcloud_deploy.DeployOvercloud (auth=True)
2020-03-05 11:34:53 | Using auth plugin: password
2020-03-05 11:34:54 | /usr/lib/python3.6/site-packages/tripleoclient/utils.py:2349: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
2020-03-05 11:34:54 | content = yaml.load(f)
2020-03-05 11:34:54 | No stack found, will be doing a stack create
2020-03-05 11:36:36 | /usr/lib64/python3.6/json/decoder.py:355: ResourceWarning: unclosed <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.1', 38046), raddr=('192.168.24.1', 8080)>
2020-03-05 11:36:36 | obj, end = self.scan_once(s, idx)
2020-03-05 11:36:36 | Pulling role list from: overcloud
2020-03-05 11:36:36 | Indexing roles from: overcloud
2020-03-05 11:36:38 | Temporary directory [ /tmp/tripleoclient-1q7subm5 ] cleaned up
2020-03-05 11:36:38 | Exception occured while running the command
2020-03-05 11:36:38 | Traceback (most recent call last):
2020-03-05 11:36:38 | File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
2020-03-05 11:36:38 | super(Command, self).run(parsed_args)
2020-03-05 11:36:38 | File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
2020-03-05 11:36:38 | return super(Command, self).run(parsed_args)
2020-03-05 11:36:38 | File "/usr/lib/python3.6/site-packages/cliff/command.py", line 187, in run
2020-03-05 11:36:38 | return_code = self.take_action(parsed_args) or 0
2020-03-05 11:36:38 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 936, in take_action
2020-03-05 11:36:38 | self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
2020-03-05 11:36:38 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 363, in _deploy_tripleo_heat_templates_tmpdir
2020-03-05 11:36:38 | new_tht_root, tht_root)
2020-03-05 11:36:38 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 458, in _deploy_tripleo_heat_templates
2020-03-05 11:36:38 | deployment_options=deployment_options)
2020-03-05 11:36:38 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 477, in _try_overcloud_deploy_with_compat_yaml
2020-03-05 11:36:38 | deployment_options=deployment_options)
2020-03-05 11:36:38 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 224, in _heat_deploy
2020-03-05 11:36:38 | workflow_params.check_deprecated_parameters(self.clients, stack_name)
2020-03-05 11:36:38 | File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/parameters.py", line 96, in check_deprecated_parameters
2020-03-05 11:36:38 | heat_resource_tree_params = heat_resource_tree['parameters']
2020-03-05 11:36:38 | KeyError: 'parameters'
2020-03-05 11:36:38 | 'parameters'

Expected reason is here https://review.opendev.org/#/c/709141/7/tripleoclient/workflows/parameters.py@130

and also seeing other failures in unpinning testing at same place
https://2714a4271cb20642317a-491b37cd909c997f4e833a309b30267e.ssl.cf1.rackcdn.com/711287/4/check/tripleo-ci-centos-8-containers-multinode/6a2bec0/logs/undercloud/home/zuul/overcloud_deploy.log

2020-03-05 05:53:34 | No stack found, will be doing a stack create
2020-03-05 05:56:57 | /usr/lib64/python3.6/json/decoder.py:355: ResourceWarning: unclosed <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.1', 35870), raddr=('192.168.24.1', 8080)>
2020-03-05 05:56:57 | obj, end = self.scan_once(s, idx)
2020-03-05 05:56:57 | Pulling role list from: overcloud
2020-03-05 05:56:57 | Indexing roles from: overcloud
2020-03-05 05:56:59 | Temporary directory [ /tmp/tripleoclient-mnbmwcey ] cleaned up
2020-03-05 05:56:59 | Exception occured while running the command
2020-03-05 05:56:59 | Traceback (most recent call last):
2020-03-05 05:56:59 | File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run
2020-03-05 05:56:59 | super(Command, self).run(parsed_args)
2020-03-05 05:56:59 | File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
2020-03-05 05:56:59 | return super(Command, self).run(parsed_args)
2020-03-05 05:56:59 | File "/usr/lib/python3.6/site-packages/cliff/command.py", line 187, in run
2020-03-05 05:56:59 | return_code = self.take_action(parsed_args) or 0
2020-03-05 05:56:59 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 936, in take_action
2020-03-05 05:56:59 | self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
2020-03-05 05:56:59 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 363, in _deploy_tripleo_heat_templates_tmpdir
2020-03-05 05:56:59 | new_tht_root, tht_root)
2020-03-05 05:56:59 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 458, in _deploy_tripleo_heat_templates
2020-03-05 05:56:59 | deployment_options=deployment_options)
2020-03-05 05:56:59 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 477, in _try_overcloud_deploy_with_compat_yaml
2020-03-05 05:56:59 | deployment_options=deployment_options)
2020-03-05 05:56:59 | File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 224, in _heat_deploy
2020-03-05 05:56:59 | workflow_params.check_deprecated_parameters(self.clients, stack_name)
2020-03-05 05:56:59 | File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/parameters.py", line 130, in check_deprecated_parameters
2020-03-05 05:56:59 | for i in deprecated_params
2020-03-05 05:56:59 | File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/parameters.py", line 130, in <listcomp>
2020-03-05 05:56:59 | for i in deprecated_params
2020-03-05 05:56:59 | TypeError: unhashable type: 'dict'
2020-03-05 05:56:59 | unhashable type: 'dict'
2020-03-05 05:56:59 | END return value: 1

Issues appeared to be related to python3 dict issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/711480

[OpenStack Infra (hudson-openstack)]

Change abandoned by wes hayutin (<email address hidden>) on branch: master
Review: https://review.opendev.org/711480
Reason: 020-03-05 19:22:44 | FAILED - RETRYING: Check Keystone public endpoint status (2 retries left).
2020-03-05 19:22:49 | FAILED - RETRYING: Check Keystone public endpoint status (1 retries left).
2020-03-05 19:22:54 | failed: [undercloud] (item=None) => changed=false
2020-03-05 19:22:54 | attempts: 30
2020-03-05 19:22:54 | censored: 'the output has been hidden due to the fact that ''no_log: true'' was specified for this result'
2020-03-05 19:22:54 | fatal: [undercloud]: FAILED! => changed=true
2020-03-05 19:22:54 | censored: 'the output has been hidden due to the fact that ''no_log: true'' was specified for this result'
2020-03-05 19:22:54 |

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/711480
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=14f52713c4ae4fcebe68a34716037a6364522738
Submitter: Zuul
Branch: master

commit 14f52713c4ae4fcebe68a34716037a6364522738
Author: Arx Cruz <email address hidden>
Date: Thu Mar 5 16:27:47 2020 +0100

    Fix KeyError exception on check_deprecated_parameters

    heat_resource_tree_params is returning a KeyError.
    Probably this was not caught before because it always had this key.
    If the key doesn't exist, returning a empty dictionary because of
    several for loops used and NoneType is not iterable.

    Change-Id: I745c43b8b4461ae2f60834a262ab358a7434efeb
    Closes-Bug: #1866184

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 13.2.0 release.