cs9 fs39 ovb job is failing with 'RuntimeError: Ansible execution failed.'

[Soniya Murlidhar Vyas]

periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master fails with 'RuntimeError: Ansible execution failed.'

Following is the traceback observed:-
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 34, in run
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args)
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args)
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1272, in take_action
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.set_deployment_status(
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.force_reraise()
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise self.value
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1244, in take_action
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.config_download(
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/workflows/deployment.py", line 406, in config_download
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud utils.run_ansible_playbook(
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 729, in run_ansible_playbook
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise RuntimeError(err_msg)
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud RuntimeError: Ansible execution failed. playbook: /home/zuul/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/zuul/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
2022-06-10 21:12:24 | 2022-06-10 21:12:24.238 144954 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 
2022-06-10 21:12:24 | 2022-06-10 21:12:24.258 144954 ERROR openstack [-] Ansible execution failed. playbook: /home/zuul/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/zuul/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh: RuntimeError: Ansible execution failed. playbook: /home/zuul/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/zuul/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
2022-06-10 21:12:24 | 2022-06-10 21:12:24.298 144954 INFO osc_lib.shell [-] END return value: 1
2022-06-10 21:12:25 | + status_code=1
2022-06-10 21:12:25 | + exit 1

For more details please refer the following links:-

1. https://logserver.rdoproject.org/55/36255/89/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/ca5aea4/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
2. https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/9315921/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
3. https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/ea97646/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[chandan kumar (chkumar246)]

By taking a look at the history of the job https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master&skip=0

Here is What I am seeing in these jobs
- Jobs are failing at overcloud Deploy or tempest failure.
Timing is hard to guess, sometime Overcloud Deploy takes 2 hours 20 mins or 2 hours 10 mins.
With a successful tempest run the jobs takes 2 hr 25 mins.

[chandan kumar (chkumar246)]

May be we can close this bug as a invalid and track the issues https://bugs.launchpad.net/tripleo/+bug/1971465