Wallaby(Centos stream 9) - Overcloud failed at Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3

[umamaheswar Reddy]

2023-05-08 06:40:11.902491 | e43d1a24-19e0-d5ef-b187-000000004aac | TASK | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3
2023-05-08 10:24:06.767 430897 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleoepns8_e7 ] cleaned up
2023-05-08 10:24:06.768 430897 ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: timeout, Return Code: 254, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
2023-05-08 10:24:06.769 430897 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: timeout, Return Code: 254, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
Host 10.133.100.165 not found in /home/stack/.ssh/known_hosts
Overcloud Endpoint: http://10.*.*.*:5000
Overcloud Horizon Dashboard URL: http://10.*.*.*:80/dashboard
Overcloud rc file: /home/stack/overcloud-deploy/overcloud/overcloudrc and /home/stack/overcloudrc
Overcloud Deployed with error
2023-05-08 10:24:07.167 430897 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Stopping ephemeral heat.
2023-05-08 10:24:07.264 430897 INFO tripleoclient.heat_launcher [-] Pod does not exist: ephemeral-heat
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: timeout, Return Code: 254, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 32, in run
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args)
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args)
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/cliff/command.py", line 185, in run
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1331, in take_action
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.set_deployment_status(
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.force_reraise()
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise self.value
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1303, in take_action
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.config_download(
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/workflows/deployment.py", line 386, in config_download
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud utils.run_ansible_playbook(
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 775, in run_ansible_playbook
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise RuntimeError(err_msg)
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: timeout, Return Code: 254, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
2023-05-08 10:24:08.324 430897 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud
2023-05-08 10:24:08.326 430897 ERROR openstack [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: timeout, Return Code: 254, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: timeout, Return Code: 254, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
2023-05-08 10:24:08.328 430897 INFO osc_lib.shell [-] END return value: 1

Steps to reproduce:
1.Install Undercloud
2.Overcloud image build and Upload
3.Install overcloud using below command,
openstack overcloud deploy \
  --templates --deployed-server \
  --disable-validations \
  -e /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-environment.yaml \
  -e /home/stack/containers-prepare-parameter.yaml \
  -e /home/stack/templates/node-info.yaml \
  -e /home/stack/templates/predictive_ips.yaml \
  -e ~/templates/hostnamemap.yaml \
  -e ~/templates/deployed-server-network-environment.yaml

Actual result:
Overcloud deployed with error

Expected result:
Overcloud deployed with no errors

[Brendan Shephard (bshephar)]

Hey, are you able to run that script with -vvv to get the Ansible debug output:
/home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh -vvv

It looks like it might have timed out waiting for that task to execute. Looks like we're not seeing any related issues in CI environments:
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-containers-multinode

So it could be something configuration specific. If it's possible, it would also be nice if you can attach a tar file containing your overcloud deploy command along with any of the configuration templates you're including.