Scenario001 master failing on standalone deploy with FATAL | Gather podman infos | standalone | error={"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}

[Sandeep Yadav]

Description:-

Scenario001 master failing on standalone deploy with FATAL | Gather podman infos | standalone | error={"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}

Issue started on 06th may:-

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario001-standalone-master

Logs are not detailed enough, but these snippets are observed:-

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/ffb7cc1/job-output.txt
~~~
2021-06-06 18:15:35.409697 | primary |
2021-06-06 18:15:35.409778 | primary | TASK [tripleo.operator.tripleo_deploy : Standalone deploy] *********************
2021-06-06 18:15:35.409836 | primary | Sunday 06 June 2021 18:15:35 +0000 (0:00:00.643) 0:06:03.030 ***********
2021-06-06 18:54:50.653970 | primary | fatal: [undercloud]: FAILED! => {
2021-06-06 18:54:50.654234 | primary | "ansible_job_id": "623894129255.47881",
2021-06-06 18:54:50.654255 | primary | "changed": false,
2021-06-06 18:54:50.654289 | primary | "cmd": "openstack tripleo deploy --templates $DEPLOY_TEMPLATES --yes --output-dir $DEPLOY_OUTPUT_DIR --stack $DEPLOY_STACK --standalone-role $DEPLOY_STANDALONE_ROLE --timeout $DEPLOY_TIMEOUT_ARG -e /usr/share/openstack-tripleo-heat-templates/environments/standalone/standalone-tripleo.yaml -e /home/zuul/containers-prepare-parameters.yaml -e /home/zuul/standalone_parameters.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml -e /usr/share/openstack-tripleo-heat-templates/ci/environments/scenario001-standalone.yaml -r $DEPLOY_ROLES_FILE --deployment-user $DEPLOY_DEPLOYMENT_USER --local-ip $DEPLOY_LOCAL_IP --control-virtual-ip $DEPLOY_CONTROL_VIP >/home/zuul/standalone_deploy.log 2>&1",
2021-06-06 18:54:50.654301 | primary | "delta": "0:39:08.357938",
2021-06-06 18:54:50.654308 | primary | "end": "2021-06-06 18:54:45.523267",
2021-06-06 18:54:50.654325 | primary | "finished": 1,
2021-06-06 18:54:50.654340 | primary | "rc": 1,
2021-06-06 18:54:50.654351 | primary | "start": "2021-06-06 18:15:37.165329"
2021-06-06 18:54:50.654357 | primary | }
~~~

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/ffb7cc1/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
~~~
2021-06-06 18:53:15.504335 | fa163ea2-f424-6f3f-0d00-000000003d2b | FATAL | Gather podman infos | standalone | error={"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
~~~

Another example:
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/6c43f11/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/795077

[Sandeep Yadav (sandeepyadav93)]

Ran testproject: https://review.rdoproject.org/r/c/testproject/+/32054 with depends-On: https://review.opendev.org/c/openstack/tripleo-ansible/+/795077 to get more debug logs.

[Rabi Mishra (rabi)]

https://review.opendev.org/c/openstack/tripleo-ansible/+/789928 was supposed to fix the issue AFAIR. Looks like we still have the issue.

[Sandeep Yadav (sandeepyadav93)]

Testpatch logs:-

https://logserver.rdoproject.org/54/32054/27/check/periodic-tripleo-ci-centos-8-scenario001-standalone-master/600f0b2/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

~~~
2021-06-07 16:49:32.361146 | fa163e7d-03f0-933b-18bb-00000000610e | FATAL | Gather podman infos | standalone | error={"changed": false, "msg": "Unable to get list of containers: Error: container d7e40ed7d8c0788c5e75268f0daa5808065c752ddc0d99a6637f2ed92c2cde94 does not exist in database: no such container\n"}
~~~

[Sagi (Sergey) Shnaidman (sshnaidm)]

After we removed ansible podman module from tripleo-ansible, the workaround has gone as well. I'll release a new RPM of ansible podman collection today with the workaround and it should solve it.

[Sagi (Sergey) Shnaidman (sshnaidm)]

Adding to collection: https://github.com/containers/ansible-podman-collections/pull/269

[Sagi (Sergey) Shnaidman (sshnaidm)]

And updated in RDO: https://review.rdoproject.org/r/c/rdoinfo/+/34033
So let's just wait for update of RPM

[wes hayutin (weshayutin)]

scenario001 in master intergration is now passing.. thanks Sagi!

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Sandeep Yadav <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/795077
Reason: This was for a test and not needed anymore.