periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby is failing standlaone deploy starting container collectd - container state improper

[Ronelle Landy]

periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby is failing the standalone deployment with:

2022-09-17 23:34:17.840964 | fa163ea2-dc2b-a8de-de5d-000000004694 | FATAL | add access to podman to collectd user | standalone | error={"changed": true, "cmd": "sudo podman exec -it collectd setfacl -R -m u:collectd:rwx /run/podman", "delta": "0:00:00.125872", "end": "2022-09-18 03:34:17.811303", "msg": "non-zero return code", "rc": 255, "start": "2022-09-18 03:34:17.685431", "stderr": "Error: can only create exec sessions on running containers: container state improper", "stderr_lines": ["Error: can only create exec sessions on running containers: container state improper"], "stdout": "", "stdout_lines": []}

Failed containers shows:

https://logserver.rdoproject.org/55/36255/133/check/periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby/9337528/logs/undercloud/var/log/extra/failed_containers.log.txt.gz

036d2fefd1a8 192.168.24.1:8787/tripleowallabycentos9/openstack-collectd:93a6958b4b3d2772324fb241e308943a-updated-20220917225350 kolla_start 7 minutes ago Exited (134) 6 minutes ago (unhealthy) collectd

Container log:

https://logserver.rdoproject.org/55/36255/133/check/periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby/9337528/logs/undercloud/var/log/extra/podman/containers/collectd/stdout.log.txt.gz

The error started on 09/16:
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby&skip=0

Example logs:

https://logserver.rdoproject.org/55/36255/133/check/periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby/9337528/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby/7aed123/

[Ronelle Landy (rlandy)]

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby/7aed123/logs/undercloud/var/log/container_image_update_output.log.txt.gz is the container update list

https://github.com/openstack/tripleo-heat-templates/commit/2ab74ea06543659568d6899d3abb067edaa0b50f
possibly related

[Ronelle Landy (rlandy)]

Revert was -1 - fix is in https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854991/

[Ronelle Landy (rlandy)]

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby&skip=0 - passes - closing out