cs9 sc01 master failing with "Error: can only create exec sessions on running containers: container state improper", "stderr_lines": ["Error: can only create exec sessions on running containers: container state improper"]

[Soniya Murlidhar Vyas]

periodic-tripleo-ci-centos-9-scenario001-standalone-master failing with "Error: can only create exec sessions on running containers: container state improper", "stderr_lines": ["Error: can only create exec sessions on running containers: container state improper"]

Following is the traceback observed;-

2022-08-25 05:05:49.943073 | fa163e02-628f-531e-573f-0000000051df | FATAL | add access to podman to collectd user | standalone | error={"changed": true, "cmd": "sudo podman exec -it collectd setfacl -R -m u:collectd:rwx /run/podman", "delta": "0:00:00.112981", "end": "2022-08-25 09:05:49.917607", "msg": "non-zero return code", "rc": 255, "start": "2022-08-25 09:05:49.804626", "stderr": "Error: can only create exec sessions on running containers: container state improper", "stderr_lines": ["Error: can only create exec sessions on running containers: container state improper"], "stdout": "", "stdout_lines": []}

For more reference please refer to the following links:-
- https://logserver.rdoproject.org/83/40083/20/check/periodic-tripleo-ci-centos-9-scenario001-standalone-master/2f66046/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

- https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario001-standalone-master/a4e337c/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

- https://logserver.rdoproject.org/56/36356/68/check/periodic-tripleo-ci-centos-9-scenario001-standalone-master/b95367a/logs/undercloud/home/zuul/standalone_deploy.log.txt.g

[Douglas Viroel (dviroel)]

The error message says that collectd container is not running yet, we might need to improve this code. The error happens on a recent code added:

https://opendev.org/openstack/tripleo-heat-templates/commit/298463b8104665b7cd7afea70c6d2952384d6b89

[Matthias Runge (mrunge)]

fix proposed https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854755

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Takashi Kajinami <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854814

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854991

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854814
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/8e8efcfb3a1770cb7b0e5ab2b8bb8fb6d441b349
Submitter: "Zuul (22348)"
Branch: master

commit 8e8efcfb3a1770cb7b0e5ab2b8bb8fb6d441b349
Author: Takashi Kajinami <email address hidden>
Date: Sat Aug 27 01:05:07 2022 +0900

    Set /run/podman ACL before starting collectd

    When collectd sensubility is enabled, the collectd requires access to
    /run/podman. This change adds a separate container to update ACL of
    the path before starting the collectd container.

    Closes-Bug: #1987641
    Change-Id: I6dd514a5e52c01562645011a51afa989a60e0f93

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854991
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/fcb4a85b97e753f688c65be2c1108b2d671c54fb
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit fcb4a85b97e753f688c65be2c1108b2d671c54fb
Author: Takashi Kajinami <email address hidden>
Date: Sat Aug 27 01:05:07 2022 +0900

    Set /run/podman ACL before starting collectd

    When collectd sensubility is enabled, the collectd requires access to
    /run/podman. This change adds a separate container to update ACL of
    the path before starting the collectd container.

    Backport note:
    This change removes usage of role specific container images which is
    not present in stable/wallaby.

    Closes-Bug: #1987641
    Change-Id: I6dd514a5e52c01562645011a51afa989a60e0f93
    (cherry picked from commit 8e8efcfb3a1770cb7b0e5ab2b8bb8fb6d441b349)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 17.0.0 release.