container creation during overcloud deploy fails on c9/c8 master fs1 with "You have to remove that container to be able to reuse that name.: that name is already in use"

[Jakob Meng]

c8 master and c9 master check jobs..

* tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 [1]
* tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001 [2]

..fail during overcloud deploy with..

  tripleo_container_manage : Create containers from /var/lib/tripleo-config/container-puppet-config/step_1 | overcloud-controller-1 | 0:25:01.785431 | 0.03s
  ERROR: Can't run container container-puppet-horizon
  stderr: Error: error creating container storage: the container name "container-puppet-horizon" is already in use by "2cf2f841c4a5a816d890fab3b6140807ef9f66bf9effa718b28ae00872711dac". You have to remove that container to be able to reuse that name.: that name is already in use
  ERROR: Can't run container container-puppet-ovn_controller
  stderr: Error: error creating container storage: the container name "container-puppet-ovn_controller" is already in use by "b3a3ceecb0e4f6a45345f7622b695cf2b473ae8a8673cc5b6d68d992662a138e". You have to remove that container to be able to reuse that name.: that name is already in use
  ERROR: Can't run container container-puppet-rabbitmq
  stderr: Error: error creating container storage: the container name "container-puppet-rabbitmq" is already in use by "89b537c340e39793b06771df8c79d9ac2505f3db845b66ddf7c46a9f9527fe2b". You have to remove that container to be able to reuse that name.: that name is already in use
  FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | overcloud-controller-0 | error={"changed": false, "msg": "Failed containers: container-puppet-horizon, container-puppet-ovn_controller, container-puppet-rabbitmq"}

..or similar errors:

  ERROR: Can't run container container-puppet-neutron
  stderr: time="2022-09-13T16:36:25Z" level=info msg="podman filtering at log level info"
  time="2022-09-13T16:36:25Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
  time="2022-09-13T16:36:25Z" level=info msg="Setting parallel job count to 13"
  time="2022-09-13T16:36:25Z" level=info msg="Sysctl net.ipv4.ping_group_range=0 0 ignored in containers.conf, since Network Namespace set to host"
  Error: error creating container storage: the container name "container-puppet-neutron" is already in use by a1030cb4db738e753fd7523bc70b4672654a9abc9e0ac39d27166cb490bf6e35. You have to remove that container to be able to reuse that name: that name is already in use
  FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-0 | error={"changed": false, "msg": "Failed containers: container-puppet-neutron"}

Failures seem to be intermittent, rechecked jobs usually pass or fail on different errors.

Logs:
[1] https://logserver.rdoproject.org/82/857182/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/7ea04b3/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[2] https://logserver.rdoproject.org/03/853503/3/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/51d6860/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[Jakob Meng (jm1337)]

Build history for both jobs:

* https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001
* https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001

Both seem to be very unstable jobs.

[Ronelle Landy (rlandy)]

Will reopen if needed