Paunch fails to find the containers it should find during config-download step deployment

[Brendan Shephard]

Summary:

During step deployment, the playbook reaches the maximum number of attempts at this task:
https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/common/deploy-steps-tasks.yaml#L94-L110

When trying to run the paunch command manually like so:
paunch apply --debug --file /var/lib/tripleo-config/container-startup-config/step_3 --config-id=tripleo_step3 --managed-by=tripleo-Compute

We get:
$ podman ps -a --filter label=managed_by=tripleo-Undercloud --filter label=config_id=tripleo_step3 --format {{.Names}} {{.Labels.container_name}}
b''
b''
$ podman ps -a --filter label=managed_by=paunch --filter label=config_id=tripleo_step3 --format {{.Names}} {{.Labels.container_name}}
b''
b''
$ podman ps -a --filter label=managed_by=tripleo_Undercloud --format {{.Names}} {{.Labels.container_name}}
b''
b''
$ podman ps -a --filter label=managed_by=paunch --format {{.Names}} {{.Labels.container_name}}

Noting that this is an invalid podman command:
# podman ps -a --filter label=managed_by=tripleo-Undercloud --filter label=config_id=tripleo_step3 --format {{.Names}} {{.Labels.container_name}}
Error: `podman ps` takes no arguments

This is because it is missing a --format before {{.Labels.container_name}}. It should be:
# podman ps -a --filter label=managed_by=tripleo-Undercloud --filter label=config_id=tripleo_step3 --format {{.Names}} --format {{.Labels.container_name}}
nova_db_sync
nova_api_ensure_default_cell
ironic_inspector_db_sync
placement_api_db_sync
nova_api_map_cell0
ironic_db_sync
zaqar_db_sync
swift_rsync_fix
swift_copy_rings
placement_api_db_extract_data_from_nova_api
nova_api_db_sync
neutron_ovs_bridge
neutron_db_sync
mistral_db_sync
keystone_db_sync
heat_engine_db_sync
glance_api_db_sync
keystone
iscsid
ironic_inspector_get_ipa
ironic_inspector_init_dnsmasq_dhcp_hostsdir
swift_setup_srv
nova_statedir_owner
ironic_inspector_init_log

We can see that it's missing the --format label before the %s here:
https://opendev.org/openstack/paunch/src/tag/5.3.2/paunch/runner.py#L269
and
https://opendev.org/openstack/paunch/src/tag/5.3.2/paunch/runner.py#L253

Expected results:
paunch should find all of the containers that it needs to find and not fail during the config-download deployment

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/733883

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (bogdando) (<email address hidden>) on branch: master
Review: https://review.opendev.org/733883

[Brendan Shephard (bshephar)]

The quote issue is just how it's printed in debug. There is no bug here that I can find. My initial issue was the response from keystone was larger than the configured MTU on the interfaces. Calls from nova were going to keystone, but the response wasn't making it back.

I initially didn't see this error, and tried to run the paunch command manually to see where it was failing and noticed the "" issue. I've since been informed that this is just how it's printed with debug and not a real problem. So we can close this one off, not a bug.