Containers created in the docker_config section to perform various config tasks, such as running container_puppet_apply.sh to create sidecar container scripts in neutron are not run during update/upgrades because the previous container instance prevents it from being defined and run. This is pretty serious because upgrades/updates won't update the sidecar scripts so old container images will be used by the sidecar processes.
Log output:
"failed_when_result": false,
"start_containers_outputs.stdout_lines | default([]) | union(start_containers_outputs.stderr_lines | default([]))": [
"$ podman image exists undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-metadata-agent-ovn:20200110.1",
"b''",
"$ podman image exists undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute:20200110.1",
"$ podman ps -a --filter label=managed_by=tripleo-Compute --filter label=config_id=tripleo_step2 --format {{.Names}} {{.Labels.container_name}}",
"b'create_haproxy_wrapper create_haproxy_wrapper\\nnova_compute_init_log nova_compute_init_log\\n'",
"$ podman inspect --type container --format {{index .Config.Labels \"config_data\"}} create_haproxy_wrapper",
"b'{\"command\": [\"/container_puppet_apply.sh\", \"4\", \"file\", \"include ::tripleo::profile::base::neutron::ovn_metadata_agent_wrappers\"], \"detach\": false, \"image\": \"undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-neutron-metadata-agent-ovn:20200110.1\", \"net\": \"host\", \"pid\": \"host\", \"start_order\": 1, \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/var/lib/container-config-scripts/container_puppet_apply.sh:/container_puppet_apply.sh:ro\", \"/etc/puppet:/tmp/puppet-etc:ro\", \"/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro\", \"/run/openvswitch:/run/openvswitch:shared,z\", \"/var/lib/neutron:/var/lib/neutron:shared,z\"]}\\n'",
"$ podman inspect --type container --format {{index .Config.Labels \"config_data\"}} nova_compute_init_log",
"b'{\"command\": [\"/bin/bash\", \"-c\", \"chown -R nova:nova /var/log/nova\"], \"environment\": {\"TRIPLEO_DEPLOY_IDENTIFIER\": \"1579554251\"}, \"image\": \"undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-compute:20200110.1\", \"net\": \"none\", \"privileged\": false, \"user\": \"root\", \"volumes\": [\"/var/log/containers/nova:/var/log/nova:z\"]}\\n'",
"$ podman ps -a --filter label=managed_by=tripleo-Compute --format {{.Names}} {{.Labels.container_name}}",
"b'create_haproxy_wrapper create_haproxy_wrapper\\nnova_statedir_owner nova_statedir_owner\\nnova_compute_init_log nova_compute_init_log\\nnova_wait_for_compute_service nova_wait_for_compute_service\\nnova_compute nova_compute\\novn_metadata_agent ovn_metadata_agent\\novn_controller ovn_controller\\nsetup_ovs_manager setup_ovs_manager\\nnova_migration_target nova_migration_target\\nlogrotate_crond logrotate_crond\\nconfigure_cms_options configure_cms_options\\niscsid iscsid\\nnova_libvirt nova_libvirt\\nnova_virtlogd nova_virtlogd\\n'",
"Running container: nova_compute_init_log",
"Skipping existing container: nova_compute_init_log",
"Running container: create_haproxy_wrapper",
"Skipping existing container: create_haproxy_wrapper"
]
}
Removing the container manually on the affected nodes allows this to run.
Forgot to note this is being seen on stein but I suspect other branches are similarily affected.