In the promotion log from queens, we have a case where the recently added container existence check detected a missing container. Here's the (cleaned up and readable) version of the log. What comes out of ansible is all of this on a single line.
```
TASK [Check if dockerhub has all images] *************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "tag=a7fbaca2a97adda856c3ba5d5166fb1665f02bc0_85b157a9
for image in $(cat parsed_containers-queens.txt)
do
if curl --silent -fk -lSL \"https://index.docker.io/v1/repositories/tripleoqueens/centos-binary-${image}/tags/${tag}\" 1>/dev/null;
then
echo \"$image OK\";
else
echo \"$image ERROR\";
fi
done", "delta": "0:00:44.549722", "end": "2018-05-15 21:25:33.343641", "failed_when_result": true, "rc": 0, "start": "2018-05-15 21:24:48.793919", "stderr": "curl: (22) The requested URL returned error: 404 NOT FOUND", "stderr_lines": ["curl: (22) The requested URL returned error: 404 NOT FOUND"], "stdout": "aodh-api OK
aodh-evaluator OK
aodh-listener OK
aodh-notifier OK
barbican-api OK
barbican-keystone-listener OK
barbican-worker OK
ceilometer-central OK
ceilometer-compute OK
ceilometer-notification OK
ceilometer-ipmi OK
cinder-api OK
cinder-backup OK
cinder-scheduler OK
cinder-volume OK
collectd OK
congress-api OK
cron OK
ec2-api OK
etcd OK
fluentd OK
glance-api OK
gnocchi-api OK
gnocchi-metricd OK
gnocchi-statsd OK
haproxy OK
heat-all OK
heat-api-cfn OK
heat-api OK
heat-engine OK
horizon OK
ironic-api OK
ironic-conductor OK
ironic-inspector OK
ironic-pxe OK
iscsid OK
keepalived OK
keystone OK
manila-api OK
manila-base OK
manila-scheduler OK
manila-share OK
mariadb OK
memcached OK
mistral-api OK
mistral-engine OK
mistral-executor OK
mistral-event-engine OK
multipathd OK
neutron-dhcp-agent OK
neutron-l3-agent OK
neutron-metadata-agent OK
neutron-openvswitch-agent OK
neutron-sriov-agent OK
neutron-server-opendaylight OK
neutron-server-ovn OK
neutron-metadata-agent-ovn OK
neutron-server OK
nova-api OK
nova-compute-ironic OK
nova-compute OK
nova-conductor OK
nova-consoleauth OK
nova-libvirt ERROR <=== THE MISSING CONTAINER IS DETECTED
nova-novncproxy OK
nova-placement-api OK
nova-scheduler OK
novajoin-notifier OK
novajoin-server OK
octavia-base OK
octavia-api OK
octavia-health-manager OK
octavia-housekeeping OK
octavia-worker OK
opendaylight OK
ovn-base OK
ovn-controller OK
ovn-nb-db-server OK
ovn-northd OK
ovn-sb-db-server OK
panko-api OK
qdrouterd OK
rabbitmq OK
redis OK
sahara-api OK
sahara-engine OK
sensu-client OK
skydive-agent OK
skydive-analyzer OK
swift-account OK
swift-container OK
swift-object OK
swift-proxy-server OK
zaqar OK
tempest OK
rsyslog-base OK
base OK
openstack-base OK", "stdout_lines": ["aodh-api OK", "aodh-evaluator OK", "aodh-listener OK", "aodh-notifier OK", "barbican-api OK", "barbican-keystone-listener OK", "barbican-worker OK", "ceilometer-central OK", "ceilometer-compute OK", "ceilometer-notification OK", "ceilometer-ipmi OK", "cinder-api OK", "cinder-backup OK", "cinder-scheduler OK", "cinder-volume OK", "collectd OK", "congress-api OK", "cron OK", "ec2-api OK", "etcd OK", "fluentd OK", "glance-api OK", "gnocchi-api OK", "gnocchi-metricd OK", "gnocchi-statsd OK", "haproxy OK", "heat-all OK", "heat-api-cfn OK", "heat-api OK", "heat-engine OK", "horizon OK", "ironic-api OK", "ironic-conductor OK", "ironic-inspector OK", "ironic-pxe OK", "iscsid OK", "keepalived OK", "keystone OK", "manila-api OK", "manila-base OK", "manila-scheduler OK", "manila-share OK", "mariadb OK", "memcached OK", "mistral-api OK", "mistral-engine OK", "mistral-executor OK", "mistral-event-engine OK", "multipathd OK", "neutron-dhcp-agent OK", "neutron-l3-agent OK", "neutron-metadata-agent OK", "neutron-openvswitch-agent OK", "neutron-sriov-agent OK", "neutron-server-opendaylight OK", "neutron-server-ovn OK", "neutron-metadata-agent-ovn OK", "neutron-server OK", "nova-api OK", "nova-compute-ironic OK", "nova-compute OK", "nova-conductor OK", "nova-consoleauth OK", "nova-libvirt ERROR", "nova-novncproxy OK", "nova-placement-api OK", "nova-scheduler OK", "novajoin-notifier OK", "novajoin-server OK", "octavia-base OK", "octavia-api OK", "octavia-health-manager OK", "octavia-housekeeping OK", "octavia-worker OK", "opendaylight OK", "ovn-base OK", "ovn-controller OK", "ovn-nb-db-server OK", "ovn-northd OK", "ovn-sb-db-server OK", "panko-api OK", "qdrouterd OK", "rabbitmq OK", "redis OK", "sahara-api OK", "sahara-engine OK", "sensu-client OK", "skydive-agent OK", "skydive-analyzer OK", "swift-account OK", "swift-container OK", "swift-object OK", "swift-proxy-server OK", "zaqar OK", "tempest OK", "rsyslog-base OK", "base OK", "openstack-base OK"]}
```
The full log is attached. This check was recently added.
https://github.com/rdo-infra/ci-config/commit/485534f237fb9660ab95e5613c67873ba60d6ac1
We should add a debug: task to clearly list out the failed container, so ruck/rover can quickly determine that something is wrong. Additionally (bonus points) if we could create and artifact (file), as since the promoter runs quite frequently, while this will stop a promotion as designed, unless someone is looking for it we won't notice instances of container uploads failing.
