Container image upload can be unreliable -- retry needed?

[Attila Darazs]

I'm trying to locally reproduce a containerized environment on my virthost and my deployment stopped with:

TASK [overcloud-prep-containers : Prepare for the containerized deployment]

in the logs I see:

2017-06-23 14:11:44 | + openstack overcloud container image upload --verbose --config-file /usr/share/openstack-tripleo-common/container-images/overcloud_containers.yaml
2017-06-23 14:11:45 | START with options: [u'overcloud', u'container', u'image', u'upload', u'--verbose', u'--config-file', u'/usr/share/openstack-tripleo-common/container-images/overcloud_containers.yaml']
2017-06-23 14:11:45 | command: overcloud container image upload -> tripleoclient.v1.container_image.UploadImage (auth=False)
2017-06-23 14:11:45 | Using config files: [u'/usr/share/openstack-tripleo-common/container-images/overcloud_containers.yaml']
2017-06-23 14:11:45 | imagename: tripleoupstream/centos-binary-aodh-api:latest
2017-06-23 14:12:59 | Completed upload for docker image tripleoupstream/centos-binary-aodh-api:latest

[..]

2017-06-23 14:15:04 | imagename: tripleoupstream/centos-binary-cinder-api:latest
2017-06-23 14:15:40 | ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
2017-06-23 14:15:40 | END return value: 1

Since then I restarted the script several time and it gets further and further, but it still did not get to the end of it in ~8 runs.

Either we need a retry in the quickstart preparation script, or rather the "openstack overcloud container image upload" command should have a retry, because this will be a problem for reproducers outside of tripleo upstream ci where the network connection might not be perfect.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/477521

[Attila Darazs (adarazs)]

This might have been a result of os-collect-config going rogue and restarting most services every few minutes (which might be a separate bug) but this might not be a real issue, and if the services are stable upstream, it shouldn't fail.

I keep investigating.

[Attila Darazs (adarazs)]

This was definitely an artifact of another bug. Closing this.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Attila Darazs (<email address hidden>) on branch: master
Review: https://review.openstack.org/477521
Reason: This was an artifact of another bug, abandoning.