Against this patch https://review.opendev.org/c/openstack/puppet-tripleo/+/780887, standalone and undercloud upgrade jobs are failing while pulling container from victoria content provider
https://e452a2fb2885c4384409-d5fc38755365104a12acd94dfa76ef41.ssl.cf2.rackcdn.com/780887/1/check/tripleo-ci-centos-8-standalone-upgrade-victoria/282e790/logs/undercloud/home/zuul/standalone_upgrade.log
```
FATAL | Pull latest cinder_volume images | standalone | error={"attempts": 3, "changed": true, "cmd": ["podman", "pull", "158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555"], "delta": "0:00:00.370687", "end": "2021-03-25 14:29:03.435946", "msg": "non-zero return code", "rc": 125, "start": "2021-03-25 14:29:03.065259", "stderr": "Trying to pull 158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555...\n manifest unknown: manifest unknown\nError: error pulling image \"158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555\": unable to pull 158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555: unable to pull image: Error initializing source docker://158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555: Error reading manifest c8cbc230b3165c1bb1f93b0a57264555 in 158.69.66.166:5001/tripleovictoria/openstack-cinder-volume: manifest unknown: manifest unknown", "stderr_lines": ["Trying to pull 158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555...", " manifest unknown: manifest unknown", "Error: error pulling image \"158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555\": unable to pull 158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555: unable to pull image: Error initializing source docker://158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555: Error reading manifest c8cbc230b3165c1bb1f93b0a57264555 in 158.69.66.166:5001/tripleovictoria/openstack-cinder-volume: manifest unknown: manifest unknown"], "stdout": "", "stdout_lines": []}
```
and
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3e8/780887/1/check/tripleo-ci-centos-8-undercloud-upgrade-victoria/3e88c8a/logs/undercloud/home/zuul/undercloud_upgrade.log
```
Container image prepare | undercloud | error={"changed": false, "error": "Not found image: http://158.69.66.166:5001/v2/tripleovictoria/openstack-heat-api/manifests/c8cbc230b3165c1bb1f93b0a57264555", "msg": "Error running container image prepare: Not found image: http://158.69.66.166:5001/v2/tripleovictoria/openstack-heat-api/manifests/c8cbc230b3165c1bb1f93b0a57264555", "params": {}, "success": false}
2021-03-25 14:16:09 | 2021-03-25 14:16:09.323238 | bc764e04-373e-4be8-e74e-000000000c99 | TIMING | tripleo_container_image_prepare : Container image prepare | undercloud | 0:03:19.126051 | 3.72s
2021-03-25 14:16:09 |
```
While looking at victoria content provider:
https://2b2204763b167555199d-100ff86b174195291db1bc21f098c4ef.ssl.cf1.rackcdn.com/780887/1/check/tripleo-ci-centos-8-content-provider/d101685/logs/container-builds/2d68a3ba-00ea-49be-a278-ef5dcd821504/base/os/heat-base/heat-api/heat-api-build.log
and https://2b2204763b167555199d-100ff86b174195291db1bc21f098c4ef.ssl.cf1.rackcdn.com/780887/1/check/tripleo-ci-centos-8-content-provider/d101685/logs/container-builds/2d68a3ba-00ea-49be-a278-ef5dcd821504/base/os/cinder-base/cinder-volume/cinder-volume-build.log got build
and https://2b2204763b167555199d-100ff86b174195291db1bc21f098c4ef.ssl.cf1.rackcdn.com/780887/1/check/tripleo-ci-centos-8-content-provider/d101685/job-output.txt
2021-03-25 15:08:37.920516 | primary | 127.0.0.1:5001/tripleovictoria/openstack-cinder-volume bf767ab8693feb763c7e71915d31c80c 153180238023 3 hours ago 1.51 GB
and 2021-03-25 15:08:37.921620 | primary | 127.0.0.1:5001/tripleovictoria/openstack-heat-api bf767ab8693feb763c7e71915d31c80c 3211e75e71ef 3 hours ago 978 MB
That might be a mismatch issue, adding this bug to debug it.
The correct curren-tripleo for vistoria is bf767ab8693feb7 63c7e71915d31c8 0c.
The "c8cbc..." hash is from "current- tripleo- rdo": /trunk. rdoproject. org/centos8- victoria/ current- tripleo- rdo/c8/ cb/c8cbc230b316 5c1bb1f93b0a572 64555/
https:/
So it's probably was the previous hash and promotion happened in the middle.