Failed to pull cinder-volume and heat-api containers from victoria content provider job

[chandan kumar]

Against this patch https://review.opendev.org/c/openstack/puppet-tripleo/+/780887, standalone and undercloud upgrade jobs are failing while pulling container from victoria content provider

https://e452a2fb2885c4384409-d5fc38755365104a12acd94dfa76ef41.ssl.cf2.rackcdn.com/780887/1/check/tripleo-ci-centos-8-standalone-upgrade-victoria/282e790/logs/undercloud/home/zuul/standalone_upgrade.log

```
FATAL | Pull latest cinder_volume images | standalone | error={"attempts": 3, "changed": true, "cmd": ["podman", "pull", "158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555"], "delta": "0:00:00.370687", "end": "2021-03-25 14:29:03.435946", "msg": "non-zero return code", "rc": 125, "start": "2021-03-25 14:29:03.065259", "stderr": "Trying to pull 158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555...\n manifest unknown: manifest unknown\nError: error pulling image \"158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555\": unable to pull 158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555: unable to pull image: Error initializing source docker://158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555: Error reading manifest c8cbc230b3165c1bb1f93b0a57264555 in 158.69.66.166:5001/tripleovictoria/openstack-cinder-volume: manifest unknown: manifest unknown", "stderr_lines": ["Trying to pull 158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555...", " manifest unknown: manifest unknown", "Error: error pulling image \"158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555\": unable to pull 158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555: unable to pull image: Error initializing source docker://158.69.66.166:5001/tripleovictoria/openstack-cinder-volume:c8cbc230b3165c1bb1f93b0a57264555: Error reading manifest c8cbc230b3165c1bb1f93b0a57264555 in 158.69.66.166:5001/tripleovictoria/openstack-cinder-volume: manifest unknown: manifest unknown"], "stdout": "", "stdout_lines": []}
```

and

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3e8/780887/1/check/tripleo-ci-centos-8-undercloud-upgrade-victoria/3e88c8a/logs/undercloud/home/zuul/undercloud_upgrade.log

```
Container image prepare | undercloud | error={"changed": false, "error": "Not found image: http://158.69.66.166:5001/v2/tripleovictoria/openstack-heat-api/manifests/c8cbc230b3165c1bb1f93b0a57264555", "msg": "Error running container image prepare: Not found image: http://158.69.66.166:5001/v2/tripleovictoria/openstack-heat-api/manifests/c8cbc230b3165c1bb1f93b0a57264555", "params": {}, "success": false}
2021-03-25 14:16:09 | 2021-03-25 14:16:09.323238 | bc764e04-373e-4be8-e74e-000000000c99 | TIMING | tripleo_container_image_prepare : Container image prepare | undercloud | 0:03:19.126051 | 3.72s
2021-03-25 14:16:09 |
```

While looking at victoria content provider:

https://2b2204763b167555199d-100ff86b174195291db1bc21f098c4ef.ssl.cf1.rackcdn.com/780887/1/check/tripleo-ci-centos-8-content-provider/d101685/logs/container-builds/2d68a3ba-00ea-49be-a278-ef5dcd821504/base/os/heat-base/heat-api/heat-api-build.log

and https://2b2204763b167555199d-100ff86b174195291db1bc21f098c4ef.ssl.cf1.rackcdn.com/780887/1/check/tripleo-ci-centos-8-content-provider/d101685/logs/container-builds/2d68a3ba-00ea-49be-a278-ef5dcd821504/base/os/cinder-base/cinder-volume/cinder-volume-build.log got build

and https://2b2204763b167555199d-100ff86b174195291db1bc21f098c4ef.ssl.cf1.rackcdn.com/780887/1/check/tripleo-ci-centos-8-content-provider/d101685/job-output.txt

2021-03-25 15:08:37.920516 | primary | 127.0.0.1:5001/tripleovictoria/openstack-cinder-volume bf767ab8693feb763c7e71915d31c80c 153180238023 3 hours ago 1.51 GB

and 2021-03-25 15:08:37.921620 | primary | 127.0.0.1:5001/tripleovictoria/openstack-heat-api bf767ab8693feb763c7e71915d31c80c 3211e75e71ef 3 hours ago 978 MB

That might be a mismatch issue, adding this bug to debug it.

[Sagi (Sergey) Shnaidman (sshnaidm)]

The correct curren-tripleo for vistoria is bf767ab8693feb763c7e71915d31c80c.

The "c8cbc..." hash is from "current-tripleo-rdo":
https://trunk.rdoproject.org/centos8-victoria/current-tripleo-rdo/c8/cb/c8cbc230b3165c1bb1f93b0a57264555/

So it's probably was the previous hash and promotion happened in the middle.

[chandan kumar (chkumar246)]

We are seeing similar issue again on upgrade job.
https://d746e7a68c9276b36e1b-935e188dfa3a1a49945ee8787d93930a.ssl.cf1.rackcdn.com/781815/1/gate/tripleo-ci-centos-8-standalone-upgrade-victoria/f4f40c8/logs/undercloud/home/zuul/standalone_deploy.log
```
Pull 199.204.45.160:5001/tripleou/centos-binary-cinder-volume:dd176555e7e16a1e4eb239fb264af4ab image | standalone | error={"changed": true, "cmd": "podman pull 199.204.45.160:5001/tripleou/centos-binary-cinder-volume:dd176555e7e16a1e4eb239fb264af4ab", "delta": "0:00:00.419924", "end": "2021-03-30 08:07:28.062862", "msg": "non-zero return code", "rc": 125, "start": "2021-03-30 08:07:27.642938", "stderr": "Trying to pull 199.204.45.160:5001/tripleou/centos-binary-cinder-volume:dd176555e7e16a1e4eb239fb264af4ab...\n manifest unknown: manifest unknown\nError: error pulling image \"199.204.45.160:5001/tripleou/centos-binary-cinder-volume:dd176555e7e16a1e4eb239fb264af4ab\": unable to pull 199.204.45.160:5001/tripleou/centos-binary-cinder-volume:dd176555e7e16a1e4eb239fb264af4ab: unable to pull image: Error initializing source docker://199.204.45.160:5001/tripleou/centos-binary-cinder-volume:dd176555e7e16a1e4eb239fb264af4ab: Error reading manifest dd176555e7e16a1e4eb239fb264af4ab in 199.204.45.160:5001/tripleou/centos-binary-cinder-volume: manifest unknown: manifest unknown", "stderr_lines": ["Trying to pull 199.204.45.160:5001/tripleou/centos-binary-cinder-volume:dd176555e7e16a1e4eb239fb264af4ab...", " manifest unknown: manifest unknown", "Error: error pulling image \"199.204.45.160:5001/tripleou/centos-binary-cinder-volume:dd176555e7e16a1e4eb239fb264af4ab\": unable to pull 199.204.45.160:5001/tripleou/centos-binary-cinder-volume:dd176555e7e16a1e4eb239fb264af4ab: unable to pull image: Error initializing source docker://199.204.45.160:5001/tripleou/centos-binary-cinder-volume:dd176555e7e16a1e4eb239fb264af4ab: Error reading manifest dd176555e7e16a1e4eb239fb264af4ab in 199.204.45.160:5001/tripleou/centos-binary-cinder-volume: manifest unknown: manifest unknown"], "stdout": "", "stdout_lines": []}
2021-03-30 08:07:28.095216 | fa163eff-4ea0-943f-7859-000000000e4f | TIMING | tripleo_container_tag : Pull 199.204.45.160:5001/tripleou/centos-binary-cinder-volume:dd176555e7e16a1e4eb239fb264af4ab image | standalone | 0:02:27.690513 | 0.60s

```

and

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ca2/781815/1/gate/tripleo-ci-centos-8-undercloud-upgrade-victoria/ca2e04c/logs/undercloud/var/log/tripleo-container-image-prepare.log

```
 http://192.168.24.1:8787 "POST /v2/tripleou/centos-binary-ironic-api/blobs/uploads/ HTTP/1.1" 404 196
2021-03-30 08:04:52,622 51555 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 33f8b8d9fae958ec3b56b342534714c2d6e28787, status 404, reason Not Found, text <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>

2021-03-30 08:04:52,622 51554 DEBUG urllib3.connectionpool [ ] http://192.168.24.1:8787 "GET /v2/ HTTP/1.1" 200 2
2021-03-30 08:04:52,622 51556 INFO tripleo_common.image.imag...

[Rabi Mishra (rabi)]

I thought we agreed to get rid of the upgrade jobs and make them periodic? I guess we removed them from master already.

[Marios Andreou (marios-b)]

@Rabi Yes it's coming in due course. We have removed the master/non voting ones. It is TBD if we will be adding them for Xena.

Part of the wait is for the upgrades DFG to increase their downstream CI to fill in the gap.

We do and have had these jobs in the periodic/integration lines for a long time (and they are voting ie in the promotion criteria) already. We may add them into the upstream periodic lines too, once we remove from check/gate.

[Ronelle Landy (rlandy)]

old - closing this out