tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is failing while pulling containers

[wes hayutin]

https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates

tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates openstack/tripleo-heat-templates master check 758811,8 40 mins 35 secs 2021-03-24 18:54:03 FAILURE
tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates openstack/tripleo-heat-templates master check 782288,1 33 mins 37 secs 2021-03-24 18:26:11 FAILURE
tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates openstack/tripleo-heat-templates master gate 782302,1 36 mins 10 secs 2021-03-24 17:19:03 FAILURE
tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates openstack/tripleo-heat-templates master check 746487,1 34 mins 12 secs 2021-03-24 16:51:03 FAILURE
tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates openstack/tripleo-heat-templates master check 779801,3 34 mins 38 secs 2021-03-24 16:11:56 FAILURE
tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates openstack/tripleo-heat-templates master gate 781069,2 32 mins 35 secs 2021-03-24 15:30:50 FAILURE

2021-03-24 19:27:02.047599 | fa163ecb-39d5-6d34-76ae-000000000cf2 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: http://149.202.163.180:5001/v2/tripleomaster/openstack-neutron-server/manifests/ec263196cc051357ad315aa33ed62b26", "msg": "Error running container image prepare: Not found image: http://149.202.163.180:5001/v2/tripleomaster/openstack-neutron-server/manifests/ec263196cc051357ad315aa33ed62b26", "params": {}, "success": false}

2021-03-24 18:53:17.101377 | fa163e40-8eea-853f-4e40-000000000cf2 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: http://198.72.124.68:5001/v2/tripleomaster/openstack-rsyslog/manifests/ec263196cc051357ad315aa33ed62b26", "msg": "Error running container image prepare: Not found image: http://198.72.124.68:5001/v2/tripleomaster/openstack-rsyslog/manifests/ec263196cc051357ad315aa33ed62b26", "params": {}, "success": false}

021-03-24 17:46:45.453681 | fa163e7d-917d-8779-da5b-000000000cf2 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: http://198.72.124.66:5001/v2/tripleomaster/openstack-keystone/manifests/ec263196cc051357ad315aa33ed62b26", "msg": "Error running container image prepare: Not found image: http://198.72.124.66:5001/v2/tripleomaster/openstack-keystone/manifests/ec263196cc051357ad315aa33ed62b26", "params": {}, "success": false}

[wes hayutin (weshayutin)]

master promoted today, may be related

[wes hayutin (weshayutin)]

https://review.opendev.org/q/topic:%22container-updates%22+(status:open%20OR%20status:merged)

[Juan Badia Payno (jbadiapa)]

Pasted part of the logs in case they are deleted.

I think we hit this issue again at https://zuul.opendev.org/t/openstack/build/eec4b913fae44df6a33dfc2ee0442a2f/log/logs/undercloud/home/zuul/undercloud_install.log#1668

=====
2021-08-18 14:24:29.215817 | bc764e20-0d31-a0c7-a66d-000000000de1 | TASK | Container image prepare

2021-08-18 14:24:31.130811 | bc764e20-0d31-a0c7-a66d-000000000de1 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: http://104.130.169.80:5001/v2/tripleomaster/openstack-neutron-server/manifests/a085460a23443cfc28a1dc6c667e110e", "msg": "Error running container image prepare: Not found image: http://104.130.169.80:5001/v2/tripleomaster/openstack-neutron-server/manifests/a085460a23443cfc28a1dc6c667e110e", "params": {}, "success": false}

2021-08-18 14:24:31.133750 | bc764e20-0d31-a0c7-a66d-000000000de1 | TIMING | tripleo_container_image_prepare : Container image prepare | undercloud | 0:03:03.199511 | 1.92s

PLAY RECAP *********************************************************************

localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0

undercloud : ok=209 changed=110 unreachable=0 failed=1 skipped=36 rescued=0 ignored=2
======

There are more logs at https://zuul.opendev.org/t/openstack/build/eec4b913fae44df6a33dfc2ee0442a2f/log/logs/undercloud/var/log/tripleo-container-image-prepare.log

=====
2021-08-18 14:24:30,943 53106 DEBUG urllib3.connectionpool [ ] http://192.168.24.1:8787 "POST /v2/tripleomaster/openstack-neutron-server/blobs/uploads/ HTTP/1.1" 404 196

2021-08-18 14:24:30,944 53106 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 47c1dea99076185243b76070ac68d51264e40eea, status 404, reason Not Found, text <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<html><head>

<title>404 Not Found</title>

</head><body>

<h1>Not Found</h1>

<p>The requested URL was not found on this server.</p>

</body></html>

2021-08-18 14:24:30,944 53111 DEBUG urllib3.connectionpool [ ] http://192.168.24.1:8787 "GET /v2/ HTTP/1.1" 200 2

2021-08-18 14:24:30,944 53106 DEBUG urllib3.util.retry [ ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None)

2021-08-18 14:24:30,945 53107 DEBUG urllib3.connectionpool [ ] http://192.168.24.1:8787 "POST /v2/tripleomaster/openstack-ironic-inspector/blobs/uploads/ HTTP/1.1" 404 196
..
.
.
..
2021-08-18 14:24:30,960 53109 DEBUG urllib3.connectionpool [ ] http://104.130.169.80:5001 "GET /v2/tripleomaster/openstack-ironic-conductor/manifests/a085460a23443cfc28a1dc6c667e110e HTTP/1.1" 404 122

2021-08-18 14:24:30,960 53107 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 75b572ca18e9d1a2cf0765664e0571517b1b0da6, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"a085460a23443cfc28a1dc6c667e110e"}}]}

2021-08-18 14:24:30,960 53106 DEBUG urllib3.connectionpool [ ] http://104.130.169.80:5001 "GET /v2/tripleomaster/openstack-neutron-server/manifests/a085460a23443cfc28a1dc6c667e110e HTTP/1.1" 404 122

2...