tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates fails in task Container image prepare
https://83b304842de438a4ce44-164ae1091eb5a377552df92dcdcf8170.ssl.cf2.rackcdn.com/798522/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/1db6c15/logs/undercloud/home/zuul/undercloud_install.log
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bee/798786/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/beedd6d/logs/undercloud/home/zuul/undercloud_install.log
FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: http://77.81.6.22:5001/v2/tripleomaster/openstack-heat-all/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4", "msg": "Error running container image prepare: Not found image: http://77.81.6.22:5001/v2/tripleomaster/openstack-heat-all/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4", "params": {}, "success": false}
In tripleo-container-image-prepare.log
https://83b304842de438a4ce44-164ae1091eb5a377552df92dcdcf8170.ssl.cf2.rackcdn.com/798522/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/1db6c15/logs/undercloud/var/log/tripleo-container-image-prepare.log
2021-06-30 06:10:08,647 63843 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/ HTTP/1.1" 200 2
2021-06-30 06:10:08,647 63843 DEBUG tripleo_common.image.image_uploader [ ] http://77.81.6.22:5001/v2/ status code 200
2021-06-30 06:10:08,650 63842 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/tripleomaster/openstack-mariadb/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4 HTTP/1.1" 404 122
2021-06-30 06:10:08,650 63849 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/tripleomaster/openstack-rsyslog/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4 HTTP/1.1" 404 122
2021-06-30 06:10:08,650 63845 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/tripleomaster/openstack-neutron-dhcp-agent/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4 HTTP/1.1" 404 122
2021-06-30 06:10:08,650 63842 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id acec9e2aad166c503f766cf64f11412068fbf059, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"4dfe86fffbfd37ec10e4f2e101dc89a4"}}]}
2021-06-30 06:10:08,650 63849 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 99894970a100d6e65bbb21b0bc7bdcaaa58bb857, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"4dfe86fffbfd37ec10e4f2e101dc89a4"}}]}
2021-06-30 06:10:08,650 63845 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 6391ea8c9017c1538c704d257f18b5a09904e03b, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"4dfe86fffbfd37ec10e4f2e101dc89a4"}}]}
2021-06-30 06:10:08,650 63842 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleomaster/openstack-mariadb:4dfe86fffbfd37ec10e4f2e101dc89a4] Failed uploading the target image
2021-06-30 06:10:08,651 63849 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleomaster/openstack-rsyslog:4dfe86fffbfd37ec10e4f2e101dc89a4] Failed uploading the target image
2021-06-30 06:10:08,651 63845 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleomaster/openstack-neutron-dhcp-agent:4dfe86fffbfd37ec10e4f2e101dc89a4] Failed uploading the target image
2021-06-30 06:10:08,651 63843 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/tripleomaster/openstack-ironic-inspector/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4 HTTP/1.1" 404 122
2021-06-30 06:10:08,651 63843 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 3baad03c50488dda8243a78353292b4c02d21067, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"4dfe86fffbfd37ec10e4f2e101dc89a4"}}]}
2021-06-30 06:10:08,651 63842 INFO tripleo_common.image.image_uploader [ ] [77.81.6.22:5001/tripleomaster/openstack-qdrouterd:4dfe86fffbfd37ec10e4f2e101dc89a4] Starting upload image process
2021-06-30 06:10:08,652 63843 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleomaster/openstack-ironic-inspector:4dfe86fffbfd37ec10e4f2e101dc89a4] Failed uploading the target image
2021-06-30 06:10:08,652 63842 DEBUG urllib3.util.retry [ ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None)
2021-06-30 06:10:08,653 63842 DEBUG urllib3.connectionpool [ ] Starting new HTTP connection (1): 192.168.24.1:8787
2021-06-30 06:10:08,654 63842 DEBUG urllib3.connectionpool [ ] http://192.168.24.1:8787 "GET /v2/ HTTP/1.1" 200 2
2021-06-30 06:10:08,655 63842 DEBUG tripleo_common.image.image_uploader [ ] http://192.168.24.1:8787/v2/ status code 200
2021-06-30 06:10:08,655 63842 DEBUG urllib3.util.retry [ ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None)
2021-06-30 06:10:08,657 63842 DEBUG urllib3.connectionpool [ ] Starting new HTTP connection (1): 77.81.6.22:5001
2021-06-30 06:10:08,659 63842 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/ HTTP/1.1" 200 2
2021-06-30 06:10:08,660 63842 DEBUG tripleo_common.image.image_uploader [ ] http://77.81.6.22:5001/v2/ status code 200
2021-06-30 06:10:08,663 63842 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/tripleomaster/openstack-qdrouterd/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4 HTTP/1.1" 404 122
2021-06-30 06:10:08,664 63842 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id d51221eb628974a64c300ddfa034f49bb0e9c6b6, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"4dfe86fffbfd37ec10e4f2e101dc89a4"}}]}
2021-06-30 06:10:08,664 63842 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleomaster/openstack-qdrouterd:4dfe86fffbfd37ec10e4f2e101dc89a4] Failed uploading the target image
not sure if this is a transient issue yet - I now see a green build after the fails @ https:/ /zuul.openstack .org/builds? job_name= tripleo- ci-centos- 8-scenario000- multinode- oooq-container- updates
current-tripleo promoted last night to the new hash at 6439b21a91a11b4 64ad5b2cc147e81 cd curren- tripleo is 4dfe86fffbfd37e c10e4f2e101dc89 a4
the previous-
For some reason, the job is failing as it tries to pull the previous hash from the content provider [1]
* tag: "4dfe86fffbfd37 ec10e4f2e101dc8 9a4"
However the content provider is (correctly) building containers with the new hash [2]
provider_ dlrn_hash_ branch: &id003 64ad5b2cc147e81 cd dlrn_hash_ tag_branch: &id004 64ad5b2cc147e81 cd ip_address_ branch: &id005
master: 6439b21a91a11b4
provider_
master: 6439b21a91a11b4
registry_
master: 77.81.6.22
[1] https:/ /83b304842de438 a4ce44- 164ae1091eb5a37 7552df92dcdcf81 70.ssl. cf2.rackcdn. com/798522/ 2/check/ tripleo- ci-centos- 8-scenario000- multinode- oooq-container- updates/ 1db6c15/ logs/undercloud /home/zuul/ containers- prepare- parameter. yaml
[2] https:/ /83b304842de438 a4ce44- 164ae1091eb5a37 7552df92dcdcf81 70.ssl. cf2.rackcdn. com/798522/ 2/check/ tripleo- ci-centos- 8-scenario000- multinode- oooq-container- updates/ 1db6c15/ zuul-info/ inventory. yaml