Error running container image prepare: Not found image

Bug #1934097 reported by Ananya Banerjee
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Invalid
Critical
Unassigned

Bug Description

tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates fails in task Container image prepare
https://83b304842de438a4ce44-164ae1091eb5a377552df92dcdcf8170.ssl.cf2.rackcdn.com/798522/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/1db6c15/logs/undercloud/home/zuul/undercloud_install.log

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bee/798786/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/beedd6d/logs/undercloud/home/zuul/undercloud_install.log

FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: http://77.81.6.22:5001/v2/tripleomaster/openstack-heat-all/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4", "msg": "Error running container image prepare: Not found image: http://77.81.6.22:5001/v2/tripleomaster/openstack-heat-all/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4", "params": {}, "success": false}

In tripleo-container-image-prepare.log
https://83b304842de438a4ce44-164ae1091eb5a377552df92dcdcf8170.ssl.cf2.rackcdn.com/798522/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/1db6c15/logs/undercloud/var/log/tripleo-container-image-prepare.log

2021-06-30 06:10:08,647 63843 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/ HTTP/1.1" 200 2
2021-06-30 06:10:08,647 63843 DEBUG tripleo_common.image.image_uploader [ ] http://77.81.6.22:5001/v2/ status code 200
2021-06-30 06:10:08,650 63842 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/tripleomaster/openstack-mariadb/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4 HTTP/1.1" 404 122
2021-06-30 06:10:08,650 63849 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/tripleomaster/openstack-rsyslog/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4 HTTP/1.1" 404 122
2021-06-30 06:10:08,650 63845 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/tripleomaster/openstack-neutron-dhcp-agent/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4 HTTP/1.1" 404 122
2021-06-30 06:10:08,650 63842 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id acec9e2aad166c503f766cf64f11412068fbf059, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"4dfe86fffbfd37ec10e4f2e101dc89a4"}}]}

2021-06-30 06:10:08,650 63849 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 99894970a100d6e65bbb21b0bc7bdcaaa58bb857, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"4dfe86fffbfd37ec10e4f2e101dc89a4"}}]}

2021-06-30 06:10:08,650 63845 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 6391ea8c9017c1538c704d257f18b5a09904e03b, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"4dfe86fffbfd37ec10e4f2e101dc89a4"}}]}

2021-06-30 06:10:08,650 63842 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleomaster/openstack-mariadb:4dfe86fffbfd37ec10e4f2e101dc89a4] Failed uploading the target image
2021-06-30 06:10:08,651 63849 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleomaster/openstack-rsyslog:4dfe86fffbfd37ec10e4f2e101dc89a4] Failed uploading the target image
2021-06-30 06:10:08,651 63845 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleomaster/openstack-neutron-dhcp-agent:4dfe86fffbfd37ec10e4f2e101dc89a4] Failed uploading the target image
2021-06-30 06:10:08,651 63843 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/tripleomaster/openstack-ironic-inspector/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4 HTTP/1.1" 404 122
2021-06-30 06:10:08,651 63843 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 3baad03c50488dda8243a78353292b4c02d21067, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"4dfe86fffbfd37ec10e4f2e101dc89a4"}}]}

2021-06-30 06:10:08,651 63842 INFO tripleo_common.image.image_uploader [ ] [77.81.6.22:5001/tripleomaster/openstack-qdrouterd:4dfe86fffbfd37ec10e4f2e101dc89a4] Starting upload image process
2021-06-30 06:10:08,652 63843 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleomaster/openstack-ironic-inspector:4dfe86fffbfd37ec10e4f2e101dc89a4] Failed uploading the target image
2021-06-30 06:10:08,652 63842 DEBUG urllib3.util.retry [ ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None)
2021-06-30 06:10:08,653 63842 DEBUG urllib3.connectionpool [ ] Starting new HTTP connection (1): 192.168.24.1:8787
2021-06-30 06:10:08,654 63842 DEBUG urllib3.connectionpool [ ] http://192.168.24.1:8787 "GET /v2/ HTTP/1.1" 200 2
2021-06-30 06:10:08,655 63842 DEBUG tripleo_common.image.image_uploader [ ] http://192.168.24.1:8787/v2/ status code 200
2021-06-30 06:10:08,655 63842 DEBUG urllib3.util.retry [ ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None)
2021-06-30 06:10:08,657 63842 DEBUG urllib3.connectionpool [ ] Starting new HTTP connection (1): 77.81.6.22:5001
2021-06-30 06:10:08,659 63842 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/ HTTP/1.1" 200 2
2021-06-30 06:10:08,660 63842 DEBUG tripleo_common.image.image_uploader [ ] http://77.81.6.22:5001/v2/ status code 200
2021-06-30 06:10:08,663 63842 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/tripleomaster/openstack-qdrouterd/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4 HTTP/1.1" 404 122
2021-06-30 06:10:08,664 63842 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id d51221eb628974a64c300ddfa034f49bb0e9c6b6, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"4dfe86fffbfd37ec10e4f2e101dc89a4"}}]}

2021-06-30 06:10:08,664 63842 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleomaster/openstack-qdrouterd:4dfe86fffbfd37ec10e4f2e101dc89a4] Failed uploading the target image

Changed in tripleo:
status: New → Triaged
importance: Undecided → Critical
milestone: none → xena-2
tags: added: promotion-blocker
Revision history for this message
Marios Andreou (marios-b) wrote :

not sure if this is a transient issue yet - I now see a green build after the fails @ https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates

current-tripleo promoted last night to the new hash at 6439b21a91a11b464ad5b2cc147e81cd
the previous-curren-tripleo is 4dfe86fffbfd37ec10e4f2e101dc89a4

For some reason, the job is failing as it tries to pull the previous hash from the content provider [1]

        * tag: "4dfe86fffbfd37ec10e4f2e101dc89a4"

However the content provider is (correctly) building containers with the new hash [2]

      provider_dlrn_hash_branch: &id003
        master: 6439b21a91a11b464ad5b2cc147e81cd
      provider_dlrn_hash_tag_branch: &id004
        master: 6439b21a91a11b464ad5b2cc147e81cd
      registry_ip_address_branch: &id005
        master: 77.81.6.22

[1] https://83b304842de438a4ce44-164ae1091eb5a377552df92dcdcf8170.ssl.cf2.rackcdn.com/798522/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/1db6c15/logs/undercloud/home/zuul/containers-prepare-parameter.yaml

[2] https://83b304842de438a4ce44-164ae1091eb5a377552df92dcdcf8170.ssl.cf2.rackcdn.com/798522/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/1db6c15/zuul-info/inventory.yaml

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :
Revision history for this message
Marios Andreou (marios-b) wrote :

moving this to invalid - as commented in https://bugs.launchpad.net/tripleo/+bug/1934097/comments/1 this seems to be now resolved with green builds at https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates

proxy/mirror issue likely

Changed in tripleo:
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.