tripleo

Error running container image prepare: Not found image

Bug #1934097 reported by Ananya Banerjee on 2021-06-30

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Invalid	Critical	Unassigned	tripleo xena-2

Bug Description

tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates fails in task Container image prepare
https://83b304842de438a4ce44-164ae1091eb5a377552df92dcdcf8170.ssl.cf2.rackcdn.com/798522/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/1db6c15/logs/undercloud/home/zuul/undercloud_install.log

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bee/798786/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/beedd6d/logs/undercloud/home/zuul/undercloud_install.log

FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Not found image: http://77.81.6.22:5001/v2/tripleomaster/openstack-heat-all/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4", "msg": "Error running container image prepare: Not found image: http://77.81.6.22:5001/v2/tripleomaster/openstack-heat-all/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4", "params": {}, "success": false}

In tripleo-container-image-prepare.log
https://83b304842de438a4ce44-164ae1091eb5a377552df92dcdcf8170.ssl.cf2.rackcdn.com/798522/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/1db6c15/logs/undercloud/var/log/tripleo-container-image-prepare.log

2021-06-30 06:10:08,647 63843 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/ HTTP/1.1" 200 2
2021-06-30 06:10:08,647 63843 DEBUG tripleo_common.image.image_uploader [ ] http://77.81.6.22:5001/v2/ status code 200
2021-06-30 06:10:08,650 63842 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/tripleomaster/openstack-mariadb/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4 HTTP/1.1" 404 122
2021-06-30 06:10:08,650 63849 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/tripleomaster/openstack-rsyslog/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4 HTTP/1.1" 404 122
2021-06-30 06:10:08,650 63845 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/tripleomaster/openstack-neutron-dhcp-agent/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4 HTTP/1.1" 404 122
2021-06-30 06:10:08,650 63842 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id acec9e2aad166c503f766cf64f11412068fbf059, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"4dfe86fffbfd37ec10e4f2e101dc89a4"}}]}

2021-06-30 06:10:08,650 63849 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 99894970a100d6e65bbb21b0bc7bdcaaa58bb857, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"4dfe86fffbfd37ec10e4f2e101dc89a4"}}]}

2021-06-30 06:10:08,650 63845 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 6391ea8c9017c1538c704d257f18b5a09904e03b, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"4dfe86fffbfd37ec10e4f2e101dc89a4"}}]}

2021-06-30 06:10:08,650 63842 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleomaster/openstack-mariadb:4dfe86fffbfd37ec10e4f2e101dc89a4] Failed uploading the target image
2021-06-30 06:10:08,651 63849 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleomaster/openstack-rsyslog:4dfe86fffbfd37ec10e4f2e101dc89a4] Failed uploading the target image
2021-06-30 06:10:08,651 63845 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleomaster/openstack-neutron-dhcp-agent:4dfe86fffbfd37ec10e4f2e101dc89a4] Failed uploading the target image
2021-06-30 06:10:08,651 63843 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/tripleomaster/openstack-ironic-inspector/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4 HTTP/1.1" 404 122
2021-06-30 06:10:08,651 63843 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id 3baad03c50488dda8243a78353292b4c02d21067, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"4dfe86fffbfd37ec10e4f2e101dc89a4"}}]}

2021-06-30 06:10:08,651 63842 INFO tripleo_common.image.image_uploader [ ] [77.81.6.22:5001/tripleomaster/openstack-qdrouterd:4dfe86fffbfd37ec10e4f2e101dc89a4] Starting upload image process
2021-06-30 06:10:08,652 63843 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleomaster/openstack-ironic-inspector:4dfe86fffbfd37ec10e4f2e101dc89a4] Failed uploading the target image
2021-06-30 06:10:08,652 63842 DEBUG urllib3.util.retry [ ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None)
2021-06-30 06:10:08,653 63842 DEBUG urllib3.connectionpool [ ] Starting new HTTP connection (1): 192.168.24.1:8787
2021-06-30 06:10:08,654 63842 DEBUG urllib3.connectionpool [ ] http://192.168.24.1:8787 "GET /v2/ HTTP/1.1" 200 2
2021-06-30 06:10:08,655 63842 DEBUG tripleo_common.image.image_uploader [ ] http://192.168.24.1:8787/v2/ status code 200
2021-06-30 06:10:08,655 63842 DEBUG urllib3.util.retry [ ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None)
2021-06-30 06:10:08,657 63842 DEBUG urllib3.connectionpool [ ] Starting new HTTP connection (1): 77.81.6.22:5001
2021-06-30 06:10:08,659 63842 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/ HTTP/1.1" 200 2
2021-06-30 06:10:08,660 63842 DEBUG tripleo_common.image.image_uploader [ ] http://77.81.6.22:5001/v2/ status code 200
2021-06-30 06:10:08,663 63842 DEBUG urllib3.connectionpool [ ] http://77.81.6.22:5001 "GET /v2/tripleomaster/openstack-qdrouterd/manifests/4dfe86fffbfd37ec10e4f2e101dc89a4 HTTP/1.1" 404 122
2021-06-30 06:10:08,664 63842 INFO tripleo_common.image.image_uploader [ ] Non-2xx: id d51221eb628974a64c300ddfa034f49bb0e9c6b6, status 404, reason Not Found, text {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"4dfe86fffbfd37ec10e4f2e101dc89a4"}}]}

2021-06-30 06:10:08,664 63842 ERROR tripleo_common.image.image_uploader [ ] [192.168.24.1:8787/tripleomaster/openstack-qdrouterd:4dfe86fffbfd37ec10e4f2e101dc89a4] Failed uploading the target image

Tags:

Ananya Banerjee (frenzyfriday) on 2021-06-30

Changed in tripleo:
status:	New → Triaged
importance:	Undecided → Critical
milestone:	none → xena-2
tags:	added: promotion-blocker

Revision history for this message

Marios Andreou (marios-b) wrote on 2021-06-30:

not sure if this is a transient issue yet - I now see a green build after the fails @ https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates

current-tripleo promoted last night to the new hash at 6439b21a91a11b464ad5b2cc147e81cd
the previous-curren-tripleo is 4dfe86fffbfd37ec10e4f2e101dc89a4

For some reason, the job is failing as it tries to pull the previous hash from the content provider [1]

* tag: "4dfe86fffbfd37ec10e4f2e101dc89a4"

However the content provider is (correctly) building containers with the new hash [2]

      provider_dlrn_hash_branch: &id003
        master: 6439b21a91a11b464ad5b2cc147e81cd
      provider_dlrn_hash_tag_branch: &id004
        master: 6439b21a91a11b464ad5b2cc147e81cd
      registry_ip_address_branch: &id005
        master: 77.81.6.22

[1] https://83b304842de438a4ce44-164ae1091eb5a377552df92dcdcf8170.ssl.cf2.rackcdn.com/798522/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/1db6c15/logs/undercloud/home/zuul/containers-prepare-parameter.yaml