Error initializing source docker://198.72.124.50:5001/tripleomaster/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64: Error reading manifest v6.0.0-stable-6.0-pacific-centos-8-x86_64

[wes hayutin]

tripleo-ci-centos-8-scenario001-standalone failing in gate.

http://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c23/804788/3/gate/tripleo-ci-centos-8-scenario001-standalone/c23c238/logs/undercloud/var/log/ceph/cephadm.log

2021-08-17 19:58:14,106 INFO Mon IP 192.168.24.1 is in CIDR network 192.168.24.0/24
2021-08-17 19:58:14,106 INFO - internal network (--cluster-network) has not been provided, OSD replication will default to the public_network
2021-08-17 19:58:14,107 INFO Pulling container image 198.72.124.50:5001/tripleomaster/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64...
2021-08-17 19:58:14,108 DEBUG Running command: /bin/podman pull 198.72.124.50:5001/tripleomaster/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64
2021-08-17 19:58:14,304 DEBUG /bin/podman: Trying to pull 198.72.124.50:5001/tripleomaster/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64...
2021-08-17 19:58:14,332 DEBUG /bin/podman: manifest unknown: manifest unknown
2021-08-17 19:58:14,333 DEBUG /bin/podman: Error: Error initializing source docker://198.72.124.50:5001/tripleomaster/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64: Error reading manifest v6.0.0-stable-6.0-pacific-centos-8-x86_64 in 198.72.124.50:5001/tripleomaster/daemon: manifest unknown: manifest unknown
2021-08-17 19:58:14,342 INFO Non-zero exit code 125 from /bin/podman pull 198.72.124.50:5001/tripleomaster/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64
2021-08-17 19:58:14,343 INFO /bin/podman: stderr Trying to pull 198.72.124.50:5001/tripleomaster/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64...
2021-08-17 19:58:14,343 INFO /bin/podman: stderr manifest unknown: manifest unknown
2021-08-17 19:58:14,343 INFO /bin/podman: stderr Error: Error initializing source docker://198.72.124.50:5001/tripleomaster/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64: Error reading manifest v6.0.0-stable-6.0-pacific-centos-8-x86_64 in 198.72.124.50:5001/tripleomaster/daemon: manifest unknown: manifest unknown
2021-08-17 19:58:14,358 DEBUG Releasing lock 140565620556352 on /run/cephadm/4b5c8c0a-ff60-454b-a1b4-9747aa737d19.lock
2021-08-17 19:58:14,358 DEBUG Lock 140565620556352 released on /run/cephadm/4b5c8c0a-ff60-454b-a1b4-9747aa737d19.lock

[wes hayutin (weshayutin)]

https://opendev.org/openstack/tripleo-common/src/branch/master/container-images/tripleo_containers.yaml#L176

 podman pull quay.io/ceph/daemon:v6.0.4-stable-6.0-pacific-centos-8-x86_64
Trying to pull quay.io/ceph/daemon:v6.0.4-stable-6.0-pacific-centos-8-x86_64...
Getting image source signatures
Copying blob 7a0437f04f83 skipped: already exists
Copying blob 6f3a4a880cb6 done
Copying blob 096fc341997f done
Copying blob f9da06115118 done
Copying blob e1735ec7f277 done
Copying blob 2c5eea32f553 done
Copying blob 62fd6cda4cee done
Copying blob 99fa4d0909e0 done
Copying blob 29bfee0bedac done
Copying blob d7c20c204a32 done
Copying config dc26d0b02f done
Writing manifest to image destination
Storing signatures
dc26d0b02f6c4070f3bdf52d45e23a98ba8894e69f50d4ef7e9100ec0d37bdc1

works fine

[wes hayutin (weshayutin)]

Another example w/ content provider logs

SCENARIO001:
https://28743c7efa40ca8edb15-47c67d6f96b63324a4b586b546351bb6.ssl.cf2.rackcdn.com/804139/1/gate/tripleo-ci-centos-8-scenario001-standalone/ae5d28b/

CONTENT-PROVIDER server ^:
https://cad9f1be8ad76db76348-0e1449dc5644b92533134afe78c4dfb9.ssl.cf2.rackcdn.com/804139/1/gate/tripleo-ci-centos-8-content-provider/3d47065/job-output.txt

'Writing manifest to image destination', 'Storing signatures'], 'failed': False, 'item': 'quay.io/ceph/daemon:v6.0.4-stable-6.0-pacific-centos-8-x86_64', 'ansible_loop_var': 'item'})

Seems like pulling from quay atm.. is just failing

[wes hayutin (weshayutin)]

Sorry.. it's not failing to pull from quay.. failed:false.

[wes hayutin (weshayutin)]

Content provider IP :

https://cad9f1be8ad76db76348-0e1449dc5644b92533134afe78c4dfb9.ssl.cf2.rackcdn.com/804139/1/gate/tripleo-ci-centos-8-content-provider/3d47065/zuul-info/inventory.yaml

    primary:
      ansible_connection: ssh
      ansible_host: 158.69.67.34
      ansible_port: 22
      ansible_python_interpreter: /usr/bin/python3
      ansible_user: zuul

021-08-17 16:17:03.551787 | primary | TASK [container-build : Pull non-tripleo containers (ceph, alertmanager, prometheus) to the content provider registry] ***
2021-08-17 16:17:03.551835 | primary | Tuesday 17 August 2021 16:17:03 +0000 (0:00:03.223) 0:52:33.592 ********
2021-08-17 16:17:15.892807 | primary | ok: [undercloud] => (item=quay.io/ceph/daemon:v6.0.4-stable-6.0-pacific-centos-8-x86_64)
2021-08-17 16:17:17.852530 | primary | ok: [undercloud] => (item=quay.ceph.io/prometheus/prometheus:v2.7.2)
2021-08-17 16:17:19.153099 | primary | ok: [undercloud] => (item=quay.ceph.io/prometheus/alertmanager:v0.16.2)
2021-08-17 16:17:20.382108 | primary | ok: [undercloud] => (item=quay.ceph.io/prometheus/node-exporter:v0.17.0)
2021-08-17 16:17:23.364184 | primary | ok: [undercloud] => (item=quay.ceph.io/app-sre/grafana:6.7.4)

Pull request from scenario001 job:
 /bin/podman pull 158.69.67.34:5001/tripleomaster/daemon:v6.0.0-stable-6.0-pacific-centos-8-x86_64

HRM.. something is wrong w/ the container??

2021-08-17T16:17:04.461725266+00:00 stdout F 10.88.0.1 - - [17/Aug/2021:16:17:04 +0000] "HEAD /v2/tripleomaster/daemon/blobs/sha256:4c2e734e0ae7373e88f0a303e4983b4af1748a4243df1061195a004a250ee989 HTTP/1.1" 404 157 "" "Buildah/1.19.8"

Ya.. lots of 404's in this log

Here is the registry log:
https://cad9f1be8ad76db76348-0e1449dc5644b92533134afe78c4dfb9.ssl.cf2.rackcdn.com/804139/1/gate/tripleo-ci-centos-8-content-provider/3d47065/logs/undercloud/var/run/libpod/socket/aecd8020bd3cfa9842930a104defc252c3dadd39bf2712b9d086d7bfc5a5d8af/ctr.log

[wes hayutin (weshayutin)]

21-08-17T16:52:05.881329452+00:00 stdout F 158.69.69.236 - - [17/Aug/2021:16:52:05 +0000] "GET /v2/tripleomaster/daemon/manifests/v6.0.0-stable-6.0-pacific-centos-8-x86_64 HTTP/1.1" 404 131 "" "libpod/3.0.2-dev"
2021-08-17T16:52:05.881367645+00:00 stderr F time="2021-08-17T16:52:05.880804484Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown tag=v6.0.0-stable-6.0-pacific-centos-8-x86_64" err.message="manifest unknown" go.version=go1.11.2 http.request.host="158.69.67.34:5001" http.request.id=b971c168-ac26-4daa-8ad1-018e43576986 http.request.method=GET http.request.remoteaddr="158.69.69.236:53792" http.request.uri="/v2/tripleomaster/daemon/manifests/v6.0.0-stable-6.0-pacific-centos-8-x86_64" http.request.useragent="libpod/3.0.2-dev" http.response.contenttype="application/json; charset=utf-8" http.response.duration=5.116353ms http.response.status=404 http.response.written=131 vars.name="tripleomaster/daemon" vars.reference="v6.0.0-stable-6.0-pacific-centos-8-x86_64"

[wes hayutin (weshayutin)]

AH.. thank you RONELLE!

https://github.com/openstack/tripleo-common/commit/ace4a4b6c8fd88c5d1e8b2a312a1222aae15ec70

[Ronelle Landy (rlandy)]

https://github.com/openstack/tripleo-common/commit/ace4a4b6c8fd88c5d1e8b2a312a1222aae15ec70

      ceph_tag: v6.0.0-stable-6.0-pacific-centos-8-x86_64
      ceph_tag: v6.0.4-stable-6.0-pacific-centos-8-x86_64

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/804920

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ronelle Landy <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/804920

[Ronelle Landy (rlandy)]

https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/803493

[John Fulton (jfulton-org)]

I'm sorry that this work caused a bug which affected CI. We merged the following to promote our containers and am I correct to assume that 803493 without 804896 caused this? It's tricky to ensure they both land at the same time. Maybe we should make one get the value from the other so we don't hard code it in two spots?

804379 ← merged
803859 ← merged
803860 ← merged
803861 ← merged
772896 ←merged
797226 ←merged
803493 ← +w’d
Tag cephadm in cbs
804896

[wes hayutin (weshayutin)]

No worries John,
Ronelle and I are going to fix up our settings to directly come from tripleo-common this week.
It's been a week :)

[Francesco Pantano (fmount)]

I think this bug can be closed as 803493 (the quickstart-extras change) is now merged.