Wallaby/Victoria/Train Tripleo Content-provider job failing, Unable to pull ceph related containers because quay.ceph.io is not accessible

[Sandeep Yadav]

Description:

Wallaby/Victoria/Train Tripleo Content-provider job failing in check jobs, Unable to pull ceph related container because quay.ceph.io is not accessible

https://59df0b985d3bab70a477-9b8733996223bc8ec92919ead98525d0.ssl.cf1.rackcdn.com/841464/2/check/tripleo-ci-centos-9-content-provider-wallaby/c26b31d/job-output.txt
~~~
2022-05-12 02:01:28.874676 | primary | TASK [container-build : Fail the job if missing containers exists] *************
2022-05-12 02:01:28.874764 | primary | Thursday 12 May 2022 02:01:28 +0000 (0:00:00.303) 0:41:51.978 **********
2022-05-12 02:01:28.944183 | primary | fatal: [undercloud]: FAILED! => {"changed": false, "msg": "Failed to pull following containers: ['quay.ceph.io/prometheus/prometheus:v2.7.2', 'quay.ceph.io/prometheus/alertmanager:v0.16.2', 'quay.ceph.io/prometheus/node-exporter:v0.17.0', 'quay.ceph.io/app-sre/grafana:6.7.4']"}
~~~

https://f1acd45cdb5b1f1a73d9-8a3936d67b27dfc7deb137fd1cb55ee3.ssl.cf2.rackcdn.com/841513/1/check/tripleo-ci-centos-8-content-provider-train/b44d3b1/job-output.txt
~~~
2022-05-12 02:22:16.156916 | primary | TASK [container-build : Fail the job if missing containers exists] *************
2022-05-12 02:22:16.156967 | primary | Thursday 12 May 2022 02:22:16 +0000 (0:00:00.387) 0:52:46.512 **********
2022-05-12 02:22:16.236722 | primary | fatal: [undercloud]: FAILED! => {"changed": false, "msg": "Failed to pull following containers: ['quay.ceph.io/prometheus/prometheus:v2.7.2', 'quay.ceph.io/prometheus/alertmanager:v0.16.2', 'quay.ceph.io/prometheus/node-exporter:v0.17.0', 'quay.ceph.io/app-sre/grafana:5.4.3']"}
~~~

Master branch use the different registry to pull ceph related containers:-

https://github.com/openstack/tripleo-common/blob/master/container-images/container_image_prepare_defaults.yaml#L39
~~~
ceph_alertmanager_namespace: quay.io/prometheus
~~~

vs wallaby and before:-

~~~
ceph_prometheus_namespace: quay.ceph.io/prometheus
~~~

[Sandeep Yadav (sandeepyadav93)]

Douglas have proposed a patch [1] trying to switch the registry to pull ceph related containers and waiting for ceph team reviews on the patch[2].

[1] https://review.opendev.org/c/openstack/tripleo-common/+/841512

Victoria/train branch will need a similar fix/cherry-pick with correct container tag.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-common/+/841547

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-common/+/841549

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-common/+/841550

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/841568

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/841550
Committed: https://opendev.org/openstack/tripleo-common/commit/a23c41ddf4ec70e8ea6df3e8e957ef8a5d4ee012
Submitter: "Zuul (22348)"
Branch: stable/train

commit a23c41ddf4ec70e8ea6df3e8e957ef8a5d4ee012
Author: Francesco Pantano <email address hidden>
Date: Thu May 12 09:06:41 2022 +0200

    Move the ceph monitoring stack containers to quay.io

    quay.ceph.io is no longer accessible and the Ceph Org moved to
    quay.io. This patch just updates the container image references
    to point to the new registry.

    Closes-Bug: #1973115
    Change-Id: I5673ff0480341cce4e0680587648c10c82ab1285
    (cherry picked from commit a75b38b20da3cc501d49f7e953aa5be5e57dbcf9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/841549
Committed: https://opendev.org/openstack/tripleo-common/commit/f20606909ecc6d363b9e2e7ad721221ebaad4169
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit f20606909ecc6d363b9e2e7ad721221ebaad4169
Author: Francesco Pantano <email address hidden>
Date: Thu May 12 09:06:41 2022 +0200

    Move the ceph monitoring stack containers to quay.io

    quay.ceph.io is no longer accessible and the Ceph Org moved to
    quay.io. This patch just updates the container image references
    to point to the new registry.

    We are also marking Ussuri content-provider and victoria upgrade jobs
    non-voting because Ussuri content provider is failing with similiar
    error - Unable to pull containers from quay.ceph.io and its not
    possible to fix Ussuri as its EOL and not accepting patches anymore.

    [1] https://zuul.opendev.org/t/openstack/build/3997166729c9416b8ed634db844e0302

    Closes-Bug: #1973115
    Change-Id: I5673ff0480341cce4e0680587648c10c82ab1285

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/841547
Committed: https://opendev.org/openstack/tripleo-common/commit/bed57eb65beb733cd47ce3f4e25fa075133f35e1
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit bed57eb65beb733cd47ce3f4e25fa075133f35e1
Author: Francesco Pantano <email address hidden>
Date: Thu May 12 08:08:50 2022 +0200

    Move the ceph monitoring stack containers to quay.io

    quay.ceph.io is no longer accessible and the Ceph Org moved to
    quay.io. This patch just updates the container image references
    to point to the new registry.

    Closes-Bug: #1973115
    Change-Id: Ibe83555f090bea4e21451423d88b6086917c159b

[Sandeep Yadav (sandeepyadav93)]

Based on the incident ticket[1] the issue affecting the quay.ceph.io is resolved, Also we have moved from quay.ceph.io to quay.io in [2] to pull ceph containers for stable branches(Master branch was already using quay.io).

[1] https://status.sepia.ceph.com/incidents/4053
[2] https://review.opendev.org/q/topic:ceph_monitoring_containers

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-common/+/841769

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/841769
Committed: https://opendev.org/openstack/tripleo-common/commit/d76477f1c168dae3bd6f26e40bb9629398618872
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit d76477f1c168dae3bd6f26e40bb9629398618872
Author: Sandeep Yadav <email address hidden>
Date: Fri May 13 21:16:44 2022 +0530

    Make U content-provider & V upgrades voting again

    In patch[1], we marked Ussuri content-provider and Victoria
    upgrade jobs non-voting because Ussuri content-provider was
    down because of quay.ceph.io outage.

    Removing voting:false as Ussuri content-provider is back
    after quay.ceph.io restoration.

    [1] https://review.opendev.org/c/openstack/tripleo-common/+/841549

    Related-Bug: #1973115
    Change-Id: I9bde591cd2cd3f8133189ec238106a6e31af4d27

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common victoria-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common train-eol release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ghanshyam <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/841568
Reason: TrieplO project is retiring now, for details, please see https://review.opendev.org/c/openstack/governance/+/905145 or reach out to OpenStack TC.