The nightly ceph jobs [1] have been failing to pull quay.rdoproject.org/tripleomastercentos9/daemon:latest-pacific-devel [2]
[1] https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-scenario001-ceph-nightly&job_name=tripleo-ci-centos-9-scenario004-ceph-nightly&skip=0
[2] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_175/periodic/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-9-scenario004-ceph-nightly/1754ef1/logs/undercloud/home/zuul/ansible.log
2022-07-11 03:51:48,488 p=64590 u=root n=ansible | 2022-07-11 03:51:48.487781 | bc764e20-0814-29c3-e9b9-000000000070 | FATAL | Run cephadm bootstrap | standalone.localdomain | error={"changed": true, "cmd": "/usr/sbin/cephadm --image quay.rdoproject.org/tripleomastercentos9/daemon:latest-pacific-devel \\bootstrap --skip-firewalld --ssh-private-key /home/ceph-admin/.ssh/id_rsa --ssh-public-key /home/ceph-admin/.ssh/id_rsa.pub --ssh-user ceph-admin --allow-fqdn-hostname --output-keyring /etc/ceph/ceph.client.admin.keyring --output-config /etc/ceph/ceph.conf --fsid 9dd4d899-0dcc-5f50-a744-1d88701e712a --config /home/ceph-admin/assimilate_ceph.conf \\--single-host-defaults \\--skip-monitoring-stack --skip-dashboard --log-to-file --skip-mon-network \\--mon-ip 192.168.42.1\n", "delta": "0:00:01.519565", "end": "2022-07-11 03:51:48.461089", "msg": "non-zero return code", "rc": 1, "start": "2022-07-11 03:51:46.941524", "stderr": "Verifying podman|docker is present...\nVerifying lvm2 is present...\nVerifying time synchronization is in place...\nUnit chronyd.service is enabled and running\nRepeating the final host check...\npodman (/bin/podman) version 4.1.1 is present\nsystemctl is present\nlvcreate is present\nUnit chronyd.service is enabled and running\nHost looks OK\nCluster fsid: 9dd4d899-0dcc-5f50-a744-1d88701e712a\nVerifying IP 192.168.42.1 port 3300 ...\nVerifying IP 192.168.42.1 port 6789 ...\nInternal network (--cluster-network) has not been provided, OSD replication will default to the public_network\nAdjusting default settings to suit single-host cluster...\nPulling container image quay.rdoproject.org/tripleomastercentos9/daemon:latest-pacific-devel...\nNon-zero exit code 125 from /bin/podman pull quay.rdoproject.org/tripleomastercentos9/daemon:latest-pacific-devel\n/bin/podman: stderr Trying to pull quay.rdoproject.org/tripleomastercentos9/daemon:latest-pacific-devel...\n/bin/podman: stderr Error: initializing source docker://quay.rdoproject.org/tripleomastercentos9/daemon:latest-pacific-devel: reading manifest latest-pacific-devel in quay.rdoproject.org/tripleomastercentos9/daemon: manifest unknown: manifest unknown\nERROR: Failed command: /bin/podman pull quay.rdoproject.org/tripleomastercentos9/daemon:latest-pacific-devel", "stderr_lines": ["Verifying podman|docker is present...", "Verifying lvm2 is present...", "Verifying time synchronization is in place...", "Unit chronyd.service is enabled and running", "Repeating the final host check...", "podman (/bin/podman) version 4.1.1 is present", "systemctl is present", "lvcreate is present", "Unit chronyd.service is enabled and running", "Host looks OK", "Cluster fsid: 9dd4d899-0dcc-5f50-a744-1d88701e712a", "Verifying IP 192.168.42.1 port 3300 ...", "Verifying IP 192.168.42.1 port 6789 ...", "Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network", "Adjusting default settings to suit single-host cluster...", "Pulling container image quay.rdoproject.org/tripleomastercentos9/daemon:latest-pacific-devel...", "Non-zero exit code 125 from /bin/podman pull quay.rdoproject.org/tripleomastercentos9/daemon:latest-pacific-devel", "/bin/podman: stderr Trying to pull quay.rdoproject.org/tripleomastercentos9/daemon:latest-pacific-devel...", "/bin/podman: stderr Error: initializing source docker://quay.rdoproject.org/tripleomastercentos9/daemon:latest-pacific-devel: reading manifest latest-pacific-devel in quay.rdoproject.org/tripleomastercentos9/daemon: manifest unknown: manifest unknown", "ERROR: Failed command: /bin/podman pull quay.rdoproject.org/tripleomastercentos9/daemon:latest-pacific-devel"], "stdout": "", "stdout_lines": []}
09:22 <dviroel|rover> fpantano: hey there, fyi: we need to fix usptream ceph-nightly jobs later, the new ceph deploy changed some bits, the job is not getting nightly image from the correct registry/namespace
10:11 <fultonj> dviroel|rover: i see it couldn't pull quay.rdoproject
10:13 <fpantano> is that something related to the same issue we see a couple of weeks ago?
10:13 <fpantano> the container being purged ?
10:13 <dviroel|rover> fultonj: we don't push latest-
10:14 <fpantano> fultonj: we pull it from https:/
10:14 <fpantano> dviroel|rover: and push it under tripleomasterce
10:16 <dviroel|rover> fpantano: we dont push latest-devel one, nightly job pulls it from quay.io/ceph
10:17 <fpantano> dviroel|rover: ah I see, right, we don't push that container daily, is this issue related to the recent quickstart patches?
10:17 <dviroel|rover> ack
10:18 <dviroel|rover> fpantano: yes, most probably