periodic-tripleo-ci-centos-9-scenario001-standalone failed to download the ceph container during bootstrap.
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario001-standalone-master/105b6ec/logs/undercloud/home/zuul/ansible.log.txt.gz
2022-06-16 13:15:42,927 p=78755 u=root n=ansible | 2022-06-16 13:15:42.926608 | fa163e75-edd1-8fa8-5429-00000000006d | FATAL | Run cephadm bootstrap | standalone.localdomain | error={"changed": true, "cmd": "/usr/sbin/cephadm --image 192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph \\bootstrap --skip-firewalld --ssh-private-key /home/ceph-admin/.ssh/id_rsa --ssh-public-key /home/ceph-admin/.ssh/id_rsa.pub --ssh-user ceph-admin --allow-fqdn-hostname --output-keyring /etc/ceph/ceph.client.admin.keyring --output-config /etc/ceph/ceph.conf --fsid cc82bd4c-d566-5468-9fa7-51072fb08780 --config /home/ceph-admin/assimilate_ceph.conf \\--single-host-defaults \\--skip-monitoring-stack --skip-dashboard --log-to-file --skip-mon-network \\--mon-ip 192.168.42.1\n", "delta": "0:01:06.369211", "end": "2022-06-16 13:15:42.898100", "msg": "non-zero return code", "rc": 1, "start": "2022-06-16 13:14:36.528889", "stderr": "Verifying podman|docker is present...\nVerifying lvm2 is present...\nVerifying time synchronization is in place...\nUnit chronyd.service is enabled and running\nRepeating the final host check...\npodman (/bin/podman) version 4.1.0 is present\nsystemctl is present\nlvcreate is present\nUnit chronyd.service is enabled and running\nHost looks OK\nCluster fsid: cc82bd4c-d566-5468-9fa7-51072fb08780\nVerifying IP 192.168.42.1 port 3300 ...\nVerifying IP 192.168.42.1 port 6789 ...\nInternal network (--cluster-network) has not been provided, OSD replication will default to the public_network\nAdjusting default settings to suit single-host cluster...\nPulling container image 192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph...\nNon-zero exit code 125 from /bin/podman pull 192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph\n/bin/podman: stderr Trying to pull 192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph...\n/bin/podman: stderr time=\"2022-06-16T13:14:44-04:00\" level=warning msg=\"Failed, retrying in 1s ... (1/3). Error: initializing source docker://192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph: pinging container registry 192.168.24.1:8787: Get \\\"https://192.168.24.1:8787/v2/\\\": dial tcp 192.168.24.1:8787: connect: network is unreachable\"\n/bin/podman: stderr time=\"2022-06-16T13:14:59-04:00\" level=warning msg=\"Failed, retrying in 1s ... (2/3). Error: initializing source docker://192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph: pinging container registry 192.168.24.1:8787: Get \\\"https://192.168.24.1:8787/v2/\\\": dial tcp 192.168.24.1:8787: connect: network is unreachable\"\n/bin/podman: stderr time=\"2022-06-16T13:15:04-04:00\" level=warning msg=\"Failed, retrying in 1s ... (3/3). Error: initializing source docker://192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph: pinging container registry 192.168.24.1:8787: Get \\\"https://192.168.24.1:8787/v2/\\\": dial tcp 192.168.24.1:8787: connect: network is unreachable\"\n/bin/podman: stderr Error: initializing source docker://192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph: pinging container registry 192.168.24.1:8787: Get \"https://192.168.24.1:8787/v2/\": dial tcp 192.168.24.1:8787: connect: network is unreachable\nERROR: Failed command: /bin/podman pull 192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph", "stderr_lines": ["Verifying podman|docker is present...", "Verifying lvm2 is present...", "Verifying time synchronization is in place...", "Unit chronyd.service is enabled and running", "Repeating the final host check...", "podman (/bin/podman) version 4.1.0 is present", "systemctl is present", "lvcreate is present", "Unit chronyd.service is enabled and running", "Host looks OK", "Cluster fsid: cc82bd4c-d566-5468-9fa7-51072fb08780", "Verifying IP 192.168.42.1 port 3300 ...", "Verifying IP 192.168.42.1 port 6789 ...", "Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network", "Adjusting default settings to suit single-host cluster...", "Pulling container image 192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph...", "Non-zero exit code 125 from /bin/podman pull 192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph", "/bin/podman: stderr Trying to pull 192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph...", "/bin/podman: stderr time=\"2022-06-16T13:14:44-04:00\" level=warning msg=\"Failed, retrying in 1s ... (1/3). Error: initializing source docker://192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph: pinging container registry 192.168.24.1:8787: Get \\\"https://192.168.24.1:8787/v2/\\\": dial tcp 192.168.24.1:8787: connect: network is unreachable\"", "/bin/podman: stderr time=\"2022-06-16T13:14:59-04:00\" level=warning msg=\"Failed, retrying in 1s ... (2/3). Error: initializing source docker://192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph: pinging container registry 192.168.24.1:8787: Get \\\"https://192.168.24.1:8787/v2/\\\": dial tcp 192.168.24.1:8787: connect: network is unreachable\"", "/bin/podman: stderr time=\"2022-06-16T13:15:04-04:00\" level=warning msg=\"Failed, retrying in 1s ... (3/3). Error: initializing source docker://192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph: pinging container registry 192.168.24.1:8787: Get \\\"https://192.168.24.1:8787/v2/\\\": dial tcp 192.168.24.1:8787: connect: network is unreachable\"", "/bin/podman: stderr Error: initializing source docker://192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph: pinging container registry 192.168.24.1:8787: Get \"https://192.168.24.1:8787/v2/\": dial tcp 192.168.24.1:8787: connect: network is unreachable", "ERROR: Failed command: /bin/podman pull 192.168.24.1:8787/tripleomastercentos9/daemon:current-ceph"], "stdout": "", "stdout_lines": []}
192.168.24.1 cannot be used as a container registry to bootstrap ceph because the 24.1 address is not yet configured on the server (that happens after ceph is deployed).
When https:/ /review. opendev. org/834352 was written this was known and I observed I didn't need to worry about this because a local mirror is used in place of the undercloud's registry.
The question is why is 192.168.24.1 being used instead of the local mirror in the periodic job?
Relevant Ansible to start reading to understand what's happening:
https:/ /github. com/openstack/ tripleo- quickstart- extras/ blob/master/ roles/standalon e/tasks/ main.yml# L2-L3
https:/ /github. com/openstack/ tripleo- quickstart- extras/ blob/master/ roles/standalon e/tasks/ containers. yml