Comment 7 for bug 1981634

Analysis

We know /etc/ceph/ceph.conf contained an empty fsid definition [1] but why?
Both fsid and mon_host are missing from /etc/ceph/ceph.conf [2]
The missing values are present in config-download dir's ceph_client.yml [3]
As per cephadm-extra-vars-ansible.yml [4] the path to ceph_client.yml is set [5] to [3]
The deployed ceph ansible tasks and config-download ansible tasks logged to logged to /home/zuul/ansible.log [6]
The external deploy steps tasks logged to config-download/cephadm/cephadm_command.log [7]
We see in [7] that "Save tripleo_ceph_client_vars file | standalone -> localhost" from [8] ran
We see in [6] that "Render ceph config for the Ceph Clients" [9] ran with the template [10]
Rendered template behaves as if tripleo_ceph_client_fsid and tripleo_ceph_client_mon_ips are empty
Those variables should have been set by [11] but [6] shows "Load variables produced by.." was skipped
So during config-download deploy steps (not external deploy steps) tripleo_ceph_client_vars was empty.
When the job was green task "Load variables produced by the cephadm provisioning process" wasn't skipped [12]

[1] https://a9f1aef221b9e8d1cf76-922433d163de5a07cac84d974d42345f.ssl.cf1.rackcdn.com/849688/2/check/tripleo-ci-centos-9-scenario010-ovn-provider-standalone/e93e550/logs/undercloud/var/log/extra/podman/containers/nova_libvirt_init_secret/stdout.log

[2] https://a9f1aef221b9e8d1cf76-922433d163de5a07cac84d974d42345f.ssl.cf1.rackcdn.com/849688/2/check/tripleo-ci-centos-9-scenario010-ovn-provider-standalone/e93e550/logs/undercloud/etc/ceph/ceph.conf

[3] https://a9f1aef221b9e8d1cf76-922433d163de5a07cac84d974d42345f.ssl.cf1.rackcdn.com/849688/2/check/tripleo-ci-centos-9-scenario010-ovn-provider-standalone/e93e550/logs/undercloud/home/zuul/tripleo-deploy/standalone-ansible-35o06oew/cephadm/ceph_client.yml

[4] https://a9f1aef221b9e8d1cf76-922433d163de5a07cac84d974d42345f.ssl.cf1.rackcdn.com/849688/2/check/tripleo-ci-centos-9-scenario010-ovn-provider-standalone/e93e550/logs/undercloud/home/zuul/tripleo-deploy/standalone-ansible-35o06oew/cephadm/cephadm-extra-vars-ansible.yml

[5] tripleo_ceph_client_vars: "/home/zuul/tripleo-deploy/standalone-ansible-35o06oew/cephadm/ceph_client.yml"

[6] https://a9f1aef221b9e8d1cf76-922433d163de5a07cac84d974d42345f.ssl.cf1.rackcdn.com/849688/2/check/tripleo-ci-centos-9-scenario010-ovn-provider-standalone/e93e550/logs/undercloud/home/zuul/ansible.log

[7] https://a9f1aef221b9e8d1cf76-922433d163de5a07cac84d974d42345f.ssl.cf1.rackcdn.com/849688/2/check/tripleo-ci-centos-9-scenario010-ovn-provider-standalone/e93e550/logs/undercloud/home/zuul/tripleo-deploy/standalone-ansible-35o06oew/cephadm/cephadm_command.log

[8] https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_cephadm/tasks/export.yaml#L86

[9] https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_ceph_client/tasks/main.yml#L62-L73

[10] https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_ceph_client/templates/ceph_conf.j2#L14-L19

[11] https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_ceph_client/tasks/main.yml#L27-L34

[12] https://9aaab4987a89aa80a02d-56852c2a39b5b5caf6602061c30b0cc0.ssl.cf1.rackcdn.com/849580/1/check/tripleo-ci-centos-9-scenario001-standalone/a9d7773/logs/undercloud/home/zuul/ansible.log