Comment 4 for bug 1736692

I ran into this when rebuilding a controller node, where we have controller001 and controller002 running fine and controller003 is being rebuilt. All 3 controllers have ceph_mon deployed.

I reviewed the Ansible file here:
https://github.com/openstack/kolla-ansible/blob/stable/rocky/ansible/roles/ceph/tasks/bootstrap_mons.yml

where the delegate_host fact gets set.

This file:
https://github.com/openstack/kolla-ansible/blob/stable/rocky/ansible/roles/ceph/tasks/distribute_keyrings.yml

relies on a node "other than" new nodes being created to provide the existing keyrings. In our case, the script would have to rely on controller001 or controller002 to provide the keyring data. Keyring data is pulled by executing "docker exec ceph_mon fetch_ceph_keys.py" in the distribute_keyrings.yml script.

However, the logic used to determine which node is "existing" versus "new" simply looks at whether the ceph_mon_config volume was created as shown here:
https://github.com/openstack/kolla-ansible/blob/c13b8e243937ae54d2e42244cf78def8ff413ef3/ansible/roles/ceph/tasks/bootstrap_mons.yml#L14

The "ceph_mon_config_volume" flag is set if the volume exists on the node.

The issue is that if a previous deploy made it far enough to create this volume, but not far enough to create a new ceph_mon container, future deploys will fail randomly since the script can't distinguish between new and existing nodes and delegate_host can be incorrectly set to the hostname of the node we are trying to add! In our case, delegate_host was set to "controller003", the node we are trying to rebuild, whereas we needed it set to controller001 or controller002.

The command "docker exec ceph_mon fetch_ceph_keys.py" fails since there is no ceph_mon container on the new node (controller003).

The solution is to simply delete the orphaned volume on the new MON node (controller003).

See if it exists by running this on the new node (controller003):
docker volume ls

which should show the ceph_mon_config volume. If it exists, this is the issue causing the problem. So now delete the volume by running this on the new node (controller003):
docker volume rm ceph_mon_config

Afterwards, the logic to properly determine the delegate_host fact will be correct, as long as there isn't any other issue with the installation. So, delegate_host should be set to controller001 or controller002.

Note that this works fine with the --limit flag during deployment. Some comments I have seen incorrectly assume that possibly the --limit flag is at fault. It "can" be an issue, but only if you do not include a "good" ceph_mon node in the list, since the keyring must be pulled from an existing good node. So, if you are adding a new node, be sure to include a good node with the new node with your --limit argument. In our case, limiting the install to controller002 and controller003 should work fine.

Eric