Comment 1 for bug 1912218

root cause

This is the task that sets the IP:

    - name: Get ceph_mon_ip addresses
      set_fact:
        tripleo_ceph_client_mon_ips: "{{ (tripleo_ceph_client_mon_ips | default([]))
                                         | union([(hostvars[item]['storage_ip']
                                         | default(hostvars[item]['ctlplane_ip']))]) }}"
      loop: "{{ groups['ceph_mon'] | list }}"

It would work with network isolation PROVIDED that you used the default name of the storage network. In my case I had customized it.

- name: Controller
  description: |
    Controller role that has all the controler services loaded and handles
    Database, Messaging and Network functions.
  CountDefault: 1
  tags:
    - primary
    - controller
    # Create external Neutron bridge for SNAT (and floating IPs when using
    # ML2/OVS without DVR)
    - external_bridge
  networks:
    External:
      subnet: external_cloud_0_subnet
    InternalApi:
      subnet: internal_api_cloud_0_subnet
    Storage:
      subnet: storage_cloud_0_subnet
    StorageMgmt:
      subnet: storage_mgmt_cloud_0_subnet
    Tenant:
      subnet: tenant_cloud_0_subnet
...

So the task does the right thing when I modify it like this:

    - name: Get ceph_mon_ip addresses
      set_fact:
        tripleo_ceph_client_mon_ips: "{{ (tripleo_ceph_client_mon_ips | default([]))
                                         | union([(hostvars[item]['storage_cloud_0_ip']
                                         | default(hostvars[item]['ctlplane_ip']))]) }}"
      loop: "{{ groups['ceph_mon'] | list }}"

Focussing on the workaround:

                                         | union([(hostvars[item]['storage_ip']
vs
                                         | union([(hostvars[item]['storage_cloud_0_ip']

We should probably not hard code either of the above.

We could back track futher to determine what they set based on the composable networks feature, for example:

 https://github.com/openstack/tripleo-heat-templates/blob/master/roles/Controller.yaml

has:

  networks:
    Storage:
      subnet: storage_subnet

While my customization used:

  networks:
    Storage:
      subnet: storage_cloud_0_subnet

So we COULD backtrack all the way up to networks and then work our way down to Storage and then grab whatever subnet is there to reason

storage_subnet --> storage_ip
storage_cloud_0_subnet --> storage_cloud_0_ip