glance and cinder are talking to one Ceph MON

[Nobuto Murata]

I’m following multi-node scenario as per the tutorial:
https://microstack.run/docs/multi-node
But I assume this issue happens with the MAAS scenario too.

After completing the resize step to have 3 controller units for HA, looks like ceph.conf in glance and cinder units has one Ceph MON IP address from the initial bootstrapped host, which mean those services cannot talk to Ceph once the first host dies.

The following output confirms only 10.0.123.11 is written into the config files.

$ for i in {0..2}; do juju ssh -m openstack --container glance-api glance/$i 'echo ===== $HOSTNAME =====; grep ceph /etc/glance/glance-api.conf'; done
===== glance-0 =====
enabled_backends = filestore:file, ceph:rbd
default_backend = ceph
[ceph]
rbd_store_ceph_conf = /etc/ceph/ceph.conf
===== glance-1 =====
enabled_backends = filestore:file, ceph:rbd
default_backend = ceph
[ceph]
rbd_store_ceph_conf = /etc/ceph/ceph.conf
===== glance-2 =====
enabled_backends = filestore:file, ceph:rbd
default_backend = ceph
[ceph]
rbd_store_ceph_conf = /etc/ceph/ceph.conf

$ for i in {0..2}; do juju ssh -m openstack --container glance-api glance/$i 'echo ===== $HOSTNAME =====; cat /etc/ceph/ceph.conf'; done
===== glance-0 =====
###############################################################################
# [ WARNING ]
# ceph configuration file maintained in aso
# local changes may be overwritten.
###############################################################################
[global]
auth_supported = cephx
mon host = 10.0.123.11
keyring = /etc/ceph/$cluster.$name.keyring
log to syslog = false
err to syslog = false
clog to syslog = false

[client]
===== glance-1 =====
###############################################################################
# [ WARNING ]
# ceph configuration file maintained in aso
# local changes may be overwritten.
###############################################################################
[global]
auth_supported = cephx
mon host = 10.0.123.11
keyring = /etc/ceph/$cluster.$name.keyring
log to syslog = false
err to syslog = false
clog to syslog = false

[client]
===== glance-2 =====
###############################################################################
# [ WARNING ]
# ceph configuration file maintained in aso
# local changes may be overwritten.
###############################################################################
[global]
auth_supported = cephx
mon host = 10.0.123.11
keyring = /etc/ceph/$cluster.$name.keyring
log to syslog = false
err to syslog = false
clog to syslog = false

[client]

$ snap list openstack
Name Version Rev Tracking Publisher Notes
openstack 2024.1 506 2024.1/edge canonical✓ -

[network env]

10.0.123.0/24 - single network deployment

10.0.123.1 - gateway, DNS server

10.0.123.11 - sunbeam-1: controller, compute, storage <- the bootstrap node
10.0.123.12 - sunbeam-2: controller, compute, storage
10.0.123.13 - sunbeam-3: controller, compute, storage

10.0.123.51-80 - floating IP range

10.0.123.81-90 - MetalLB range
+ 10.0.123.81 - rabbitmq
+ 10.0.123.82 - traefik
+ 10.0.123.83 - traefik-public
+ 10.0.123.84 - ovn-relay

[Nobuto Murata (nobuto)]

ceph-public-address in the relation data only has one IP address.

$ sudo microceph status
MicroCeph deployment summary:
- sunbeam-1 (10.0.123.11)
  Services: mds, mgr, mon, osd
  Disks: 2
- sunbeam-2 (10.0.123.12)
  Services: mds, mgr, mon, osd
  Disks: 2
- sunbeam-3 (10.0.123.13)
  Services: mds, mgr, mon, osd
  Disks: 2

$ juju show-unit -m openstack glance/2

...

  - relation-id: 50
    endpoint: ceph
    cross-model: true
    related-endpoint: ceph
    application-data: {}
    related-units:
      microceph/0:
        in-scope: true
        data:
          auth: cephx
          broker-rsp-glance-0: '{"exit-code": 0, "request-id": "cc6dca75e3585dfba9af7293ce83810bffaa1a97"}'
          broker-rsp-glance-1: '{"exit-code": 0, "request-id": "cc6dca75e3585dfba9af7293ce83810bffaa1a97"}'
          broker-rsp-glance-2: '{"exit-code": 0, "request-id": "cc6dca75e3585dfba9af7293ce83810bffaa1a97"}'
          ceph-public-address: 10.0.123.11
          egress-subnets: 10.0.123.11/32
          ingress-address: 10.0.123.11
          key: AQDhGEtme4xqIBAAUoDECPLPeod728vTLZNvAQ==
          private-address: 10.0.123.11
      microceph/1:
        in-scope: true
        data:
          egress-subnets: 10.0.123.12/32
          ingress-address: 10.0.123.12
          private-address: 10.0.123.12
      microceph/2:
        in-scope: true
        data:
          egress-subnets: 10.0.123.13/32
          ingress-address: 10.0.123.13
          private-address: 10.0.123.13

          broker-rsp-glance-0: '{"exit-code": 0, "request-id": "cc6dca75e3585dfba9af7293ce83810bffaa1a97"}'
          broker-rsp-glance-1: '{"exit-code": 0, "request-id": "cc6dca75e3585dfba9af7293ce83810bffaa1a97"}'
          broker-rsp-glance-2: '{"exit-code": 0, "request-id": "cc6dca75e3585dfba9af7293ce83810bffaa1a97"}'
          ceph-public-address: 10.0.123.11
          egress-subnets: 10.0.123.11/32
          ingress-address: 10.0.123.11
          key: AQDhGEtme4xqIBAAUoDECPLPeod728vTLZNvAQ==
          private-address: 10.0.123.11
      microceph/1:
        in-scope: true
        data:
          egress-subnets: 10.0.123.12/32
          ingress-address: 10.0.123.12
          private-address: 10.0.123.12
      microceph/2:
        in-scope: true
        data:
          egress-subnets: 10.0.123.13/32
          ingress-address: 10.0.123.13
          private-address: 10.0.123.13

[Nobuto Murata (nobuto)]

https://github.com/canonical/charm-microceph/issues/77