ceph-mon remove-unit doesn't remove monitor from cluster and from related charms

[macchese]

I needed to move a ceph-mon from one node to another one so I executed:
  1) juju add-unit ceph-mon --to lxd:n, where n is the "another one" node
  2) juju remove-unit ceph-mon/x where x is the mon I would like to remove.

After that juju added the new monitor and removed the old one but accourding to ceph the monitor I removed is "out of qorum"

 root@juju-206dbb-4-lxd-1:~# ceph -s
  cluster:
    id: 7dfd9e3a-a5b6-11ee-9798-619012c1bb3a
    health: HEALTH_WARN
            1/4 mons down, quorum juju-206dbb-4-lxd-1,juju-206dbb-5-lxd-1,juju-206dbb-10-lxd-1

  services:
    mon: 4 daemons, quorum juju-206dbb-4-lxd-1,juju-206dbb-5-lxd-1,juju-206dbb-10-lxd-1 (age 58m), out of quorum: juju-206dbb-3-lxd-1

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy

juju --version
3.4.0-genericlinux-amd64

maas
3.3.5-13222-g.78dd996c0

juju status --relations
Model Controller Cloud/Region Version SLA Timestamp
openstack maas-controller1 maas-cloud/default 3.2.4 unsupported 18:50:52Z

App Version Status Scale Charm Channel Rev Exposed Message
ceph-dashboard active 3 ceph-dashboard quincy/stable 50 no Unit is ready
ceph-fs 17.2.6 active 3 ceph-fs reef/stable 61 no Unit is ready
ceph-mon 17.2.6 active 3 ceph-mon reef/stable 192 no Unit is ready and clustered
ceph-osd 18.2.0 active 4 ceph-osd reef/stable 577 no Unit is ready (10 OSD)
ceph-radosgw 17.2.6 active 1 ceph-radosgw reef/stable 566 no Unit is ready
cinder 23.0.0 active 1 cinder 2023.2/stable 663 no Unit is ready
cinder-ceph 23.0.0 waiting 1 cinder-ceph 2023.2/stable 528 no Ceph broker request incomplete
cinder-mysql-router 8.0.36 active 1 mysql-router 8.0/stable 111 no Unit is ready
dashboard-mysql-router 8.0.36 active 1 mysql-router 8.0/stable 111 no Unit is ready
glance 27.0.0 waiting 1 glance 2023.2/stable 595 no Ceph broker request incomplete
glance-mysql-router 8.0.36 active 1 mysql-router 8.0/stable 111 no Unit is ready
keystone 24.0.0 active 1 keystone 2023.2/stable 667 no Application Ready
keystone-mysql-router 8.0.36 active 1 mysql-router 8.0/stable 111 no Unit is ready
mysql-innodb-cluster 8.0.36 active 3 mysql-innodb-cluster 8.0/stable 107 no Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
neutron-api 23.0.0 active 1 neutron-api 2023.2/stable 568 no Unit is ready
neutron-api-plugin-ovn 23.0.0 active 1 neutron-api-plugin-ovn 2023.2/stable 113 no Unit is ready
neutron-mysql-router 8.0.36 active 1 mysql-router 8.0/stable 111 no Unit is ready
nova-cloud-controller 28.0.0 active 1 nova-cloud-controller 2023.2/stable 717 no Unit is ready
nova-compute 28.0.1 active 3 nova-compute 2023.2/stable 703 no Unit is ready
nova-mysql-router 8.0.36 active 1 mysql-router 8.0/stable 111 no Unit is ready
ntp 4.2 active 3 ntp stable 50 no chrony: Ready
openstack-dashboard 23.3.0 active 1 openstack-dashboard 2023.2/stable 626 no Unit is ready
openstack-loadbalancer active 3 openstack-loadbalancer jammy/stable 10 no Unit is ready
openstack-loadbalancer-hacluster 2.1.2 active 3 hacluster 2.4/stable 131 no Unit is ready and clustered
ovn-central 22.09.1 active 3 ovn-central 23.09/stable 158 no Unit is ready
ovn-chassis 23.09.0 active 3 ovn-chassis 23.09/stable 178 no Unit is ready
placement 10.0.0 active 1 placement 2023.2/stable 100 no Unit is ready
placement-mysql-router 8.0.36 active 1 mysql-router 8.0/stable 111 no Unit is ready
rabbitmq-server 3.9.13 active 1 rabbitmq-server 3.9/stable 182 no Unit is ready
vault 1.8.8 active 1 vault 1.8/stable 183 no Unit is ready (active: true, mlock: disabled)
vault-mysql-router 8.0.36 active 1 mysql-router 8.0/stable 111 no Unit is ready

Unit Workload Agent Machine Public address Ports Message
ceph-fs/2 active idle 4/lxd/0 192.168.70.15 Unit is ready
ceph-fs/4* active idle 5/lxd/0 192.168.70.31 Unit is ready
ceph-fs/6 active idle 10/lxd/0 192.168.70.27 Unit is ready
ceph-mon/2 active idle 4/lxd/1 192.168.70.14 Unit is ready and clustered
  ceph-dashboard/5 active idle 192.168.70.14 Unit is ready
ceph-mon/4 active idle 5/lxd/1 192.168.70.33 Unit is ready and clustered
  ceph-dashboard/6* active idle 192.168.70.33 Unit is ready
ceph-mon/5* active idle 10/lxd/1 192.168.70.28 Unit is ready and clustered
  ceph-dashboard/7 active idle 192.168.70.28 Unit is ready
ceph-osd/1 active idle 3 192.168.6.113 Unit is ready (10 OSD)
ceph-osd/2* active idle 4 192.168.6.112 Unit is ready (10 OSD)
ceph-osd/4 active idle 5 192.168.6.111 Unit is ready (11 OSD)
ceph-osd/6 active idle 10 192.168.6.114 Unit is ready (10 OSD)
ceph-radosgw/2* active idle 5/lxd/2 192.168.70.32 80/tcp Unit is ready
cinder/3* active idle 8/lxd/0 192.168.70.241 8776/tcp Unit is ready
  cinder-ceph/6* waiting idle 192.168.70.241 Ceph broker request incomplete
  cinder-mysql-router/3* active idle 192.168.70.241 Unit is ready
glance/1* waiting idle 7/lxd/0 192.168.70.251 9292/tcp Ceph broker request incomplete
  glance-mysql-router/1* active idle 192.168.70.251 Unit is ready
keystone/1* active idle 7/lxd/1 192.168.70.250 5000/tcp Unit is ready
  keystone-mysql-router/1* active idle 192.168.70.250 Unit is ready
mysql-innodb-cluster/4* active idle 8/lxd/1 192.168.70.245 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/5 active idle 7/lxd/3 192.168.70.252 Unit is ready: Mode: R/W, Cluster is ONLINE and can tolerate up to ONE failure.
mysql-innodb-cluster/8 active idle 3/lxd/4 192.168.70.29 Unit is ready: Mode: R/O, Cluster is ONLINE and can tolerate up to ONE failure.
neutron-api/2* active idle 8/lxd/2 192.168.70.246 9696/tcp Unit is ready
  neutron-api-plugin-ovn/2* active idle 192.168.70.246 Unit is ready
  neutron-mysql-router/2* active idle 192.168.70.246 Unit is ready
nova-cloud-controller/3* active idle 7/lxd/4 192.168.70.253 8774-8775/tcp Unit is ready
  nova-mysql-router/3* active idle 192.168.70.253 Unit is ready
nova-compute/6* active idle 7 192.168.6.101 Unit is ready
  ntp/7* active idle 192.168.6.101 123/udp chrony: Ready
  ovn-chassis/10* active idle 192.168.6.101 Unit is ready
nova-compute/7 active idle 8 192.168.6.110 Unit is ready
  ntp/6 active idle 192.168.6.110 123/udp chrony: Ready
  ovn-chassis/9 active idle 192.168.6.110 Unit is ready
nova-compute/8 active idle 3 192.168.6.113 Unit is ready
  ntp/8 active idle 192.168.6.113 123/udp chrony: Ready
  ovn-chassis/11 active idle 192.168.6.113 Unit is ready
openstack-dashboard/1* active idle 8/lxd/3 192.168.70.247 80,443/tcp Unit is ready
  dashboard-mysql-router/1* active idle 192.168.70.247 Unit is ready
openstack-loadbalancer/5* active idle 7/lxd/8 192.168.70.19 Unit is ready
  openstack-loadbalancer-hacluster/5* active idle 192.168.70.19 Unit is ready and clustered
openstack-loadbalancer/7 active idle 8/lxd/9 192.168.70.26 Unit is ready
  openstack-loadbalancer-hacluster/9 active idle 192.168.70.26 Unit is ready and clustered
openstack-loadbalancer/8 active idle 3/lxd/5 192.168.70.30 Unit is ready
  openstack-loadbalancer-hacluster/10 active idle 192.168.70.30 Unit is ready and clustered
ovn-central/6 active idle 7/lxd/5 192.168.70.17 6641-6642/tcp Unit is ready
ovn-central/7* active idle 8/lxd/4 192.168.70.248 6641-6642/tcp Unit is ready (leader: ovnnb_db)
ovn-central/8 active idle 8/lxd/5 192.168.70.243 6641-6642/tcp Unit is ready (leader: ovnsb_db northd: active)
placement/1* active idle 8/lxd/6 192.168.70.244 8778/tcp Unit is ready
  placement-mysql-router/1* active idle 192.168.70.244 Unit is ready
rabbitmq-server/1* active idle 8/lxd/7 192.168.70.242 5672,15672/tcp Unit is ready
vault/1* active idle 7/lxd/6 192.168.70.16 8200/tcp Unit is ready (active: true, mlock: disabled)
  vault-mysql-router/1* active idle 192.168.70.16 Unit is ready

Machine State Address Inst id Base AZ Message
3 started 192.168.6.113 op4 ubuntu@22.04 default Deployed
3/lxd/4 started 192.168.70.29 juju-206dbb-3-lxd-4 ubuntu@22.04 default Container started
3/lxd/5 started 192.168.70.30 juju-206dbb-3-lxd-5 ubuntu@22.04 default Container started
4 started 192.168.6.112 op3 ubuntu@22.04 default Deployed
4/lxd/0 started 192.168.70.15 juju-206dbb-4-lxd-0 ubuntu@22.04 default Container started
4/lxd/1 started 192.168.70.14 juju-206dbb-4-lxd-1 ubuntu@22.04 default Container started
5 started 192.168.6.111 op2 ubuntu@22.04 default Deployed
5/lxd/0 started 192.168.70.31 juju-206dbb-5-lxd-0 ubuntu@22.04 default Container started
5/lxd/1 started 192.168.70.33 juju-206dbb-5-lxd-1 ubuntu@22.04 default Container started
5/lxd/2 started 192.168.70.32 juju-206dbb-5-lxd-2 ubuntu@22.04 default Container started
7 started 192.168.6.101 xen01 ubuntu@22.04 xensrv Deployed
7/lxd/0 started 192.168.70.251 juju-206dbb-7-lxd-0 ubuntu@22.04 xensrv Container started
7/lxd/1 started 192.168.70.250 juju-206dbb-7-lxd-1 ubuntu@22.04 xensrv Container started
7/lxd/3 started 192.168.70.252 juju-206dbb-7-lxd-3 ubuntu@22.04 xensrv Container started
7/lxd/4 started 192.168.70.253 juju-206dbb-7-lxd-4 ubuntu@22.04 xensrv Container started
7/lxd/5 started 192.168.70.17 juju-206dbb-7-lxd-5 ubuntu@22.04 xensrv Container started
7/lxd/6 started 192.168.70.16 juju-206dbb-7-lxd-6 ubuntu@22.04 xensrv Container started
7/lxd/8 started 192.168.70.19 juju-206dbb-7-lxd-8 ubuntu@22.04 xensrv Container started
8 started 192.168.6.110 op1 ubuntu@22.04 default Deployed
8/lxd/0 started 192.168.70.241 juju-206dbb-8-lxd-0 ubuntu@22.04 default Container started
8/lxd/1 started 192.168.70.245 juju-206dbb-8-lxd-1 ubuntu@22.04 default Container started
8/lxd/2 started 192.168.70.246 juju-206dbb-8-lxd-2 ubuntu@22.04 default Container started
8/lxd/3 started 192.168.70.247 juju-206dbb-8-lxd-3 ubuntu@22.04 default Container started
8/lxd/4 started 192.168.70.248 juju-206dbb-8-lxd-4 ubuntu@22.04 default Container started
8/lxd/5 started 192.168.70.243 juju-206dbb-8-lxd-5 ubuntu@22.04 default Container started
8/lxd/6 started 192.168.70.244 juju-206dbb-8-lxd-6 ubuntu@22.04 default Container started
8/lxd/7 started 192.168.70.242 juju-206dbb-8-lxd-7 ubuntu@22.04 default Container started
8/lxd/9 started 192.168.70.26 juju-206dbb-8-lxd-9 ubuntu@22.04 default Container started
10 started 192.168.6.114 op5 ubuntu@22.04 default Deployed
10/lxd/0 started 192.168.70.27 juju-206dbb-10-lxd-0 ubuntu@22.04 default Container started
10/lxd/1 started 192.168.70.28 juju-206dbb-10-lxd-1 ubuntu@22.04 default Container started

Integration provider Requirer Interface Type Message
ceph-mon:client cinder-ceph:ceph ceph-client regular
ceph-mon:client glance:ceph ceph-client regular
ceph-mon:client nova-compute:ceph ceph-client regular
ceph-mon:dashboard ceph-dashboard:dashboard ceph-dashboard subordinate
ceph-mon:mds ceph-fs:ceph-mds ceph-mds regular
ceph-mon:mon ceph-mon:mon ceph peer
ceph-mon:osd ceph-osd:mon ceph-osd regular
ceph-mon:radosgw ceph-radosgw:mon ceph-radosgw regular
ceph-radosgw:cluster ceph-radosgw:cluster swift-ha peer
cinder-ceph:ceph-access nova-compute:ceph-access cinder-ceph-key regular
cinder-ceph:storage-backend cinder:storage-backend cinder-backend subordinate
cinder-mysql-router:shared-db cinder:shared-db mysql-shared subordinate
cinder:cinder-volume-service nova-cloud-controller:cinder-volume-service cinder regular
cinder:cluster cinder:cluster cinder-ha peer
dashboard-mysql-router:shared-db openstack-dashboard:shared-db mysql-shared subordinate
glance-mysql-router:shared-db glance:shared-db mysql-shared subordinate
glance:cluster glance:cluster glance-ha peer
glance:image-service cinder:image-service glance regular
glance:image-service nova-cloud-controller:image-service glance regular
glance:image-service nova-compute:image-service glance regular
keystone-mysql-router:shared-db keystone:shared-db mysql-shared subordinate
keystone:cluster keystone:cluster keystone-ha peer
keystone:identity-service ceph-radosgw:identity-service keystone regular
keystone:identity-service cinder:identity-service keystone regular
keystone:identity-service glance:identity-service keystone regular
keystone:identity-service neutron-api:identity-service keystone regular
keystone:identity-service nova-cloud-controller:identity-service keystone regular
keystone:identity-service openstack-dashboard:identity-service keystone regular
keystone:identity-service placement:identity-service keystone regular
mysql-innodb-cluster:cluster mysql-innodb-cluster:cluster mysql-innodb-cluster peer
mysql-innodb-cluster:coordinator mysql-innodb-cluster:coordinator coordinator peer
mysql-innodb-cluster:db-router cinder-mysql-router:db-router mysql-router regular
mysql-innodb-cluster:db-router dashboard-mysql-router:db-router mysql-router regular
mysql-innodb-cluster:db-router glance-mysql-router:db-router mysql-router regular
mysql-innodb-cluster:db-router keystone-mysql-router:db-router mysql-router regular
mysql-innodb-cluster:db-router neutron-mysql-router:db-router mysql-router regular
mysql-innodb-cluster:db-router nova-mysql-router:db-router mysql-router regular
mysql-innodb-cluster:db-router placement-mysql-router:db-router mysql-router regular
mysql-innodb-cluster:db-router vault-mysql-router:db-router mysql-router regular
neutron-api-plugin-ovn:neutron-plugin neutron-api:neutron-plugin-api-subordinate neutron-plugin-api-subordinate subordinate
neutron-api:cluster neutron-api:cluster neutron-api-ha peer
neutron-api:neutron-api nova-cloud-controller:neutron-api neutron-api regular
neutron-mysql-router:shared-db neutron-api:shared-db mysql-shared subordinate
nova-cloud-controller:cluster nova-cloud-controller:cluster nova-ha peer
nova-compute:cloud-compute nova-cloud-controller:cloud-compute nova-compute regular
nova-compute:compute-peer nova-compute:compute-peer nova peer
nova-compute:juju-info ntp:juju-info juju-info subordinate
nova-mysql-router:shared-db nova-cloud-controller:shared-db mysql-shared subordinate
ntp:ntp-peers ntp:ntp-peers ntp peer
openstack-dashboard:cluster openstack-dashboard:cluster openstack-dashboard-ha peer
openstack-loadbalancer-hacluster:ha openstack-loadbalancer:ha hacluster subordinate
openstack-loadbalancer-hacluster:hanode openstack-loadbalancer-hacluster:hanode hacluster peer
openstack-loadbalancer:loadbalancer ceph-dashboard:loadbalancer openstack-loadbalancer regular
ovn-central:coordinator ovn-central:coordinator coordinator peer
ovn-central:ovsdb ovn-chassis:ovsdb ovsdb regular
ovn-central:ovsdb-cms neutron-api-plugin-ovn:ovsdb-cms ovsdb-cms regular
ovn-central:ovsdb-peer ovn-central:ovsdb-peer ovsdb-cluster peer
ovn-chassis:nova-compute nova-compute:neutron-plugin neutron-plugin subordinate
placement-mysql-router:shared-db placement:shared-db mysql-shared subordinate
placement:cluster placement:cluster openstack-ha peer
placement:placement nova-cloud-controller:placement placement regular
rabbitmq-server:amqp cinder:amqp rabbitmq regular
rabbitmq-server:amqp glance:amqp rabbitmq regular
rabbitmq-server:amqp neutron-api:amqp rabbitmq regular
rabbitmq-server:amqp nova-cloud-controller:amqp rabbitmq regular
rabbitmq-server:amqp nova-compute:amqp rabbitmq regular
rabbitmq-server:cluster rabbitmq-server:cluster rabbitmq-ha peer
vault-mysql-router:shared-db vault:shared-db mysql-shared subordinate
vault:certificates ceph-dashboard:certificates tls-certificates regular
vault:certificates ceph-radosgw:certificates tls-certificates regular
vault:certificates cinder:certificates tls-certificates regular
vault:certificates glance:certificates tls-certificates regular
vault:certificates keystone:certificates tls-certificates regular
vault:certificates mysql-innodb-cluster:certificates tls-certificates regular
vault:certificates neutron-api-plugin-ovn:certificates tls-certificates regular
vault:certificates neutron-api:certificates tls-certificates regular
vault:certificates nova-cloud-controller:certificates tls-certificates regular
vault:certificates openstack-dashboard:certificates tls-certificates regular
vault:certificates ovn-central:certificates tls-certificates regular
vault:certificates ovn-chassis:certificates tls-certificates regular
vault:certificates placement:certificates tls-certificates regular
vault:cluster vault:cluster vault-ha peer

juju export-bundle
series: jammy
applications:
  ceph-dashboard:
    charm: ceph-dashboard
    channel: quincy/stable
    revision: 50
    options:
      public-hostname: 192.168.70.207
    bindings:
      "": op-public-space
      alertmanager-service: op-public-space
      certificates: op-public-space
      dashboard: op-public-space
      grafana-dashboard: op-public-space
      iscsi-dashboard: op-public-space
      loadbalancer: op-public-space
      prometheus: op-public-space
      public: op-public-space
      radosgw-dashboard: op-public-space
  ceph-fs:
    charm: ceph-fs
    channel: reef/stable
    revision: 61
    num_units: 3
    to:
    - lxd:4
    - lxd:5
    - lxd:10
    annotations:
      gui-x: "790"
      gui-y: "1540"
    constraints: tags=cloud,ceph
    bindings:
      "": op-public-space
      ceph-mds: op-public-space
      public: op-public-space
  ceph-mon:
    charm: ceph-mon
    channel: reef/stable
    revision: 192
    resources:
      alert-rules: 1
    num_units: 3
    to:
    - lxd:4
    - lxd:5
    - lxd:10
    options:
      balancer-mode: upmap
      expected-osd-count: 9
      monitor-count: 3
    annotations:
      gui-x: "790"
      gui-y: "1540"
    constraints: tags=cloud,ceph
    bindings:
      "": op-public-space
      admin: op-public-space
      bootstrap-source: op-public-space
      client: op-public-space
      cluster: op-cluster-space
      dashboard: op-public-space
      mds: op-public-space
      metrics-endpoint: op-public-space
      mon: op-public-space
      nrpe-external-master: op-public-space
      osd: op-public-space
      prometheus: op-public-space
      public: op-public-space
      radosgw: op-public-space
      rbd-mirror: op-public-space
  ceph-osd:
    charm: ceph-osd
    channel: reef/stable
    revision: 577
    num_units: 4
    to:
    - "3"
    - "4"
    - "5"
    - "10"
    options:
      config-flags: '{ "global": { "osd crush initial weight": 0 } }'
      crush-initial-weight: 0
      osd-devices: /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg
        /dev/sdh /dev/sdi
    annotations:
      gui-x: "1065"
      gui-y: "1540"
    constraints: tags=cloud,ceph
    storage:
      bluestore-db: loop,1024M
      bluestore-wal: loop,1024M
      cache-devices: loop,10240M
      osd-devices: loop,1024M
      osd-journals: loop,1024M
    bindings:
      "": op-public-space
      cluster: op-cluster-space
      mon: op-public-space
      nrpe-external-master: op-public-space
      public: op-public-space
      secrets-storage: op-public-space
  ceph-radosgw:
    charm: ceph-radosgw
    channel: reef/stable
    revision: 566
    num_units: 1
    to:
    - lxd:5
    annotations:
      gui-x: "850"
      gui-y: "900"
    constraints: tags=cloud,ceph
    bindings:
      "": op-public-space
      admin: op-public-space
      certificates: op-public-space
      cluster: op-cluster-space
      gateway: op-public-space
      ha: op-public-space
      identity-service: op-public-space
      internal: op-public-space
      master: op-public-space
      mon: op-public-space
      nrpe-external-master: op-public-space
      object-store: op-public-space
      primary: op-public-space
      public: op-public-space
      radosgw-user: op-public-space
      secondary: op-public-space
      slave: op-public-space
  cinder:
    charm: cinder
    channel: 2023.2/stable
    revision: 663
    resources:
      policyd-override: 0
    num_units: 1
    to:
    - lxd:8
    options:
      block-device: None
      glance-api-version: 2
    annotations:
      gui-x: "980"
      gui-y: "1270"
    constraints: arch=amd64 tags=cloud,ccompute
    storage:
      block-devices: loop,10240M
    bindings:
      "": op-public-space
      admin: op-public-space
      amqp: op-public-space
      backup-backend: op-public-space
      ceph: op-public-space
      certificates: op-public-space
      cinder-volume-service: op-public-space
      cluster: op-public-space
      ha: op-public-space
      identity-credentials: op-public-space
      identity-service: op-public-space
      image-service: op-public-space
      internal: op-public-space
      nrpe-external-master: op-public-space
      public: op-public-space
      shared-db: op-public-space
      storage-backend: op-public-space
  cinder-ceph:
    charm: cinder-ceph
    channel: 2023.2/stable
    revision: 528
    options:
      rbd-pool-name: SAS
    annotations:
      gui-x: "1120"
      gui-y: "1400"
    bindings:
      "": op-public-space
      ceph: op-public-space
      ceph-access: op-public-space
      ceph-replication-device: op-public-space
      juju-info: op-public-space
      storage-backend: op-public-space
  cinder-mysql-router:
    charm: mysql-router
    channel: 8.0/stable
    revision: 111
    annotations:
      gui-x: "900"
      gui-y: "1400"
    bindings:
      "": op-public-space
      certificates: op-public-space
      db-router: op-public-space
      juju-info: op-public-space
      shared-db: op-public-space
  dashboard-mysql-router:
    charm: mysql-router
    channel: 8.0/stable
    revision: 111
    annotations:
      gui-x: "510"
      gui-y: "1030"
    bindings:
      "": op-public-space
      certificates: op-public-space
      db-router: op-public-space
      juju-info: op-public-space
      shared-db: op-public-space
  glance:
    charm: glance
    channel: 2023.2/stable
    revision: 595
    resources:
      policyd-override: 0
    num_units: 1
    to:
    - lxd:7
    annotations:
      gui-x: "-230"
      gui-y: "1270"
    constraints: arch=amd64 tags=cloud,ccompute
    bindings:
      "": op-public-space
      admin: op-public-space
      amqp: op-public-space
      ceph: op-public-space
      certificates: op-public-space
      cinder-volume-service: op-public-space
      cluster: op-public-space
      ha: op-public-space
      identity-service: op-public-space
      image-service: op-public-space
      internal: op-public-space
      nrpe-external-master: op-public-space
      object-store: op-public-space
      public: op-public-space
      shared-db: op-public-space
      storage-backend: op-public-space
  glance-mysql-router:
    charm: mysql-router
    channel: 8.0/stable
    revision: 111
    annotations:
      gui-x: "-290"
      gui-y: "1400"
    bindings:
      "": op-public-space
      certificates: op-public-space
      db-router: op-public-space
      juju-info: op-public-space
      shared-db: op-public-space
  keystone:
    charm: keystone
    channel: 2023.2/stable
    revision: 667
    resources:
      policyd-override: 0
    num_units: 1
    to:
    - lxd:7
    options:
      region: RegionOne
    annotations:
      gui-x: "300"
      gui-y: "1270"
    constraints: arch=amd64 tags=cloud,ccompute
    bindings:
      "": op-public-space
      admin: op-public-space
      certificates: op-public-space
      cluster: op-public-space
      domain-backend: op-public-space
      ha: op-public-space
      identity-admin: op-public-space
      identity-credentials: op-public-space
      identity-notifications: op-public-space
      identity-service: op-public-space
      internal: op-public-space
      keystone-fid-service-provider: op-public-space
      keystone-middleware: op-public-space
      nrpe-external-master: op-public-space
      public: op-public-space
      shared-db: op-public-space
      websso-trusted-dashboard: op-public-space
  keystone-mysql-router:
    charm: mysql-router
    channel: 8.0/stable
    revision: 111
    annotations:
      gui-x: "230"
      gui-y: "1400"
    bindings:
      "": op-public-space
      certificates: op-public-space
      db-router: op-public-space
      juju-info: op-public-space
      shared-db: op-public-space
  mysql-innodb-cluster:
    charm: mysql-innodb-cluster
    channel: 8.0/stable
    revision: 107
    resources:
      mysql-shell: 0
    num_units: 3
    to:
    - lxd:8
    - lxd:7
    - lxd:3
    annotations:
      gui-x: "535"
      gui-y: "1550"
    constraints: arch=amd64 tags=cloud,ccompute
    bindings:
      "": op-public-space
      certificates: op-public-space
      cluster: op-public-space
      coordinator: op-public-space
      db-monitor: op-public-space
      db-router: op-public-space
      shared-db: op-public-space
  neutron-api:
    charm: neutron-api
    channel: 2023.2/stable
    revision: 568
    resources:
      policyd-override: 0
    num_units: 1
    to:
    - lxd:8
    options:
      enable-dvr: true
      enable-l3ha: true
      enable-ml2-port-security: true
      flat-network-providers: physnet1
      global-physnet-mtu: 9000
      neutron-security-groups: true
      vlan-ranges: dcfabric
    annotations:
      gui-x: "580"
      gui-y: "1270"
    constraints: arch=amd64 tags=cloud,ccompute
    bindings:
      "": op-public-space
      admin: op-public-space
      amqp: op-public-space
      certificates: op-public-space
      cluster: op-public-space
      etcd-proxy: op-public-space
      external-dns: op-public-space
      ha: op-public-space
      identity-service: op-public-space
      infoblox-neutron: op-public-space
      internal: op-public-space
      midonet: op-public-space
      neutron-api: op-public-space
      neutron-load-balancer: op-public-space
      neutron-plugin-api: op-public-space
      neutron-plugin-api-subordinate: op-public-space
      nrpe-external-master: op-public-space
      public: op-public-space
      shared-db: op-public-space
      vsd-rest-api: op-public-space
  neutron-api-plugin-ovn:
    charm: neutron-api-plugin-ovn
    channel: 2023.2/stable
    revision: 113
    annotations:
      gui-x: "690"
      gui-y: "1385"
    bindings:
      "": op-public-space
      certificates: op-public-space
      container: op-public-space
      neutron-plugin: op-public-space
      ovsdb-cms: op-public-space
  neutron-mysql-router:
    charm: mysql-router
    channel: 8.0/stable
    revision: 111
    annotations:
      gui-x: "505"
      gui-y: "1385"
    bindings:
      "": op-public-space
      certificates: op-public-space
      db-router: op-public-space
      juju-info: op-public-space
      shared-db: op-public-space
  nova-cloud-controller:
    charm: nova-cloud-controller
    channel: 2023.2/stable
    revision: 717
    resources:
      policyd-override: 0
    num_units: 1
    to:
    - lxd:7
    options:
      console-access-protocol: novnc
      network-manager: Neutron
    annotations:
      gui-x: "35"
      gui-y: "1270"
    constraints: arch=amd64 tags=cloud,ccompute
    bindings:
      "": op-public-space
      admin: op-public-space
      amqp: op-public-space
      amqp-cell: op-public-space
      certificates: op-public-space
      cinder-volume-service: op-public-space
      cloud-compute: op-public-space
      cloud-controller: op-public-space
      cluster: op-public-space
      dashboard: op-public-space
      ha: op-public-space
      identity-service: op-public-space
      image-service: op-public-space
      internal: op-public-space
      memcache: op-public-space
      neutron-api: op-public-space
      nova-cell-api: op-public-space
      nova-vmware: op-public-space
      nrpe-external-master: op-public-space
      placement: op-public-space
      public: op-public-space
      quantum-network-service: op-public-space
      shared-db: op-public-space
      shared-db-cell: op-public-space
  nova-compute:
    charm: nova-compute
    channel: 2023.2/stable
    revision: 703
    num_units: 3
    to:
    - "7"
    - "8"
    - "3"
    options:
      config-flags: default_ephemeral_format=ext4
      enable-live-migration: true
      enable-resize: true
      migration-auth-type: ssh
    annotations:
      gui-x: "190"
      gui-y: "890"
    constraints: arch=amd64 tags=cloud,ccompute
    storage:
      ephemeral-device: loop,10240M
    bindings:
      "": op-public-space
      amqp: op-public-space
      ceph: op-public-space
      ceph-access: op-public-space
      cloud-compute: op-public-space
      cloud-credentials: op-public-space
      compute-peer: op-public-space
      ephemeral-backend: op-public-space
      image-service: op-public-space
      internal: op-public-space
      ironic-api: op-public-space
      lxd: op-public-space
      migration: op-public-space
      neutron-plugin: op-public-space
      nova-ceilometer: op-public-space
      nova-vgpu: op-public-space
      nrpe-external-master: op-public-space
      secrets-storage: op-public-space
  nova-mysql-router:
    charm: mysql-router
    channel: 8.0/stable
    revision: 111
    annotations:
      gui-x: "-30"
      gui-y: "1385"
    bindings:
      "": op-public-space
      certificates: op-public-space
      db-router: op-public-space
      juju-info: op-public-space
      shared-db: op-public-space
  ntp:
    charm: ntp
    channel: stable
    revision: 50
    options:
      pools: 192.168.6.254 192.168.70.254 192.168.71.254
    annotations:
      gui-x: "315"
      gui-y: "1030"
    bindings:
      "": op-public-space
      juju-info: op-public-space
      master: op-public-space
      nrpe-external-master: op-public-space
      ntp-peers: op-public-space
      ntpmaster: op-public-space
  openstack-dashboard:
    charm: openstack-dashboard
    channel: 2023.2/stable
    revision: 626
    resources:
      policyd-override: 0
      theme: 0
    num_units: 1
    to:
    - lxd:8
    annotations:
      gui-x: "585"
      gui-y: "900"
    constraints: arch=amd64 tags=cloud,ccompute
    bindings:
      "": op-public-space
      application-dashboard: op-public-space
      certificates: op-public-space
      cluster: op-public-space
      dashboard: op-public-space
      dashboard-plugin: op-public-space
      ha: op-public-space
      identity-service: op-public-space
      nrpe-external-master: op-public-space
      public: op-public-space
      shared-db: op-public-space
      website: op-public-space
      websso-fid-service-provider: op-public-space
      websso-trusted-dashboard: op-public-space
  openstack-loadbalancer:
    charm: openstack-loadbalancer
    channel: jammy/stable
    revision: 10
    num_units: 3
    to:
    - lxd:7
    - lxd:8
    - lxd:3
    options:
      vip: 192.168.70.207
    constraints: arch=amd64
    bindings:
      "": op-public-space
      admin: op-public-space
      ha: op-public-space
      internal: op-public-space
      loadbalancer: op-public-space
      public: op-public-space
  openstack-loadbalancer-hacluster:
    charm: hacluster
    channel: 2.4/stable
    revision: 131
    bindings:
      "": op-public-space
      ha: op-public-space
      hanode: op-public-space
      juju-info: op-public-space
      nrpe-external-master: op-public-space
      pacemaker-remote: op-public-space
      peer-availability: op-public-space
  ovn-central:
    charm: ovn-central
    channel: 23.09/stable
    revision: 158
    num_units: 3
    to:
    - lxd:7
    - lxd:8
    - lxd:8
    annotations:
      gui-x: "70"
      gui-y: "1550"
    constraints: arch=amd64 tags=cloud,ccompute
    bindings:
      "": op-public-space
      certificates: op-public-space
      coordinator: op-public-space
      metrics-endpoint: op-public-space
      nrpe-external-master: op-public-space
      ovsdb: op-public-space
      ovsdb-cms: op-public-space
      ovsdb-peer: op-public-space
      ovsdb-server: op-public-space
  ovn-chassis:
    charm: ovn-chassis
    channel: 23.09/stable
    revision: 178
    options:
      bridge-interface-mappings: br-provider:bond0
      ovn-bridge-mappings: dcfabric:br-provider
    annotations:
      gui-x: "120"
      gui-y: "1030"
    bindings:
      "": op-public-space
      amqp: op-public-space
      certificates: op-public-space
      data: op-public-space
      juju-info: op-public-space
      metrics-endpoint: op-public-space
      nova-compute: op-public-space
      nrpe-external-master: op-public-space
      ovsdb: op-public-space
      ovsdb-subordinate: op-public-space
  placement:
    charm: placement
    channel: 2023.2/stable
    revision: 100
    num_units: 1
    to:
    - lxd:8
    annotations:
      gui-x: "1320"
      gui-y: "1270"
    constraints: arch=amd64 tags=cloud,ccompute
    bindings:
      "": op-public-space
      admin: op-public-space
      amqp: op-public-space
      certificates: op-public-space
      cluster: op-public-space
      ha: op-public-space
      identity-service: op-public-space
      internal: op-public-space
      placement: op-public-space
      public: op-public-space
      shared-db: op-public-space
  placement-mysql-router:
    charm: mysql-router
    channel: 8.0/stable
    revision: 111
    annotations:
      gui-x: "1320"
      gui-y: "1385"
    bindings:
      "": op-public-space
      certificates: op-public-space
      db-router: op-public-space
      juju-info: op-public-space
      shared-db: op-public-space
  rabbitmq-server:
    charm: rabbitmq-server
    channel: 3.9/stable
    revision: 182
    num_units: 1
    to:
    - lxd:8
    annotations:
      gui-x: "300"
      gui-y: "1550"
    constraints: arch=amd64 tags=cloud,ccompute
    bindings:
      "": op-public-space
      amqp: op-public-space
      certificates: op-public-space
      cluster: op-public-space
      dashboards: op-public-space
      ha: op-public-space
      nrpe-external-master: op-public-space
      prometheus-rules: op-public-space
      scrape: op-public-space
  vault:
    charm: vault
    channel: 1.8/stable
    revision: 183
    resources:
      core: 0
      vault: 0
    num_units: 1
    to:
    - lxd:7
    annotations:
      gui-x: "1610"
      gui-y: "1430"
    constraints: arch=amd64 tags=cloud,ccompute
    bindings:
      "": op-public-space
      access: op-public-space
      certificates: op-public-space
      cluster: op-public-space
      db: op-public-space
      etcd: op-public-space
      external: op-public-space
      ha: op-public-space
      lb-provider: op-public-space
      nrpe-external-master: op-public-space
      secrets: op-public-space
      shared-db: op-public-space
  vault-mysql-router:
    charm: mysql-router
    channel: 8.0/stable
    revision: 111
    annotations:
      gui-x: "1535"
      gui-y: "1560"
    bindings:
      "": op-public-space
      certificates: op-public-space
      db-router: op-public-space
      juju-info: op-public-space
      shared-db: op-public-space
machines:
  "3":
    constraints: tags=cloud,ceph spaces=op-public-space,op-cluster-space
  "4":
    constraints: tags=cloud,ceph spaces=op-public-space,op-cluster-space
  "5":
    constraints: tags=op2 spaces=op-cluster-space,op-public-space
  "7":
    constraints: tags=cloud,compute,xen01 spaces=op-public-space,op-cluster-space
  "8":
    constraints: tags=cloud,compute,op1 spaces=op-public-space,op-cluster-space
  "10":
    constraints: arch=amd64 tags=cloud,ceph spaces=op-cluster-space,op-public-space
relations:
- - ceph-osd:mon
  - ceph-mon:osd
- - ceph-radosgw:mon
  - ceph-mon:radosgw
- - ceph-mon:mds
  - ceph-fs:ceph-mds
- - ceph-dashboard:dashboard
  - ceph-mon:dashboard
- - nova-compute:amqp
  - rabbitmq-server:amqp
- - nova-cloud-controller:identity-service
  - keystone:identity-service
- - glance:identity-service
  - keystone:identity-service
- - neutron-api:identity-service
  - keystone:identity-service
- - neutron-api:amqp
  - rabbitmq-server:amqp
- - glance:amqp
  - rabbitmq-server:amqp
- - nova-cloud-controller:image-service
  - glance:image-service
- - nova-compute:image-service
  - glance:image-service
- - nova-cloud-controller:cloud-compute
  - nova-compute:cloud-compute
- - nova-cloud-controller:amqp
  - rabbitmq-server:amqp
- - openstack-dashboard:identity-service
  - keystone:identity-service
- - nova-cloud-controller:neutron-api
  - neutron-api:neutron-api
- - cinder:image-service
  - glance:image-service
- - cinder:amqp
  - rabbitmq-server:amqp
- - cinder:identity-service
  - keystone:identity-service
- - cinder:cinder-volume-service
  - nova-cloud-controller:cinder-volume-service
- - cinder-ceph:storage-backend
  - cinder:storage-backend
- - ceph-mon:client
  - nova-compute:ceph
- - nova-compute:ceph-access
  - cinder-ceph:ceph-access
- - ceph-mon:client
  - cinder-ceph:ceph
- - ceph-mon:client
  - glance:ceph
- - ceph-radosgw:identity-service
  - keystone:identity-service
- - placement:identity-service
  - keystone:identity-service
- - placement:placement
  - nova-cloud-controller:placement
- - keystone:shared-db
  - keystone-mysql-router:shared-db
- - cinder:shared-db
  - cinder-mysql-router:shared-db
- - glance:shared-db
  - glance-mysql-router:shared-db
- - nova-cloud-controller:shared-db
  - nova-mysql-router:shared-db
- - neutron-api:shared-db
  - neutron-mysql-router:shared-db
- - openstack-dashboard:shared-db
  - dashboard-mysql-router:shared-db
- - placement:shared-db
  - placement-mysql-router:shared-db
- - vault:shared-db
  - vault-mysql-router:shared-db
- - keystone-mysql-router:db-router
  - mysql-innodb-cluster:db-router
- - cinder-mysql-router:db-router
  - mysql-innodb-cluster:db-router
- - nova-mysql-router:db-router
  - mysql-innodb-cluster:db-router
- - glance-mysql-router:db-router
  - mysql-innodb-cluster:db-router
- - neutron-mysql-router:db-router
  - mysql-innodb-cluster:db-router
- - dashboard-mysql-router:db-router
  - mysql-innodb-cluster:db-router
- - placement-mysql-router:db-router
  - mysql-innodb-cluster:db-router
- - vault-mysql-router:db-router
  - mysql-innodb-cluster:db-router
- - neutron-api-plugin-ovn:neutron-plugin
  - neutron-api:neutron-plugin-api-subordinate
- - ovn-central:certificates
  - vault:certificates
- - ovn-central:ovsdb-cms
  - neutron-api-plugin-ovn:ovsdb-cms
- - neutron-api:certificates
  - vault:certificates
- - ovn-chassis:nova-compute
  - nova-compute:neutron-plugin
- - ovn-chassis:certificates
  - vault:certificates
- - ovn-chassis:ovsdb
  - ovn-central:ovsdb
- - vault:certificates
  - neutron-api-plugin-ovn:certificates
- - vault:certificates
  - cinder:certificates
- - vault:certificates
  - glance:certificates
- - vault:certificates
  - keystone:certificates
- - vault:certificates
  - nova-cloud-controller:certificates
- - vault:certificates
  - openstack-dashboard:certificates
- - vault:certificates
  - placement:certificates
- - vault:certificates
  - ceph-radosgw:certificates
- - vault:certificates
  - mysql-innodb-cluster:certificates
- - ntp:juju-info
  - nova-compute:juju-info
- - ceph-dashboard:certificates
  - vault:certificates
- - openstack-loadbalancer:ha
  - openstack-loadbalancer-hacluster:ha
- - ceph-dashboard:loadbalancer
  - openstack-loadbalancer:loadbalancer

[Luciano Lo Giudice (lmlogiudice)]

Hello,

Could you ssh into one of the active ceph monitors and paste the contents of the '/etc/ceph.conf' file?

[macchese (max-liccardo)]

cat ceph.conf

[global]
auth cluster required = cephx
auth service required = cephx
auth client required = cephx

mon host = 192.168.70.14 192.168.70.28 192.168.70.33
fsid = 7dfd9e3a-a5b6-11ee-9798-619012c1bb3a

log to syslog = false
err to syslog = false
clog to syslog = false
mon cluster log to syslog = false
debug mon = 1/5
debug osd = 1/5

# NOTE(jamespage):
# Disable object skew warnings as these only use
# the number of objects and not their size in the
# skew calculation.
mon pg warn max object skew = -1

public network =
cluster network =
public addr = 192.168.70.14
cluster addr = 192.168.71.13

[mon]
keyring = /var/lib/ceph/mon/$cluster-$id/keyring

mon data avail warn = 30
mon data avail crit = 5

[mds]
keyring = /var/lib/ceph/mds/$cluster-$id/keyring

[Luciano Lo Giudice (lmlogiudice)]

OK, so I'm assuming that the removed unit (juju-206dbb-3-lxd-1) is not among the listed IPs in the field 'mon host'. If that's the case, then the monmap and conf file are correct, which may imply that the use case of moving nodes among LXD containers is somehow problematic.

Is the ceph status still showing an out-of-quorum situation? Sometimes it takes a bit for it to be updated.

[macchese (max-liccardo)]

correct, after 12 hours I removed by hand by
ceph mon rm xxxx
where the xxxx was the mon.
Until I removed it ceph status showed him out of quorum, now it is all ok

[macchese (max-liccardo)]

looking into ceph.conf's glance and cinder-ceph, I noticed that the old mon (192.168.70.12) I removed still exist even after 3 days, where the new mon unit is 192.1678.70.28

##############################################################################
# [ WARNING ]
# ceph configuration file maintained by Juju
# local changes may be overwritten.
###############################################################################
[global]
auth_supported = cephx
keyring = /etc/ceph/$cluster.$name.keyring
mon host = 192.168.70.12 192.168.70.14 192.168.70.28 192.168.70.33
log to syslog = false
err to syslog = false
clog to syslog = false

[macchese (max-liccardo)]

another side effect I found is when using ceph-dashboard in HA mode by charm-openstack-loadbalancer.
In that case the removed ip mon (192.168.70.12:8443) remained into the haproxy conf

frontend ceph_dashboard_front
  mode tcp
  bind *:8443
  option tcplog
  default_backend ceph_dashboard_back

backend ceph_dashboard_back
  mode tcp
  option httpchk GET /
  http-check expect status 200
  server ceph-dashboard_3 192.168.70.12:8443 check-ssl check verify none
  server ceph-dashboard_5 192.168.70.14:8443 check-ssl check verify none
  server ceph-dashboard_6 192.168.70.33:8443 check-ssl check verify none
  server ceph-dashboard_7 192.168.70.28:8443 check-ssl check verify none