Monitor bootstrapped but waiting for number of OSDs to reach expected-osd-count

[Vidmantas]

Ubuntu LTS 18.04 juju deployed Openstack Queens bundle
ceph-mon/7 waiting idle 7/lxd/1 Monitor bootstrapped but waiting for number of OSDs to reach expected-osd-count (3)
ceph-mon/8* waiting idle 8/lxd/2 Monitor bootstrapped but waiting for number of OSDs to reach expected-osd-count (3)
ceph-mon/9 waiting idle 11/lxd/1 Monitor bootstrapped but waiting for number of OSDs to reach expected-osd-count (3)
ceph-osd/1 active idle 7 Unit is ready (1 OSD)
ceph-osd/2* active idle 8 Unit is ready (1 OSD)
ceph-osd/3 active idle 11 Unit is ready (1 OSD)

root@juju-b054bf-11-lxd-1:~# ceph status
  cluster:
    id: 62133458-0e96-11e9-aa87-00163efa872e
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum juju-b054bf-8-lxd-2,juju-b054bf-7-lxd-1,juju-b054bf-11-lxd-1
    mgr: juju-b054bf-8-lxd-2(active), standbys: juju-b054bf-11-lxd-1, juju-b054bf-7-lxd-1
    osd: 0 osds: 0 up, 0 in

  data:
    pools: 0 pools, 0 pgs
    objects: 0 objects, 0B
    usage: 0B used, 0B / 0B avail
    pgs:
root@juju-b054bf-11-lxd-1:~# ceph df
GLOBAL:
    SIZE AVAIL RAW USED %RAW USED
      0B 0B 0B 0
POOLS:
    NAME ID USED %USED MAX AVAIL OBJECTS

Ceph not detecting OSD

[Fabrizio Cardarello (f.cardarello)]

I also deployed a charm openstack-base-bionic-queens-3 and encountred you same issue.
I tried to downgrade the charm ceph-mon, I didn't resolve the problem. The only things is too cheet.
set expected-osd-count to 1 and everythings work fine.

[Chris MacNaughton (chris.macnaughton)]

It looks, to me, like those machines have been re-used? The ceph-osd charm's osd count "(1 OSD)" comes from a `pgrep ceph-osd`, so re-using machines can lead to a scenario where there is a ceph osd process running on the machine, but that OSD is _not_ configured to talk to the monitors.

Can you confirm how you deployed these applications? A reproducing minimal bundle would be very helpful.

The reason that I suspect that these OSDs are not correctly configured to talk to the monitors is that the `ceph status` shows "0 in" in the OSD data, which means that ceph has NOT bootstrapped any OSDs into the cluster.

[Launchpad Janitor (janitor)]

[Expired for OpenStack ceph-mon charm because there has been no activity for 60 days.]

[Frankline Ngatia (fngatia)]

Please try and re-add the relationship again: juju add-relation ceph-mon:osd ceph-osd:mon