bootstrapped-osds not updated when using zap/add-disk actions

[Edward Hope-Morley]

I deployed a ceph cluster using next (20.02) charms where each node has 6 osds. All was well, every unit active/idle. I then reformatted two osds (keeping the same device name) and they came up ok but the ceph-mon is now stuck with:

ceph-mon/0 waiting idle 0/lxd/0 10.230.56.225 Monitor bootstrapped but waiting for number of OSDs to reach expected-osd-count (18)

Checking the relation between mon and osd i see:

$ juju run -u ceph-mon/0 -- relation-get -r osd:10 bootstrapped-osds ceph-osd/0
4

But they are all definitely up.

$ juju ssh ceph-osd/0 -- pgrep -alf bin/ceph-osd
1108725 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
1109368 /usr/bin/ceph-osd -f --cluster ceph --id 4 --setuser ceph --setgroup ceph
1109958 /usr/bin/ceph-osd -f --cluster ceph --id 6 --setuser ceph --setgroup ceph
1110639 /usr/bin/ceph-osd -f --cluster ceph --id 8 --setuser ceph --setgroup ceph
1113021 /usr/bin/ceph-osd -f --cluster ceph --id 10 --setuser ceph --setgroup ceph
1113667 /usr/bin/ceph-osd -f --cluster ceph --id 12 --setuser ceph --setgroup ceph

[Edward Hope-Morley (hopem)]

Bit more visualistaion - https://pastebin.ubuntu.com/p/wk4M9bgXGn/

So the count is updated in ceph_hooks.prepare_disks_and_activate() which is not called directly from any action but in my case was called subsequent to a zap-disk but not after the add-disk actions hence why the count was never updated.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/705203

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/705203
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-osd/commit/?id=871b0573355a05d52e7082d00bc3abcc29eae9cf
Submitter: Zuul
Branch: master

commit 871b0573355a05d52e7082d00bc3abcc29eae9cf
Author: Edward Hope-Morley <email address hidden>
Date: Fri Jan 31 12:12:08 2020 +0000

    Ensure bootstrapped-osds count updated after add-disk

    When we add/zap disks it may change the overall osd
    count so need to ensure this is kept up-to-date.

    Change-Id: Ib55547f88316e80a8948ce808ea992c1402458f5
    Closes-Bug: #1861293