bootstrapped-osds in ceph-mon relation does not equal number of OSDs in status

[Drew Freiberger]

ceph-mon/0 waiting executing 0/lxd/0 1.2.3.4 Monitor bootstrapped but waiting for number of OSDs to reach expected-osd-count (38)

I'm seeing the above error when upgrading charms from 2018 versions to 21.01.

It seems that there is a discrepancy or non-updating relation when additional OSDs are identified.

What I see is that my mon shows it's expecting 38 OSDS, and when I count all of the 'Unit is ready (3 OSD)' and 'Unit is ready (2 OSD)' statuses of all of my related ceph-osd units, they add up to 38, however, when I inspect the relation information, I see that there are only "2" bootstrapped-osds showing in each unit's relation.

I've worked around this by running a relation-set on the units with 3 OSDS to update bootstrapped-osds=3, but I wonder if config-changed or update-status could catch this relation's discrepancy vs the status count and update the count in the relation.

This has mostly (maybe only?) been seen in environments upgrading to a newer ceph-mon that supports waiting for expected OSD count before bootstrapping from a version that was prior to that functionality.