client relations are never processed when configuration change (monitor-count, expected-osd-count) causes cluster to become ready

[Edward Hope-Morley]

If i deploy 1 unit of ceph-mon with monitor-count=3 it failes to bootstrap as expected. If i then set monitor-count=1 i would expect it to then bootstap but instead it does nothing and I have to remove and re-add the ceph client relations to get it to happen.

[Edward Hope-Morley (hopem)]

I have now found some additional issues with this logic. If I start by deploying 3 units with monitor-count=1 (which I know if wrong) what happens is that all 3 units will bootstrap independently i.e. at the point at which they have what they consider to be sufficient hosts i.e. 1 a.k.a themselves. The problem is that if I then set monitor-count=3 it remains wedged and the only way to fix it is to delete 2 units and start again. The same is also True if I have N mon units with monitor-count=N and i scale out my ceph-mon application before updating monitor-count. I think the charm should be able to manage these scenarios somehow.

[Edward Hope-Morley (hopem)]

and https://bugs.launchpad.net/charm-ceph-mon/+bug/1784860

[Chris MacNaughton (chris.macnaughton)]

I suspect that the best way to manage some of these issues would be to only allow the juju leader to bootstrap, as there can be only one ;-)

[Trent Lloyd (lathiat)]

The problem here is that both client_relation_joined and client_relation_changed hooks (which provides ceph keys & processes broker requests to create pools etc) check for "if ready_for_service()" before processing those requests.

If not ready for service, the hooks do nothing. However charm hooks are not re-run just because we weren't ready to process them.

The charm either needs to queue a list of such relations to process later, or, needs to iteratively check all such relations in some way when it later decides it is ready for service.

This seems to be a common charm design problem that probably needs some documentation written for charm authors to describe how to handle this situation cleanly. As it's not 100% obvious.

The ceph-mon charm does indeed have such a function to re-process the hook requests, which is notify_client(). However this function is not re-run in the case of config_changed. Presently it appears to happen only in the case of osd_relation (presumably to check if expected-monitor-count is now exceeded), upgrade_charm and also mon_relation.

As a result in the originally described situation (changing monitor-count=1) the client relations are not re-triggered. It also doesn't happen if expected-osd-count is updated, which is the situation I hit this same issue. expected-osd-count was not hit, so the relations were skipped and then even if you update the config they are not reprocessed.

[Trent Lloyd (lathiat)]

I also filed a bug that if expected-osd-count is not meant (or otherwise ready_for_service check fails), there is no juju status information to convey that information to the admin:
https://bugs.launchpad.net/charm-ceph-mon/+bug/1807652

[Trent Lloyd (lathiat)]

Fix Proposed: https://review.openstack.org/#/c/624010/

[Trent Lloyd (lathiat)]

This was fixed by:
https://review.opendev.org/#/c/640345/

Which appears to have made the 19.04 charm release

[OpenStack Infra (hudson-openstack)]

Change abandoned by Trent Lloyd (<email address hidden>) on branch: master
Review: https://review.opendev.org/624010
Reason: already fixed by 640345 instead

[Alex Kavanagh (ajkavanagh)]

Changing to fix released as Trent indicates that it has already been fixed in the 19.04 charms. Thanks for confirming!