The following code is to be found in the config_changed hook of the nova-compute charm:
for rid in relation_ids('ceph'):
for unit in related_units(rid):
ceph_changed(rid=rid, unit=unit)
This causes the ceph_changed() function to run once per rid.
ceph_changed() calls is_broker_action_done(), which checks for the existence of broker-rsp-UNITNAME-0. ceph_changed() then conditionally restarts nova-compute:
if is_request_complete(get_ceph_request()):
log('Request complete')
# Ensure that nova-compute is restarted since only now can we
# guarantee that ceph resources are ready, but only if not paused.
if (not is_unit_paused_set() and
not is_broker_action_done('nova_compute_restart', rid,
unit)):
service_restart('nova-compute')
mark_broker_action_done('nova_compute_restart', rid, unit)
else:
send_request_if_needed(get_ceph_request())
If the unit we are processing does not have broker-rsp-UNITNAME-0 set, then is_broker_action_done() will always return False:
rdata = relation_get(rid=rid, unit=unit) or {}
broker_rsp = rdata.get(get_broker_rsp_key())
if not broker_rsp:
return False
Therefore the conditional expression in ceph_changed() will evaluate to True, unless the unit is paused, and restart the nova-compute service.
However only the leader has broker-rsp-UNITNAME-0 set; it is set by the handle_broker_request() function in the ceph-mon charm.
The non-leader units don't have broker-rsp-UNITNAME-0 set, as can be seen below:
ceph-mon/0
{u'auth': u'cephx', u'private-address': u'10.0.0.25', u'broker-rsp-nova-compute-0': u'{"request-id": "551b0a73-9bb4-11e9-9544-525400eb408b", "exit-code": 0}', u'broker-rsp-nova-compute-1': u'{"exit-code": 0, "request-id": "b8d82fa1-9c62-11e9-b901-5254005cde21"}', u'key': u'AQCHhBldMRF3JBAAmIHYqPHRbUJiFlrCdYHBCw==', u'broker_rsp': u'{"exit-code": 0, "request-id": "b8d82fa1-9c62-11e9-b901-5254005cde21"}', u'ceph-public-address': u'10.0.0.25'}
ceph-mon/1
{u'key': u'AQCHhBldMRF3JBAAmIHYqPHRbUJiFlrCdYHBCw==', u'private-address': u'10.0.0.30', u'ceph-public-address': u'10.0.0.30', u'auth': u'cephx'}
..
The upshot is that when jujud-unit-nova-compute-x is restarting, it restarts nova-compute service several times as well, once for each ceph-mon unit, which is undesirable.
I would add the code below to remove those unnecessary restarts, and restart only once - when we process the ceph-mon leader:
1) charm-helpers in hooks/charmhelpers/contrib/storage/linux/ceph.py
def has_broker_rsp(rid=None, unit=None):
rdata = relation_get(rid=rid, unit=unit) or {}
broker_rsp = rdata.get(get_broker_rsp_key())
if not broker_rsp:
return False
return True
2) charm-nova-compute in hooks/nova_compute_hooks.py
if has_broker_rsp(rid, unit):
if is_request_complete(get_ceph_request()):
log('Request complete')
# Ensure that nova-compute is restarted since only now can we
# guarantee that ceph resources are ready, but only if not paused.
if (not is_unit_paused_set() and
not is_broker_action_done('nova_compute_restart', rid,
unit)):
service_restart('nova-compute')
mark_broker_action_done('nova_compute_restart', rid, unit)
else:
send_request_if_needed(get_ceph_request())
PR for charm-helpers
https:/