charms.openstack based charms don't handle update-status very well

Bug #1702316 reported by Alex Kavanagh on 2017-07-04
This bug affects 1 person
Affects Status Importance Assigned to Milestone

Bug Description

Several of the charms.openstack based charms.reactive charms exhibit a similar problem: unintended behaviour when the update-status hook is called. This is due to the way that charms.reactive models handlers; handlers will always run when their 'conditions' are true, and this happens for EVERY hook that invokes the charm code.

An example of this is the designate charm, with a bug in charms.openstack core code interacting with a rendering of the designate.conf file. The bug was that there is a number of servers in a dictionary (hash) and these have indeterminate order in Python 3.5 -- Python 3.6 fixes this behaviour once-and-for-all by making the dictionary always respect the insert order. Python 2.7 also has indeterminate dictionary order, but seemed to always return the same order (not necessarily insert-order). These servers were written (in an indeterminate order) to the conf file, which more often than not, resulted in a restart of the services in the payload.

The keystone-ldap charm also appears to have a similar issue, except that it causes a restart of the keystone charm (probably) due to the keystone-ldap writing a new dictionary (serialised as a string on the relation) to the keystone charm, resulting in a restart of a service in keystone.

There is an additional element to this in that the 'update-status' hook is often "doing too much work". i.e. running all of the true-condition handlers, which may kick off "apt-get update", setting up databases, and all other manner of things.

So several approaches can be made to alievate this issue:

1. Work through each of the charms to resolve the individual handlers that are not idempotent. i.e. they shouldn't be exhibiting behaviour that causes themselves, or related charms, to do things when no external data has changed (config or relation). This should probably be done anyway.

2. Add a feature to (probably) layer-openstack & charms.openstack to 'deactivate' the update-status hook such that no 'handlers' are run, but that the update-status code can still read the relation states and config to determine the current workload status. This means not calling into charms.reactive a little differently (and this may be brittle as charms.reactive matures, as it wouldn't necessarily be using a 'public' API).

An additional issue with 2. is that some reactive charms may have latent bugs in the form of race-hazards where a relation hook doesn't get the charm to an operational state and it requires an additional 'kick' in the form of update-status to move the charm to completion. I've seen manila and designate exhibit this behaviour, but have no concrete ideas on what the race-hazard might be.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers