I've done some more digging, and my working theory is that it's a race condition triggered when the charm refresh and config change is done by cou (charmed-openstack-upgrader) in quick succession (without waiting in between).
This is what the flow of events seems to look like for ceph-mon:
1. cou upgrades the charm channel, this is registered in juju
2. cou changes the source config from cloud:focal-victoria -> cloud:focal-wallaby, this is registered immediately in juju
3. the charm runs the upgrade-charm hook
4. the charm loads config from juju (the new config after being updated in step 2)
5. charmhelpers saves the config to .juju-persistent-config (with the new value)
6. upgrade-charm hook completes, config-changed hook begins
7. upgrade procedure in the config-changed hook compares config.previous('source') with the live config, but these are the same, so no upgrade happens
I'm finding it very difficult to trace the flow of logic for aodh (and probably same will go for the other openstack charms here), but I'd guess a similar thing is happening.
Consider:
- these bugs affect many charms, but the bugs have only been observed by upgrading with cou
- cou will be different to a human upgrading: it fires charm refresh and config change very quickly without waiting in between. A human is likely to be waiting longer between commands (eg. waiting for charm upgrade to complete before changing the config).
I can't explain yet why these bugs have only been observed in victoria -> wallaby upgrades.
I've done some more digging, and my working theory is that it's a race condition triggered when the charm refresh and config change is done by cou (charmed- openstack- upgrader) in quick succession (without waiting in between).
This is what the flow of events seems to look like for ceph-mon:
1. cou upgrades the charm channel, this is registered in juju victoria -> cloud:focal- wallaby, this is registered immediately in juju t-config (with the new value) previous( 'source' ) with the live config, but these are the same, so no upgrade happens
2. cou changes the source config from cloud:focal-
3. the charm runs the upgrade-charm hook
4. the charm loads config from juju (the new config after being updated in step 2)
5. charmhelpers saves the config to .juju-persisten
6. upgrade-charm hook completes, config-changed hook begins
7. upgrade procedure in the config-changed hook compares config.
I'm finding it very difficult to trace the flow of logic for aodh (and probably same will go for the other openstack charms here), but I'd guess a similar thing is happening.
Consider:
- these bugs affect many charms, but the bugs have only been observed by upgrading with cou
- cou will be different to a human upgrading: it fires charm refresh and config change very quickly without waiting in between. A human is likely to be waiting longer between commands (eg. waiting for charm upgrade to complete before changing the config).
I can't explain yet why these bugs have only been observed in victoria -> wallaby upgrades.