charm upgrade breakage
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
keystone (Juju Charms Collection) |
Fix Released
|
Critical
|
Edward Hope-Morley | ||
nova-cloud-controller (Juju Charms Collection) |
Fix Released
|
Critical
|
Edward Hope-Morley |
Bug Description
If you upgrade the keystone charm and deploy services or add units and relate them with keystone and you see the following (and your db is already initialised/
* New services fail to have their [keystone_
* /var/log/
It is most probably because of:
At some point the keystone charm added support for checking whether the database is ready and initialised before allowing relations to be configured. The addition uses a db-initialised setting on the cluster (peer) relation to determine whether the database is ready for action and gates identity-relation handling on this (by deferring handling of identity relation changes until the db is ready).
The problem is that if you upgrade from a charm version prior to this being added to one that does AND your database is already initialised and ready, your identity relations will block indefinitely (unless the shared-db hook happens to fire).
I have tried the following workaround but this should be fixed in the charm:
If you upgrade the keystone charm and deploy services or add units and
relate them with keystone and the new services fail to have their
[keystone_
* Establish leader
(>= Juju 1.24)
$ juju run --unit keystone/0 "is-leader"
...
(< Juju 1.24)
$ juju ssh keystone/0 sudo crm status
* Check that the db is ready (you need to ensure that it is initialised as well which can be checked in few ways but e.g. if any of your services are already using the DB then it is likely ready and initialised). Here we see that all keystone units are listed in "allowed units"
$ juju run --unit keystone/0 "relation-ids shared-db"
shared-db:24
$ juju run --unit keystone/0 "relation-get -r shared-db:24 - percona-cluster/0"
allowed_units: keystone/0 keystone/1 keystone/2
db_host: 10.0.0.1
password: ubuntu
private-
* Check that you don't have db-initialised set anywhere
$ juju run --unit keystone/0 "relation-ids cluster"
cluster:9
$ for i in {0..2}; do juju run --unit keystone/$i "relation-get -r cluster:9 - keystone/$i"| grep db-initialised; done
* Set it and wait for things to repair themselves
# Assuming keystone/0 is leader
$ juju run --unit keystone/0 "relation-set -r cluster:9 db-initialised=
Related branches
- Liam Young (community): Approve
-
Diff: 294 lines (+95/-40)2 files modifiedhooks/keystone_hooks.py (+33/-23)
unit_tests/test_keystone_hooks.py (+62/-17)
- Liam Young (community): Approve
-
Diff: 249 lines (+90/-33)3 files modifiedhooks/nova_cc_hooks.py (+55/-27)
hooks/nova_cc_utils.py (+18/-1)
unit_tests/test_nova_cc_hooks.py (+17/-5)
description: | updated |
summary: |
- keystone upgrade breaks identity-relation hooks + keystone charm upgrade breaks identity-relation hooks |
description: | updated |
summary: |
- keystone charm upgrade breaks identity-relation hooks + charm upgrade breakage |
Changed in nova-cloud-controller (Juju Charms Collection): | |
milestone: | none → 16.01 |
importance: | Undecided → Critical |
Changed in keystone (Juju Charms Collection): | |
assignee: | nobody → Edward Hope-Morley (hopem) |
status: | New → In Progress |
Changed in nova-cloud-controller (Juju Charms Collection): | |
status: | New → In Progress |
assignee: | nobody → Edward Hope-Morley (hopem) |
Changed in keystone (Juju Charms Collection): | |
status: | In Progress → Fix Committed |
Changed in nova-cloud-controller (Juju Charms Collection): | |
status: | In Progress → Fix Committed |
Changed in keystone (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in nova-cloud-controller (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
The nova-cloud- controller charm appears to be suffering from a similar case of upgrade flu. If i upgrade to the latest charm from one prior to having support for 'dbsync_state' on the cluster relation, the charm disables all services until dbsync_state gets set "complete"...which never happens because it can only happen if you do an action- managed- upgrade. ..which we are not.