charm upgrade breakage

[Edward Hope-Morley]

If you upgrade the keystone charm and deploy services or add units and relate them with keystone and you see the following (and your db is already initialised/migrated):

    * New services fail to have their [keystone_authtoken] section populated

    * /var/log/juju/unit-keystone-0.log has "Database is NOT initialised" or "Database not ready - deferring to shared-db relation"

It is most probably because of:

At some point the keystone charm added support for checking whether the database is ready and initialised before allowing relations to be configured. The addition uses a db-initialised setting on the cluster (peer) relation to determine whether the database is ready for action and gates identity-relation handling on this (by deferring handling of identity relation changes until the db is ready).

The problem is that if you upgrade from a charm version prior to this being added to one that does AND your database is already initialised and ready, your identity relations will block indefinitely (unless the shared-db hook happens to fire).

I have tried the following workaround but this should be fixed in the charm:

If you upgrade the keystone charm and deploy services or add units and
relate them with keystone and the new services fail to have their
[keystone_authtoken] section populated, it is most probably because of this:

* Establish leader

    (>= Juju 1.24)
    $ juju run --unit keystone/0 "is-leader"
    ...

    (< Juju 1.24)
    $ juju ssh keystone/0 sudo crm status

* Check that the db is ready (you need to ensure that it is initialised as well which can be checked in few ways but e.g. if any of your services are already using the DB then it is likely ready and initialised). Here we see that all keystone units are listed in "allowed units"

    $ juju run --unit keystone/0 "relation-ids shared-db"
    shared-db:24
    $ juju run --unit keystone/0 "relation-get -r shared-db:24 - percona-cluster/0"
    allowed_units: keystone/0 keystone/1 keystone/2
    db_host: 10.0.0.1
    password: ubuntu
    private-address: 10.0.0.2

* Check that you don't have db-initialised set anywhere

    $ juju run --unit keystone/0 "relation-ids cluster"
    cluster:9
    $ for i in {0..2}; do juju run --unit keystone/$i "relation-get -r cluster:9 - keystone/$i"| grep db-initialised; done

* Set it and wait for things to repair themselves

    # Assuming keystone/0 is leader
    $ juju run --unit keystone/0 "relation-set -r cluster:9 db-initialised=True"

[Edward Hope-Morley (hopem)]

The nova-cloud-controller charm appears to be suffering from a similar case of upgrade flu. If i upgrade to the latest charm from one prior to having support for 'dbsync_state' on the cluster relation, the charm disables all services until dbsync_state gets set "complete"...which never happens because it can only happen if you do an action-managed-upgrade...which we are not.