feature: per-unit passwords for rolling password updates on control plane services

[Dmitrii Shcherbakov]

A common security requirement is an ability to change passwords for database users over time in a controlled manner.

Highly-available control plane setups need to be taken down completely if one username and password is used for all units of a control plane service.

The per-unit approach could be done in a form of an action:

1) an operator decides that a password for keystone-0 unit used to access the "keystone" database must be changed because the unit has been compromised in some way;
2) the operator executes an action on the percona-cluster leader unit to generate a new random password for the keystone/0 unit which also propagates that information to keystone/0 over the relation;
3) keystone/0 unit restarts the keystone service which only results in a downtime for a single unit while user requests are forwarded to other units (via pacemaker + VIP-based HA, DNS-HA etc.)

This way an operator would be able to change passwords for the whole control plane one unit at a time.

[Dmitrii Shcherbakov (dmitriis)]

Currently, the only procedure to change an application password I found is as follows:

# removing a relation does not result in removal or re-rendering of configs or service downtime (the Keystone charm does not stop the service)
juju remove-relation keystone mysql

# a service-name based password entry in leader settings needs to be updated
juju run --unit mysql/0 'leader-set mysql-keystone.passwd=newpasswd'

# triggers password retrieval from leader settings followed by placing it into the relation bucket for keystone units to consume
juju add-relation keystone mysql

[Dmitrii Shcherbakov (dmitriis)]

The procedure mentioned in #1 does not take into account that grants are not created if they already exist so even if a password is changed in a service config file and leader settings of the percona application, it is not changed in the DB itself:

2018-10-06 15:44:33 DEBUG juju-log shared-db:94: Grant exists for host '10.232.6.200' on db 'keystone'

And so services stop working after a restart.

What needs to be done manually is something like this:

GRANT ALL PRIVILEGES ON `keystone`.* TO 'keystone'@'<client-ip>' IDENTIFIED BY 'newpasswd'