keystone-charm does not change user password if the user exists on identity-service relation joined/changed

[Mihai Gheorghe]

We have a need to move all Openstack services from the current relation with keystone to a another keystone in another model (inter-model relations).
We created a new model with keystone, mysql-innodb-cluster and vault which will operate independantly from the current deployed model.
Dumped the current keystone DB and imported it into the new keystone deployment.
We proceeded and created an offer from the new keystone for identity-service relation.
For each charm (glance,cinder,nova etc.) related to the old keystone via identity-service relation we removed the relation (juju remove-relation), waited for things to settle and added the relation with the new keystone (juju add-relation)

As well known, keystone charm creates 2 sets of users for the services it relates via identity-service endpoint, one in the default domain (which afaik is not used, it is only there for backwards compat with v2 api) and one user in service_domain domain, which is actually used in services configs to auth with keystone.

When a new relation is created on the identity-service endpoint, keystone charm creates the user and a random password which it sends it to the related charm as relation data and also creates the user and sets the password in Openstack, so it ends up in the DB.

As I said, we imported the keystone DB from the current deployment in the new keystone deployement, which means that all the users are already created and have their passwords set (reflected in the Keystone DB as hashes).

Let's take Glance charm as an example:
After the relation is removed from the old keystone and the charm joins the identity-service relation with the new Keystone, the Keystone charm randomly creates a new password for the glance service user (both in the default and service_domain domains) and sends the data as relation data, which in turn, glance charm writes it in the config under [keystone_auth] section.

But here is where things do not go as expected. Because the user already exists in Openstack but with a different password (the old password from the first relation with the old keystone), Glance is unable to authenticate with Keystone because it is using the new supplied password for the service user but the Openstack user is still using the old password.
Bassically, Keystone charm send the new password as relation data but does not update the Openstack user.
More strangely is that the glance user in the default domain has it's password changed in Openstack by the Keystone charm, but the service users in the service_domain domain are not updated with the new password. This behaviour can be seen directly in the Keystone DB, password table, where the users in default domain have 2 entries, one with an expiration date for the old password and a new row with the hash for the new password, where the users from service_domain are not update and only have the original entry.

The only charm which we encountered that does not have this behaviour is cinder-charm, because the original user is named cinderv2_cinderv3, but when creating the relation with the new keystone a new user named cinderv3 is created.

We are currently thinking of either polling the identity-service relation for the new password and manually change it using openstack-client, or adding the code in keystone-charm to automatically change the password.

Is this behaviour by design or is it something else?

Juju version: 2.9.42
Mysql-innodb-cluster charm channel: 8.0/stable
Openstack charms channel: yoga/stable
Openstack release: Yoga
Ubuntu: Focal

Thanks

[Alex Kavanagh (ajkavanagh)]

Yup, you've entered 'undefined behaviour' territory for the keystone charm; the design is not really set up to swap out a keystone and swap in another; there's obviously some behaviour on either the related charms (e.g. glance) or on keystone on the relations that's causing the issue.

You could try the "rotate-service-user-password" action on the keystone charm, which will generate a new password and then try to push it out to the related service. This may correct things // however, it's definitely not been tested in this environment.

I'm setting this to wish-list as it will probably require some clean-up code/action to 'reset' the relation and re-fetch all the data from keystone in related charms. need a bit of digging to work out what is going on.