openstack app's admin account gets locked after change password

[George Postolache]

Brief Description
-----------------
After changing the password the keyring is still showing the old password. For a short while openstack commands work using the new password and then the account gets locked.

1853017 similar to this defect except that "keyring get CGCS admin" shows the old password

Severity
--------
Provide the severity of the defect.
Major

Steps to Reproduce
------------------
1. openstack user set --password "N3wpassword*" admin
2. keyring get CGCS admin (is still showing the old password)
3. edit clouds.yml with the new password
4. export OS_CLOUD=openstack_helm
5. openstack network list (will execute normaly for a short while and then The account is locked for user: c8aab57d3ffa447283918b5088ea47d4. (HTTP 401) (Request-ID: req-b3e81f4a-df41-4a84-8498-4f940b048ca1))

Expected Behavior
------------------
keyring updated, commands working with the new password and account not getting locked

Actual Behavior
----------------
keyring is not updated and account is getting locked

Reproducibility
---------------
allways

System Configuration
--------------------
Multi-node system

Branch/Pull Time/Commit
-----------------------
controller-0:~$ cat /etc/build.info
###
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="20.06"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20200708T013409Z"

JOB="STX_build_layer_flock_master_master"
<email address hidden>"
BUILD_NUMBER="165"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2020-07-08 01:34:09 +0000"

FLOCK_OS="centos"
FLOCK_JOB="STX_build_layer_flock_master_master"
<email address hidden>"
FLOCK_BUILD_NUMBER="165"
FLOCK_BUILD_HOST="starlingx_mirror"
FLOCK_BUILD_DATE="2020-07-08 01:34:09 +0000"

Last Pass
---------

Timestamp/Logs
--------------
https://files.starlingx.kube.cengn.ca/launchpad/1887755

Test Activity
-------------
Regression Testing

Workaround
----------

[Ghada Khalil (gkhalil)]

The issue here is that there are 2 keystone admin users: one for the host keystone instance and the second for the containerized openstack keystone. However, platform services (ex: nfv-vim) do not distinguish between the two. This is a design gap since the introduction of containerized openstack. More investigation is required to determine the best option forward.

[Lin Shuicheng (shuicheng)]

Hi,
There are two admin, 1 for host platform services, 1 for containerized openstack.
Per your bash history, you executed the openstack cmd in OS_CLOUD environment, it means you changed "containerized openstack"'s admin password. "keyring get CGCS admin" is for host's admin password, that is why it is un-changed.
You could run "openstack user list" in both environment. And you will find both "admin" account with different GUID.

[George Postolache (gpostola)]

Thank you for the clarification, it works as you said

[George Postolache (gpostola)]

isn't it still a problem though if changing the password like i did gets the account locked?

[Andy (andy.wrs)]

Yes it is a problem. But as explained by Shuicheng, it's an issue on containerized openstack admin password change. I think you can open a Launchpad for it.

[Lin Shuicheng (shuicheng)]

It is an issue. The issue is that openstack admin password is changed in keystone, but other openstack services still have the old password in secrets, and cause the lock issue.
Here is the service list which have the admin password:
[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get secrets -n openstack | grep keystone-admin
cinder-keystone-admin Opaque 9 88m
fm-keystone-admin Opaque 9 73m
glance-keystone-admin Opaque 9 90m
heat-keystone-admin Opaque 9 74m
keystone-keystone-admin Opaque 9 92m
neutron-keystone-admin Opaque 9 86m
nova-keystone-admin Opaque 9 86m
placement-keystone-admin Opaque 9 86m

Try to open the secret, we could get password in "OS_PASSWORD:". And use base64 to decode it.

We may need to monitor the password change in keystone, and update the secrets. The keystone is running in container, so it is not easy to do it.
The quick WA for it may disable the lock operation. Keystone just report error in log, but not force the lock operation. To do it, we need remove below setting in keystone container's /etc/keystone/keystone.conf:
"
[security_compliance]
lockout_duration = 1800
lockout_failure_attempts = 5
"

Don't have a good solution for it yet. Any suggestion is welcome.

[Andy (andy.wrs)]

This is what we've done for platform keystone admin user:

https://review.opendev.org/#/c/728492/
https://review.opendev.org/#/c/730834/

We don't want to exempt all users from fail auth lockout. We only exempt admin. This is done by setting a user option by a PATCH call.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/743503

[yong hu (yhu6)]

Root cause was clear, but the scale of changes might be too big for stx.4.0.
So, we move this issue to stx.5.0.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/744823

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/747143

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lin Shuicheng (<email address hidden>) on branch: master
Review: https://review.opendev.org/744823

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lin Shuicheng (<email address hidden>) on branch: master
Review: https://review.opendev.org/743503

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/753971

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/754578

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/754580

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lin Shuicheng (<email address hidden>) on branch: master
Review: https://review.opendev.org/754578

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lin Shuicheng (<email address hidden>) on branch: master
Review: https://review.opendev.org/754580

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/763314

[Lin Shuicheng (shuicheng)]

Issue has been fixed with below CL:
https://review.opendev.org/c/starlingx/openstack-armada-app/+/763314
https://review.opendev.org/c/starlingx/openstack-armada-app/+/747143
https://review.opendev.org/c/starlingx/config/+/753971