openstack app's admin account gets locked after change password

Bug #1887755 reported by George Postolache
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Lin Shuicheng

Bug Description

Brief Description
-----------------
After changing the password the keyring is still showing the old password. For a short while openstack commands work using the new password and then the account gets locked.

1853017 similar to this defect except that "keyring get CGCS admin" shows the old password

Severity
--------
Provide the severity of the defect.
Major

Steps to Reproduce
------------------
1. openstack user set --password "N3wpassword*" admin
2. keyring get CGCS admin (is still showing the old password)
3. edit clouds.yml with the new password
4. export OS_CLOUD=openstack_helm
5. openstack network list (will execute normaly for a short while and then The account is locked for user: c8aab57d3ffa447283918b5088ea47d4. (HTTP 401) (Request-ID: req-b3e81f4a-df41-4a84-8498-4f940b048ca1))

Expected Behavior
------------------
keyring updated, commands working with the new password and account not getting locked

Actual Behavior
----------------
keyring is not updated and account is getting locked

Reproducibility
---------------
allways

System Configuration
--------------------
Multi-node system

Branch/Pull Time/Commit
-----------------------
controller-0:~$ cat /etc/build.info
###
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="20.06"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20200708T013409Z"

JOB="STX_build_layer_flock_master_master"
<email address hidden>"
BUILD_NUMBER="165"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2020-07-08 01:34:09 +0000"

FLOCK_OS="centos"
FLOCK_JOB="STX_build_layer_flock_master_master"
<email address hidden>"
FLOCK_BUILD_NUMBER="165"
FLOCK_BUILD_HOST="starlingx_mirror"
FLOCK_BUILD_DATE="2020-07-08 01:34:09 +0000"

Last Pass
---------

Timestamp/Logs
--------------
https://files.starlingx.kube.cengn.ca/launchpad/1887755

Test Activity
-------------
Regression Testing

Workaround
----------

description: updated
Ghada Khalil (gkhalil)
tags: added: stx.distro.openstack
Revision history for this message
Ghada Khalil (gkhalil) wrote :

The issue here is that there are 2 keystone admin users: one for the host keystone instance and the second for the containerized openstack keystone. However, platform services (ex: nfv-vim) do not distinguish between the two. This is a design gap since the introduction of containerized openstack. More investigation is required to determine the best option forward.

yong hu (yhu6)
Changed in starlingx:
assignee: nobody → Lin Shuicheng (shuicheng)
yong hu (yhu6)
Changed in starlingx:
importance: Undecided → Medium
Revision history for this message
Lin Shuicheng (shuicheng) wrote :

Hi,
There are two admin, 1 for host platform services, 1 for containerized openstack.
Per your bash history, you executed the openstack cmd in OS_CLOUD environment, it means you changed "containerized openstack"'s admin password. "keyring get CGCS admin" is for host's admin password, that is why it is un-changed.
You could run "openstack user list" in both environment. And you will find both "admin" account with different GUID.

Revision history for this message
George Postolache (gpostola) wrote :

Thank you for the clarification, it works as you said

Revision history for this message
George Postolache (gpostola) wrote :

isn't it still a problem though if changing the password like i did gets the account locked?

Revision history for this message
Andy (andy.wrs) wrote :

Yes it is a problem. But as explained by Shuicheng, it's an issue on containerized openstack admin password change. I think you can open a Launchpad for it.

Ghada Khalil (gkhalil)
tags: added: stx.4.0
Revision history for this message
Lin Shuicheng (shuicheng) wrote :

It is an issue. The issue is that openstack admin password is changed in keystone, but other openstack services still have the old password in secrets, and cause the lock issue.
Here is the service list which have the admin password:
[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get secrets -n openstack | grep keystone-admin
cinder-keystone-admin Opaque 9 88m
fm-keystone-admin Opaque 9 73m
glance-keystone-admin Opaque 9 90m
heat-keystone-admin Opaque 9 74m
keystone-keystone-admin Opaque 9 92m
neutron-keystone-admin Opaque 9 86m
nova-keystone-admin Opaque 9 86m
placement-keystone-admin Opaque 9 86m

Try to open the secret, we could get password in "OS_PASSWORD:". And use base64 to decode it.

We may need to monitor the password change in keystone, and update the secrets. The keystone is running in container, so it is not easy to do it.
The quick WA for it may disable the lock operation. Keystone just report error in log, but not force the lock operation. To do it, we need remove below setting in keystone container's /etc/keystone/keystone.conf:
"
[security_compliance]
lockout_duration = 1800
lockout_failure_attempts = 5
"

Don't have a good solution for it yet. Any suggestion is welcome.

Revision history for this message
Andy (andy.wrs) wrote :

This is what we've done for platform keystone admin user:

https://review.opendev.org/#/c/728492/
https://review.opendev.org/#/c/730834/

We don't want to exempt all users from fail auth lockout. We only exempt admin. This is done by setting a user option by a PATCH call.

yong hu (yhu6)
Changed in starlingx:
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.opendev.org/743503

Changed in starlingx:
status: Confirmed → In Progress
Revision history for this message
yong hu (yhu6) wrote : Re: keyring password not updated and account gets locked

Root cause was clear, but the scale of changes might be too big for stx.4.0.
So, we move this issue to stx.5.0.

tags: added: stx.5.0
removed: stx.4.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (master)

Fix proposed to branch: master
Review: https://review.opendev.org/744823

summary: - keyring password not updated and account gets locked
+ openstack app's admin account gets locked after change password
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-armada-app (master)

Fix proposed to branch: master
Review: https://review.opendev.org/747143

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on stx-puppet (master)

Change abandoned by Lin Shuicheng (<email address hidden>) on branch: master
Review: https://review.opendev.org/744823

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on config (master)

Change abandoned by Lin Shuicheng (<email address hidden>) on branch: master
Review: https://review.opendev.org/743503

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.opendev.org/753971

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nfv (master)

Fix proposed to branch: master
Review: https://review.opendev.org/754578

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to utilities (master)

Fix proposed to branch: master
Review: https://review.opendev.org/754580

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nfv (master)

Change abandoned by Lin Shuicheng (<email address hidden>) on branch: master
Review: https://review.opendev.org/754578

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on utilities (master)

Change abandoned by Lin Shuicheng (<email address hidden>) on branch: master
Review: https://review.opendev.org/754580

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-armada-app (master)

Fix proposed to branch: master
Review: https://review.opendev.org/763314

Revision history for this message
Lin Shuicheng (shuicheng) wrote :
Changed in starlingx:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers