Comment 46 for bug 1853017

Revision history for this message
Lin Shuicheng (shuicheng) wrote :

Hi Peng,
There is controller-1 only in the log tarball, controller-0 is missed.
Could you share me the detail step to reproduce the issue?
When do you change the password? And what operation before and after the password change?
From the log, the failure is still due to authentication failure with registry-token-server. But I don't know where the access request from.
I could find when password is changed, secrets are updated also. And no application is in applying stage.
I need to reproduce the issue to check where does the registry-token-server access come from.

Here is some log from controller-1:
Password change cmd at 2:40:58:
2020-02-23T02:40:58.000 controller-1 -sh: info HISTORY: PID=241256 UID=42425 openstack --os-username 'admin' --os-password '!Li69nux*9' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne user set --password xxxxxx admin

Secrets update at 2:41:01:
sysinv 2020-02-23 02:41:01.613 238507 INFO sysinv.conductor.kube_app [-] Secret registry-local-secret under Namespace kube-system is updated
sysinv 2020-02-23 02:41:01.645 238507 INFO sysinv.conductor.kube_app [-] Secret default-registry-key under Namespace kube-system is updated

Authentication failure at 2:41:04:
./var/log/daemon.log:38427:2020-02-23T02:41:04.547 controller-1 registry-token-server[235987]: info time="2020-02-23T02:41:04Z" level=error msg="error authenticating user \"admin\": Authentication failed" go.version=go1.12.10 http.request.host="128.224.151.227:9002" http.request.id=46c3b222-24ca-46bb-936c-0ef08fbf5141 http.request.method=GET http.request.remoteaddr="192.168.204.3:51416" http.request.uri="/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fmultus%3Apush%2Cpull&service=192.168.204.1%3A9001" http.request.useragent="docker/18.09.6 go/go1.10.8 git-commit/481bc77 kernel/3.10.0-1062.1.2.el7.2.tis.x86_64 os/linux arch/amd64 UpstreamClient(docker-sdk-python/3.3.0)" instance.id=46661299-76ed-4229-8aa6-45ac24c3f1c6

Then Account lock happen at 2:41:06 after 5 time invalid authentication:
2020-02-23 02:41:06.091 239370 WARNING keystone.server.flask.application [req-cc73c0ca-5d97-4ae0-afa4-6159b677b0bb - - - - -] Authorization failed. The account is locked for user: c52f573e07d24a37b9b5627a8c82756d. from 192.168.204.3: AccountLocked: The account is locked for user: c52f573e07d24a37b9b5627a8c82756d.