Hi Peng,
There is controller-1 only in the log tarball, controller-0 is missed.
Could you share me the detail step to reproduce the issue?
When do you change the password? And what operation before and after the password change?
From the log, the failure is still due to authentication failure with registry-token-server. But I don't know where the access request from.
I could find when password is changed, secrets are updated also. And no application is in applying stage.
I need to reproduce the issue to check where does the registry-token-server access come from.
Here is some log from controller-1:
Password change cmd at 2:40:58:
2020-02-23T02:40:58.000 controller-1 -sh: info HISTORY: PID=241256 UID=42425 openstack --os-username 'admin' --os-password '!Li69nux*9' --os-project-name admin --os-auth-url http://192.168.204.1:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne user set --password xxxxxx admin
Secrets update at 2:41:01:
sysinv 2020-02-23 02:41:01.613 238507 INFO sysinv.conductor.kube_app [-] Secret registry-local-secret under Namespace kube-system is updated
sysinv 2020-02-23 02:41:01.645 238507 INFO sysinv.conductor.kube_app [-] Secret default-registry-key under Namespace kube-system is updated
Authentication failure at 2:41:04:
./var/log/daemon.log:38427:2020-02-23T02:41:04.547 controller-1 registry-token-server[235987]: info time="2020-02-23T02:41:04Z" level=error msg="error authenticating user \"admin\": Authentication failed" go.version=go1.12.10 http.request.host="128.224.151.227:9002" http.request.id=46c3b222-24ca-46bb-936c-0ef08fbf5141 http.request.method=GET http.request.remoteaddr="192.168.204.3:51416" http.request.uri="/token/?account=admin&scope=repository%3Adocker.io%2Fstarlingx%2Fmultus%3Apush%2Cpull&service=192.168.204.1%3A9001" http.request.useragent="docker/18.09.6 go/go1.10.8 git-commit/481bc77 kernel/3.10.0-1062.1.2.el7.2.tis.x86_64 os/linux arch/amd64 UpstreamClient(docker-sdk-python/3.3.0)" instance.id=46661299-76ed-4229-8aa6-45ac24c3f1c6
Then Account lock happen at 2:41:06 after 5 time invalid authentication:
2020-02-23 02:41:06.091 239370 WARNING keystone.server.flask.application [req-cc73c0ca-5d97-4ae0-afa4-6159b677b0bb - - - - -] Authorization failed. The account is locked for user: c52f573e07d24a37b9b5627a8c82756d. from 192.168.204.3: AccountLocked: The account is locked for user: c52f573e07d24a37b9b5627a8c82756d.
Hi Peng, token-server. But I don't know where the access request from. token-server access come from.
There is controller-1 only in the log tarball, controller-0 is missed.
Could you share me the detail step to reproduce the issue?
When do you change the password? And what operation before and after the password change?
From the log, the failure is still due to authentication failure with registry-
I could find when password is changed, secrets are updated also. And no application is in applying stage.
I need to reproduce the issue to check where does the registry-
Here is some log from controller-1: 23T02:40: 58.000 controller-1 -sh: info HISTORY: PID=241256 UID=42425 openstack --os-username 'admin' --os-password '!Li69nux*9' --os-project-name admin --os-auth-url http:// 192.168. 204.1:5000/ v3 --os-user- domain- name Default --os-project- domain- name Default --os-identity- api-version 3 --os-interface internal --os-region-name RegionOne user set --password xxxxxx admin
Password change cmd at 2:40:58:
2020-02-
Secrets update at 2:41:01: conductor. kube_app [-] Secret registry- local-secret under Namespace kube-system is updated conductor. kube_app [-] Secret default- registry- key under Namespace kube-system is updated
sysinv 2020-02-23 02:41:01.613 238507 INFO sysinv.
sysinv 2020-02-23 02:41:01.645 238507 INFO sysinv.
Authentication failure at 2:41:04: daemon. log:38427: 2020-02- 23T02:41: 04.547 controller-1 registry- token-server[ 235987] : info time="2020- 02-23T02: 41:04Z" level=error msg="error authenticating user \"admin\": Authentication failed" go.version= go1.12. 10 http.request. host="128. 224.151. 227:9002" http.request. id=46c3b222- 24ca-46bb- 936c-0ef08fbf51 41 http.request. method= GET http.request. remoteaddr= "192.168. 204.3:51416" http.request. uri="/token/ ?account= admin&scope= repository% 3Adocker. io%2Fstarlingx% 2Fmultus% 3Apush% 2Cpull& service= 192.168. 204.1%3A9001" http.request. useragent= "docker/ 18.09.6 go/go1.10.8 git-commit/481bc77 kernel/ 3.10.0- 1062.1. 2.el7.2. tis.x86_ 64 os/linux arch/amd64 UpstreamClient( docker- sdk-python/ 3.3.0)" instance. id=46661299- 76ed-4229- 8aa6-45ac24c3f1 c6
./var/log/
Then Account lock happen at 2:41:06 after 5 time invalid authentication: server. flask.applicati on [req-cc73c0ca- 5d97-4ae0- afa4-6159b677b0 bb - - - - -] Authorization failed. The account is locked for user: c52f573e07d24a3 7b9b5627a8c8275 6d. from 192.168.204.3: AccountLocked: The account is locked for user: c52f573e07d24a3 7b9b5627a8c8275 6d.
2020-02-23 02:41:06.091 239370 WARNING keystone.