Comment 23 for bug 1853017

As mentioned, local registry key was not updated after admin's password was changed.
In this case, whoever tried to pull docker image with "imagePullSecrets" would trigger the authentication failure.

In the attached log ~/var/log/keystone/keystone-all.log, there were indeed "subcloud" related error messages, but not sure if they were the consequence of authentication failures (and user account locked) or other causes partially.

@Yang and Peng, while we are working on the fixing patch, if you want, you can take following steps to update k8s secrets for local registry: default-registry-key and registry-local-secret.

#1. list out secrets for local registry.

kubectl -n kube-system get secrets | grep registry

#2. for encode your new user and password, by the cmd below, for example, my new password is !Li69nux*9

echo -n 'admin:!Li69nux*9' | base64

#3. for updating default-registry-key, encode the whole auth data (json format), here "YWRtaW46IUxpNjludXgqOQ==" is the output from step #2 above:

echo -n '{"auths": {"registry.local:9001": {"auth": "YWRtaW46IUxpNjludXgqOQ=="}}}' | base64

#4.Use step#3 encoded auth_data to replace value of ".dockerconfigjson:"in "default-registry-key": eyJhdXRocyI6IHsicmVnaXN0cnkubG9jYWw6OTAwMSI6IHsiYXV0aCI6ICJZV1J0YVc0NklVeHBOamx1ZFhncU9RPT0ifX19

kubectl -n kube-system edit secret default-registry-key

#5.for updating registry-local-secret, encode the whole auth data (json format), here "YWRtaW46IUxpNjludXgqOQ==" is the output from step #2 above:

echo -n '{"auths":{"registry.local:9001":{"username":"admin","password":"!Li69nux*9","auth":"YWRtaW46IUxpNjludXgqOQ=="}}}' | base64

#4.Use step5 encoded auth_data to replace value of ".dockerconfigjson:"in "registry-local-secret": eyJhdXRocyI6eyJyZWdpc3RyeS5sb2NhbDo5MDAxIjp7InVzZXJuYW1lIjoiYWRtaW4iLCJwYXNzd29yZCI6IiFMaTY5bnV4KjkiLCJhdXRoIjoiWVdSdGFXNDZJVXhwTmpsdWRYZ3FPUT09In19fQ==

kubectl -n kube-system edit secret registry-local-secret