The issue was root-caused.
In short, password for "admin" in 2 k8s secrets ("default-registry-key" and "registry-local-secret" ) was not updated after the operator "sysadmin" changed the password for "admin" user by "openstack" client.
Though the updated password in keyring and keystone (:5000), there was never chance to refresh these 2 secrets, and they kept using the default password set in ansible playbook (say. localhost.yml).
So, whenever docker client pulls image and requires authentication via "registry-token-server" which furthers turns to keystone (:5000), old/default password for "admin" triggers to authentication failure.
The attachment #1 is the packet I captured by TCPDUMP when the failures happened. "GopherCloud" inside "registry-token-server/keystone/access.go" failed to get auth from keystone because it was using the default (and obsolete) password "Local.123" (set from Ansible playbook).
The attachment #2 is the code pieces in "~/containers/registry-token-server/src/keystone/access.go" which was using the obsolete password from request (from k8s secret "default-registry-key").
After updating these passwords in 2 secrets above, the authentication went on correctly.
The issue was root-caused. registry- key" and "registry- local-secret" ) was not updated after the operator "sysadmin" changed the password for "admin" user by "openstack" client.
In short, password for "admin" in 2 k8s secrets ("default-
Though the updated password in keyring and keystone (:5000), there was never chance to refresh these 2 secrets, and they kept using the default password set in ansible playbook (say. localhost.yml). token-server" which furthers turns to keystone (:5000), old/default password for "admin" triggers to authentication failure.
So, whenever docker client pulls image and requires authentication via "registry-
The attachment #1 is the packet I captured by TCPDUMP when the failures happened. "GopherCloud" inside "registry- token-server/ keystone/ access. go" failed to get auth from keystone because it was using the default (and obsolete) password "Local.123" (set from Ansible playbook).
The attachment #2 is the code pieces in "~/containers/ registry- token-server/ src/keystone/ access. go" which was using the obsolete password from request (from k8s secret "default- registry- key").
After updating these passwords in 2 secrets above, the authentication went on correctly.