Comment 17 for bug 1853017

Revision history for this message
yong hu (yhu6) wrote :

The issue was root-caused.
In short, password for "admin" in 2 k8s secrets ("default-registry-key" and "registry-local-secret" ) was not updated after the operator "sysadmin" changed the password for "admin" user by "openstack" client.

Though the updated password in keyring and keystone (:5000), there was never chance to refresh these 2 secrets, and they kept using the default password set in ansible playbook (say. localhost.yml).
So, whenever docker client pulls image and requires authentication via "registry-token-server" which furthers turns to keystone (:5000), old/default password for "admin" triggers to authentication failure.

The attachment #1 is the packet I captured by TCPDUMP when the failures happened. "GopherCloud" inside "registry-token-server/keystone/access.go" failed to get auth from keystone because it was using the default (and obsolete) password "Local.123" (set from Ansible playbook).

The attachment #2 is the code pieces in "~/containers/registry-token-server/src/keystone/access.go" which was using the obsolete password from request (from k8s secret "default-registry-key").

After updating these passwords in 2 secrets above, the authentication went on correctly.