assertion: invalid username/password in registry_login_podman.results.0.stderr

[chandan kumar]

All the ovb jobs and rdo component pipelines are failing with following error with RETRY_LIMIT.
https://logserver.rdoproject.org/openstack-component-glance/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-glance-train/b48ae5e/job-output.txt

```
2020-12-09 09:24:37.413689 |
2020-12-09 09:24:37.413971 | TASK [Assert role has worked in (CentOS|RedHat) 8]
2020-12-09 09:24:37.486583 | primary | ERROR
2020-12-09 09:24:37.486847 | primary | {
2020-12-09 09:24:37.486907 | primary | "assertion": "\"invalid username/password\" in registry_login_podman.results.0.stderr",
2020-12-09 09:24:37.486945 | primary | "evaluated_to": false,
2020-12-09 09:24:37.486972 | primary | "msg": "Role failed authentication with ."
2020-12-09 09:24:37.486999 | primary | }
2020-12-09 09:24:37.498127 |
```

Other jobs getting affected:
* https://logserver.rdoproject.org/23/31323/1/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens/cdac3a9/job-output.txt

* https://logserver.rdoproject.org/91/765291/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039/2a471f0/job-output.txt

* https://logserver.rdoproject.org/openstack-component-glance/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-glance-train/b48ae5e/job-output.txt

* https://logserver.rdoproject.org/91/765291/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/e201ad3/job-output.txt

We can view all the failures https://review.rdoproject.org/zuul/builds?result=RETRY_LIMIT

It started happening today. May be due to expiry of docker registry zuul secret.

But it needs to be investigated.

It calls the above failure https://review.rdoproject.org/r/gitweb?p=config.git;a=blob;f=playbooks/tripleo-rdo-base/container-login.yaml#l78

[Marios Andreou (marios-b)]

issue coming from https://github.com/rdo-infra/review.rdoproject.org-config/blob/f0ea75e940b502a957e78322ee51d6cc9dc942a2/playbooks/tripleo-rdo-base/container-login.yaml#L27

apparently there is an issue with ssl certificate and jpenda working on that

12:41 < chkumar|ruck> marios|rover: sshnaidm jpena is working on fixing on an issue with ssl certificate issue

[Marios Andreou (marios-b)]

13:35 < jpena> marios|rover, chkumar|ruck: the RDO registry should be back in action now, can you check the jobs?
13:39 < marios|rover> jpena: not yet
http://dashboard-ci.tripleo.org/d/3-DYSmOGk/jobs-exploration?orgId=1&var-influxdb_filter=pipeline%7C%3D%7Copenstack-check but thanks will keep an eye
13:39 < jpena> marios|rover: it was only fixed ~5 mins ago

[Marios Andreou (marios-b)]

adding some more info for context from irc i thought I already included it:

12:41 < chkumar|ruck> marios|rover: sshnaidm jpena is working on fixing on an issue with ssl certificate issue

12:42 < marios|rover> chkumar|ruck: thank you i was looking at the creds just now wondering if we needed to update it...

12:48 < chkumar|ruck> marios|rover: not sure, yesterday dpawalik renewed the centos cert, jpena knows what is going on, he is taking care of that

12:48 < jpena> marios|rover: creds are ok, what was renewed yesterday is the CBS cert

12:48 < jpena> it's the SSL certificate for trunk.registry.rdoproject.org that is failing atm

[chandan kumar (chkumar246)]

Re opening this bug as we are again seeing it https://logserver.rdoproject.org/96/801296/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/0466946/job-output.txt

```
TASK [Assert role has worked in (CentOS|RedHat) 8]
2021-07-19 12:51:39.596774 | primary | ERROR
2021-07-19 12:51:39.597054 | primary | {
2021-07-19 12:51:39.597094 | primary | "assertion": "\"invalid username/password\" in registry_login_podman.results.0.stderr",
2021-07-19 12:51:39.597146 | primary | "evaluated_to": false,
2021-07-19 12:51:39.597176 | primary | "msg": "Role failed authentication with ."
2021-07-19 12:51:39.597202 | primary | }
```

and

Playbooks used:
https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-rdo-base/container-login.yaml calls https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_podman/tasks/tripleo_podman_login.yml#L31 where it is failing leding to retry_limit on ovb jobs

[chandan kumar (chkumar246)]

https://review.rdoproject.org/r/c/config/+/34600 registry token refreshed.

[chandan kumar (chkumar246)]

https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 jobs are back to green now and above issue is vanished. Closing this bug now.