Tripleo upstream jobs are failing undercloud install/standalone deploy with "requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url"

[Ronelle Landy]

This error shows up sporadically when running upstream jobs on various cloud providers:

2020-07-08 17:46:39 | Error detected in auth headers: error invalid_token
2020-07-08 17:46:39 | Re-authenticating: id d125512fa67e216a89bb9fe13182e57088732421, count 4
2020-07-08 17:54:59 | Exception: 429 Client Error: Too Many Requests for url: http://mirror.dfw.rax.opendev.org:8082/v2/tripleou/centos-binary-barbican-api/blobs/sha256:15a3874f2bc6e7228f99587b6d7a6cd95a92d85bc85f118503e734b68b129c68

Example log:
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8e5/739595/2/check/tripleo-ci-centos-8-undercloud-containers/8e53807/logs/undercloud/home/zuul/undercloud_install.log

Creating this LP bug to track these errors and possible changes/retries/waits to prevent this error.

[Damien Ciabrini (dciabrin)]

I think I'm hitting the issue in https://review.opendev.org/#/c/725782/
Specifically in e.g. tripleo-ci-centos-8-scenario001-standalone [1], the job can't pull container images [2]:

requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: http://mirror.iad.rax.opendev.org:8082/v2/tripleomaster/centos-binary-aodh-api/blobs/sha256:5a75d985a954725c2dd79d6816781584791a783dae08336294e46008ad55f6a2
None
429 Client Error: Too Many Requests for url: http://mirror.iad.rax.opendev.org:8082/v2/tripleomaster/centos-binary-aodh-api/blobs/sha256:5a75d985a954725c2dd79d6816781584791a783dae08336294e46008ad55f6a2

Interestingly, all the jobs that target http://mirror.ord.rax.opendev.org:8082/ are running just fine.

[1] https://zuul.opendev.org/t/openstack/build/74d59ccb7d7a4bc6bbccbeb3cb6b2c41
[2] https://f645fd146bf889d4bb5d-aac042a0844b1c7bd58db620c0fb1e04.ssl.cf5.rackcdn.com/725782/30/check/tripleo-ci-centos-8-scenario001-standalone/74d59cc/logs/undercloud/home/zuul/standalone_deploy.log

[Marios Andreou (marios-b)]

i think duplicate of https://bugs.launchpad.net/tripleo/+bug/1889122 there are a few fixes merged for that bug already and it is currently ongoing