tripleo gate jobs are failing to pull containers when running on ovh provider with "UNAUTHORIZED" error

[Ronelle Landy]

We have seen up to 30 gate failures in the tripleo queue in a period of 24 hours. Many of these failures show a variation of the following error:

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"tripleomaster/centos-binary-nova-compute-ironic","Action":"pull"}]}]}

as show in the log file:

https://logs.opendev.org/63/674363/1/gate/tripleo-ci-centos-7-undercloud-containers/9285c4c/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz?level=ERROR

The following link in logstash shows the frequency of the error:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Name%5C%5C%5C%22%3A%5C%5C%5C%22tripleomaster%2Fcentos-binary%5C%22%20AND%20tags%3A%5C%22console%5C%22

(clicking the node_provider checkbox shows the provider where the nodes display this error).

Noting that these jobs pass regularly on other providers, we suspect a setup/infra issue here in on ovh.

[Ronelle Landy (rlandy)]

Related review for tracking error:
https://review.opendev.org/675453 Add query for UNAUTHORIZED error when pulling containers

[Ronelle Landy (rlandy)]

See also: <cloudnull> and asked for us to work in a re-auth on 401 which I did here - https://review.opendev.org/#/c/674672/

[Matt Riedemann (mriedem)]

Based on http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Action%5C%5C%5C%22%3A%5C%5C%5C%22pull%5C%22%20AND%20%20%20message%3A%5C%22ERROR%20%2Fvar%2Flog%2Ftripleo-container-image-prepare.log%5C%22%20AND%20%20%20tags%3A%5C%22console%5C%22&from=7d this started around August 5 and is definitely hitting on OVH providers.

[Ronelle Landy (rlandy)]

See related bug: https://bugs.launchpad.net/tripleo/+bug/1837388

[chandan kumar (chkumar246)]

As per the #openstack-infra conversation, http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-08-18.log.html#t2019-08-18T16:38:53

CentOS image is working fine.

[chandan kumar (chkumar246)]

http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-08-19.log.html#t2019-08-19T10:55:50
There is not an issue at FN with centos. I posted links showing that is was functional yesterday.

[Alex Schultz (alex-schultz)]

https://bugs.launchpad.net/tripleo/+bug/1840973 has been identified as an issue where we end up with 0 byte layers on the local registry

[wes hayutin (weshayutin)]

elastic-recheck is no longer showing this as an issue, closing