TripleO gate jobs run on limestone nodes are failing "to retrieve repo file from https://trunk.rdoproject.org/centos7-<release>/current-tripleo/delorean.repo" after 10 retries

[Ronelle Landy]

Gate jobs should remain passing at a a high rate. Lately we have seen a number of gate failures when jobs are run on linestone nodes with the following error:

RuntimeError: Failed to retrieve repo file from https://trunk.rdoproject.org/centos7-master/current-tripleo/delorean.repo after 10 retries.

Log links are included below:

    https://logs.opendev.org/70/673270/32/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/8217483/job-output.txt.gz#_2019-08-12_08_57_57_105987

    https://logs.opendev.org/55/673955/1/gate/tripleo-ci-centos-7-undercloud-oooq/e0f11e3/job-output.txt.gz#_2019-08-12_11_56_00_969351

    https://logs.opendev.org/28/674228/12/check/tripleo-ci-centos-7-containers-multinode-pike/4129a9d/job-output.txt.gz#_2019-08-12_08_08_03_349236

We have also seen this error occur on other release jobs, including Pike.

[Ronelle Landy (rlandy)]

Related elastic-recheck review:
https://review.opendev.org/675898

[chandan kumar (chkumar246)]

Here is the more detailed info from #openstack-infra channel http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2019-08-08.log.html#t2019-08-08T13:44:07

[Javier Peña (jpena-c)]

The IPv6 issue is complicated. We have no IPv6 support where trunk.rdoproject.org is running, so we cannot enable it for now (unless we redeploy elsewhere, which is a complex issue).

Just wondering, would it be possible to get the delorean.repo file using the AFS mirror? From a NAT standpoint, all connections would come from it, so it might work better.

[chandan kumar (chkumar246)]

Sounds like a good plan about shipping using AFS mirror. Does it will cause a lag in base_url in dlrn.repo which is fetched from trunk.rdoproject.org?

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/676229

[chandan kumar (chkumar246)]

Fixed by https://review.opendev.org/#/c/677088/ review

[chandan kumar (chkumar246)]

https://review.opendev.org/#/c/677077 also fixes it.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ronelle Landy (<email address hidden>) on branch: master
Review: https://review.opendev.org/676229