Use RDO Proxy mirror instead of centos mirrors

Bug #1922032 reported by chandan kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Unassigned

Bug Description

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-ironic-python-agent-master/1e47a4d/build.log

```
2021-03-31 08:37:36.725 | dlrn-master-build-deps 2.0 MB/s | 464 kB 00:00
2021-03-31 08:37:37.305 | Advanced Virtualization mirror 0.0 B/s | 0 B 00:00
2021-03-31 08:37:37.306 | Errors during downloading metadata for repository 'advanced-virtualization':
2021-03-31 08:37:37.306 | - Curl error (7): Couldn't connect to server for http://mirror.centos.org/centos/8/virt/x86_64/advanced-virtualization/repodata/repomd.xml [Failed to connect to mirror.centos.org port 80: Connection refused]
2021-03-31 08:37:37.314 | Error: Failed to download metadata for repo 'advanced-virtualization': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
2021-03-31 08:37:37.444 |
```

Many times our job got failed due to failed to download metadata from centos mirrors.

RDO already have mirror: http://mirror.regionone.vexxhost.rdoproject.org/centos/8/

Let's use that one in periodic and avoid these issues.

Revision history for this message
chandan kumar (chkumar246) wrote :

@dpwalik is checking with #centos team to whitelist the rdo mirror.

Revision history for this message
chandan kumar (chkumar246) wrote :

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-ironic-python-agent-master/1e47a4d/yum.repos.d/delorean-master-testing.repo

[advanced-virtualization]
name=Advanced Virtualization mirror
baseurl=http://mirror.centos.org/centos/8/virt/$basearch/advanced-virtualization/
gpgcheck=0
enabled=1
module_hotfixes=1

[centos-rabbitmq-38]
name=Messaging RabbitMQ
baseurl=http://mirror.centos.org/centos/8/messaging/$basearch/rabbitmq-38
gpgcheck=0
enabled=1
module_hotfixes=1
skip_if_unavailable=1

[centos-opstools]
name=opstools
baseurl=http://mirror.centos.org/centos/8/opstools/$basearch/collectd-5/
gpgcheck=0
enabled=1
module_hotfixes=1

[centos-nfv-ovs]
name=NFV SIG OpenvSwitch
baseurl=http://mirror.centos.org/centos/8/nfv/$basearch/openvswitch-2/
gpgcheck=0
enabled=1
module_hotfixes=1

Revision history for this message
chandan kumar (chkumar246) wrote :

https://review.opendev.org/c/openstack/tripleo-ci/+/784057 proposed a review to handle it in tripleo-ci job side.

Revision history for this message
Alan Pevec (apevec) wrote :

tl;dr from #centos-devel

> people should use mirrorlist and not point to mirror.c.o
> donated machines come and go

<ykarel> Arrfab, 173.214.244.24 is one of the failing ip
<Arrfab> ykarel: that one was removed and is *not* in DNS .. so check your own resolver that doesn't obey TTL either
<ykarel> Arrfab, ok will check, btw when it was removed?
<Arrfab> yesterday
<ykarel> okk so it matches our failure times

<Arrfab> you can query pdns{1..3}.centos.org for mirror.centos.org and none is answering that old/removed ip. so wondering if your resolver doesn't obey TTL and keep longer in cache

> in /etc/resolv.conf the system has 1.1.1.1, 8.8.8.8
> you can query with "dig @pdns1.centos.org mirror.centos.org +short"
> and same for pdns2/pdns3

Revision history for this message
Alan Pevec (apevec) wrote :

Reported as https://pagure.io/centos-infra/issue/270
but real fix is to change jobs to use local AFS mirrors (patch in comment 3)

We will also need to modify delorean-deps.repo to use mirrorlist and then adjust role which set AFS mirrors, simple s|http://mirror.centos.org/centos|$NODEPOOL_CENTOS_MIRROR|g will then not work!

Changed in tripleo:
milestone: wallaby-rc1 → xena-1
Revision history for this message
Marios Andreou (marios-b) wrote :

adding a note here as there is a relevant issue happening with the dlrn builds https://bc407b42c3b88557b3de-7f9926fbce5c6ae120812541a1c87ecd.ssl.cf1.rackcdn.com/795788/1/check/tripleo-ci-centos-8-content-provider/aa18ad1/logs/delorean_logs/component/tripleo/7d/ea/7dea661f697eeda50ccd53ceda49bdbd415177f8_dev/root.log.txt.gz

"DEBUG util.py:596: Errors during downloading metadata for repository 'advanced-virtualization':
DEBUG util.py:596: - Curl error (28): Timeout was reached for http://mirror.centos.org/centos/8-stream/virt/x86_64/advancedvirt-common/repodata/repomd.xml [Connection timed out after 30000 milliseconds]
DEBUG util.py:596: Error: Failed to download metadata for repo 'advanced-virtualization': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

we can use this bug for digging into that as well (use local mirrors for the delorean builds) but it has a low priority at the moment as it seems this was a one-off otherwise the job looks OK @ https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-content-provider today

Changed in tripleo:
milestone: xena-1 → xena-2
Changed in tripleo:
milestone: xena-2 → xena-3
Revision history for this message
Alan Pevec (apevec) wrote :

afaik we moved all CI jobs to use local AFS mirrors

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.