(intermittent) gate buildah container builds master/train failing - infra related "Could not retrieve mirrorlist"

[Marios Andreou]

The tripleo-build-containers-centos-7-buildah and tripleo-build-containers-centos-7-buildah-train are intermittently failing due to slow centos.org mirrors. Failure happens during the container build e.g. [1]

        2020-02-14 08:43:45 | Traceback (most recent call last):
        2020-02-14 08:43:45 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/tripleoclient/command.py", line 32, in run
...
        2020-02-14 08:43:45 | raise SystemError(job._exception)
        2020-02-14 08:43:45 | SystemError: Unexpected error while running command.
        2020-02-14 08:43:45 | Command: sudo buildah bud --format docker --tls-verify=False --logfile /tmp/kolla-aXmixd/docker/rsyslog/rsyslog/rsyslog-build.log -t 127.0.0.1:8787/tripleotrain/centos-binary-rsyslog:8994186c65a324b172f993eeecb6f00cd0bdbd75_42a57bc6 /tmp/kolla-aXmixd/docker/rsyslog/rsyslog

Digging into the specific failed build log e.g. [2][3] you can see:

        * Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
12: Timeout on http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')

This is not happening on every run - build history at [4][5].

[1] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_11a/701937/16/check/tripleo-build-containers-centos-7-buildah/11a51b1/logs/build.log
[2] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_716/706572/21/check/tripleo-build-containers-centos-7-buildah-train/7168bff/logs/buildah-builds/kolla-u8JCUj/docker/openstack-base/openstack-base-build.log
[3] https://bcc26d834e5839076cd5-a219f47d4102f914b241b8f5223ad891.ssl.cf2.rackcdn.com/701937/16/check/tripleo-build-containers-centos-7-buildah-train/652cd89/logs/buildah-builds/kolla-aXmixd/docker/rsyslog/rsyslog/rsyslog-build.log
[4] http://zuul.openstack.org/builds?job_name=tripleo-build-containers-centos-7-buildah
[5] https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-build-containers-centos-7-buildah-train

[Marios Andreou (marios-b)]

i see you moved won't fix - looking at builds it is much greener at least right now there are no failed for either of those 18th Feb