tripleo

  1. Bug #1863246
  2. Comment #0

Comment 0 for bug 1863246

Revision history for this message
Marios Andreou (marios-b) wrote : gating container builds with buildah master/train failing - infra related "Could not retrieve mirrorlist"

The tripleo-build-containers-centos-7-buildah and tripleo-build-containers-centos-7-buildah-train are intermittently failing due to slow centos.org mirrors. Failure happens during the container build e.g. [1]

        2020-02-14 08:43:45 | Traceback (most recent call last):
        2020-02-14 08:43:45 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/tripleoclient/command.py", line 32, in run
        2020-02-14 08:43:45 | super(Command, self).run(parsed_args)
        2020-02-14 08:43:45 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
        2020-02-14 08:43:45 | return super(Command, self).run(parsed_args)
        2020-02-14 08:43:45 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/cliff/command.py", line 185, in run
        2020-02-14 08:43:45 | return_code = self.take_action(parsed_args) or 0
        2020-02-14 08:43:45 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/tripleoclient/v1/container_image.py", line 221, in take_action
        2020-02-14 08:43:45 | bb.build_all()
        2020-02-14 08:43:45 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/tripleo_common/image/builder/buildah.py", line 200, in build_all
        2020-02-14 08:43:45 | self.build_all(deps.get(container))
        2020-02-14 08:43:45 | File "/home/zuul/workspace/venv_build/lib/python2.7/site-packages/tripleo_common/image/builder/buildah.py", line 189, in build_all
        2020-02-14 08:43:45 | raise SystemError(job._exception)
        2020-02-14 08:43:45 | SystemError: Unexpected error while running command.
        2020-02-14 08:43:45 | Command: sudo buildah bud --format docker --tls-verify=False --logfile /tmp/kolla-aXmixd/docker/rsyslog/rsyslog/rsyslog-build.log -t 127.0.0.1:8787/tripleotrain/centos-binary-rsyslog:8994186c65a324b172f993eeecb6f00cd0bdbd75_42a57bc6 /tmp/kolla-aXmixd/docker/rsyslog/rsyslog

Digging into the specific failed build log e.g. [2][3] you can see:

        * Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
12: Timeout on http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')

This is not happening on every run - build history at [4][5].

[1] https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_11a/701937/16/check/tripleo-build-containers-centos-7-buildah/11a51b1/logs/build.log
[2] https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_716/706572/21/check/tripleo-build-containers-centos-7-buildah-train/7168bff/logs/buildah-builds/kolla-u8JCUj/docker/openstack-base/openstack-base-build.log
[3] https://bcc26d834e5839076cd5-a219f47d4102f914b241b8f5223ad891.ssl.cf2.rackcdn.com/701937/16/check/tripleo-build-containers-centos-7-buildah-train/652cd89/logs/buildah-builds/kolla-aXmixd/docker/rsyslog/rsyslog/rsyslog-build.log
[4] http://zuul.openstack.org/builds?job_name=tripleo-build-containers-centos-7-buildah
[5] https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-build-containers-centos-7-buildah-train