CENGN build container failure on golang

[Scott Little]

Brief Description
-----------------
Intermittent build failure seen by CENGN when constructing the build container.

Failed to yum install golang.

Error is 404 on ftp.iij.ad.jp/pub/linux/centos/7/rt/x86_64.

Why are we even looking for this repo at this stage?

Severity
--------
Major

Steps to Reproduce
------------------
docker build --build-arg MYUID=18876 --build-arg MYUNAME=jenkins --ulimit core=0 --network host --tag jenkins:master-20190617T233000Z-builder-image --file ../Dockerfile .

Expected Behavior
------------------
Successful creation of a docker image in which to run the build

Actual Behavior
----------------
Failed to construct a docker image in which to run the build

Reproducibility
---------------
Intermittent. 15%

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
20190619T013000Z

Last Pass
---------
20190617T233000Z

Timestamp/Logs
--------------
...

Step 36/37 : RUN yum install -y golang; mkdir -p ${GOPATH}/bin; curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
---> Running in fe6ae2f54414
Loaded plugins: fastestmirror, ovl
http://ftp.iij.ad.jp/pub/linux/centos/7/rt/x86_64/repodata/17896eac4e64b9ab00f84584b187ed6572996a0988e62d772b37124370704cac-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article

https://access.redhat.com/articles/1320623

If above article doesn't help to resolve this issue please create a bug on https://bugs.centos.org/

Loading mirror speeds from cached hostfile
* base: centos.mirror.netelligent.ca
* extras: centos.mirror.netelligent.ca
* updates: centos.mirror.netelligent.ca
http://ftp.iij.ad.jp/pub/linux/centos/7/rt/x86_64/repodata/17896eac4e64b9ab00f84584b187ed6572996a0988e62d772b37124370704cac-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
http://ftp.iij.ad.jp/pub/linux/centos/7/rt/x86_64/repodata/17896eac4e64b9ab00f84584b187ed6572996a0988e62d772b37124370704cac-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.

One of the configured repositories failed (Starlingx-ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64 - ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:

1. Contact the upstream for the repository and get them to fix the problem.

2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).

3. Run the command with the repository temporarily disabled
yum --disablerepo=Starlingx-ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64 ...

4. Disable the repository permanently, so yum won't use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage:

yum-config-manager --disable Starlingx-ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64
or
subscription-manager repos --disable=Starlingx-ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64

5. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:

yum-config-manager --save --setopt=Starlingx-ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64.skip_if_unavailable=true

failure: repodata/17896eac4e64b9ab00f84584b187ed6572996a0988e62d772b37124370704cac-primary.sqlite.bz2 from Starlingx-ftp.iij.ad.jp_pub_linux_centos_7_rt_x86_64: [Errno 256] No more mirrors to try.
http://ftp.iij.ad.jp/pub/linux/centos/7/rt/x86_64/repodata/17896eac4e64b9ab00f84584b187ed6572996a0988e62d772b37124370704cac-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found

Test Activity
-------------
Build

[Scott Little (slittle1)]

In the Dockerfile, we are trying to run....

   RUN yum install -y golang

after we install StarlingX repos into /etc/yum.repos.d/

    COPY centos-mirror-tools/yum.repos.d/* /etc/yum.repos.d/

It should be before!

A second question is why do we install StarlingX repos into /etc/yum.repos.d/ at all?
Doesn't yumdownloader support --config flag?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/666393

[Ghada Khalil (gkhalil)]

Marking as stx.2.0 release gating as this results in build failures.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/666393
Committed: https://git.openstack.org/cgit/starlingx/tools/commit/?id=67e2d3e4c025501caf413fd3b449b8d70a4eeb04
Submitter: Zuul
Branch: master

commit 67e2d3e4c025501caf413fd3b449b8d70a4eeb04
Author: Scott Little <email address hidden>
Date: Wed Jun 19 12:34:50 2019 -0400

    Dockerfile: relocate "yum install golang"

    Dockerfile command "RUN yum install -y golang" failed with a
    404 error on ftp.iij.ad.jp/pub/linux/centos/7/rt/x86_64.

    In the Dockerfile, we are trying to run....
       RUN yum install -y golang
    after we install StarlingX repos into /etc/yum.repos.d/ with
       COPY centos-mirror-tools/yum.repos.d/* /etc/yum.repos.d/

    All 'yum install' commands must preceed any modifications to
    /etc/yum.repos.d in support of StarlingX package downloading!

    I'll also add 'skip_if_unavailable' to all StarlingX repos
    in case other build scripts introduce a 'yum install'.

    NOTE: Package downloading itself will try CENGN's mirror in
    addition to the primary site. So unavailability of a single
    site isn't fatal in that case.

    Change-Id: If0220d3d5742810b0927f8e55e0093f0f0172c59
    Closes-Bug: 1833444
    Signed-off-by: Scott Little <email address hidden>