download_mirrors.sh: cengn urls are not validated before substitution

[Scott Little]

Brief Description
-----------------
download_mirrors.sh by default substitutes urls with their cengn equivalent. If the user is trying to introduce a new repo, a CENGN mirror of the repo will not yet exist. The substituted url will be added to the yum configuration despite being invalid. All subsequent yumdownload attempts will fail.

Severity
--------
Provide the severity of the defect.
Major: Very difficult test changes or additions to the yum repos.

Steps to Reproduce
------------------
1) Add a new repo under stx-tools/centos-mirror-tools/yum.repos.d ... e.g.
    vault.centos.org/centos/7.6.1810/cloud/Source/openstack-stein
2) download_mirrors.sh

Expected Behavior
------------------
download_mirror can still download packages

Actual Behavior
----------------
All rpm repo package downloads fail.

Reproducibility
---------------
100%

System Configuration
--------------------
<N/A>

Branch/Pull Time/Commit
-----------------------
master, 2019-04-12

Last Pass
---------
N/A

Timestamp/Logs
--------------
...
http://mirror.starlingx.cengn.ca:80/mirror/centos/centos/vault.centos.org/centos/7.6.1810/cloud/Source/openstack-stein/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article

https://access.redhat.com/articles/1320623

If above article doesn't help to resolve this issue please create a bug on https://bugs.centos.org/

 One of the configured repositories failed (CENGN_CentOS-7 - OpenStack Stein - Source),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Run the command with the repository temporarily disabled
            yum --disablerepo=CENGN_centos-openstack-Stein-source ...

     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:

            yum-config-manager --disable CENGN_centos-openstack-Stein-source
        or
            subscription-manager repos --disable=CENGN_centos-openstack-Stein-source

     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=CENGN_centos-openstack-Stein-source.skip_if_unavailable=true
failure: repodata/repomd.xml from CENGN_centos-openstack-Stein-source: [Errno 256] No more mirrors to try.
...

Test Activity
-------------
Build

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/653057

[Ghada Khalil (gkhalil)]

Marking as release gating; high priority as issue is impacting the submission of story board:
https://storyboard.openstack.org/#!/story/2004765

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/653057
Committed: https://git.openstack.org/cgit/openstack/stx-tools/commit/?id=06171f9f81a9df3fc296acd615a995da15b82330
Submitter: Zuul
Branch: master

commit 06171f9f81a9df3fc296acd615a995da15b82330
Author: Scott Little <email address hidden>
Date: Mon Apr 15 16:54:33 2019 -0400

    download_mirrors.sh: cengn urls are not validated before substitution

    Problem
    =======
    download_mirrors.sh by default substitutes urls with their cengn
    equivalent. If the user is trying to introduce a new repo, a CENGN
    mirror of the repo will not yet exist. The substituted url will be added
    to the yum configuration despite being invalid. All subsequent
    yumdownload attempts will fail with a 404 on the repodata of the
    non-existent cengn repo.

    Solution
    ========
       Only substitute yum repo urls with the cengn equivalent if
    cengn actually has the repo.

    Closes-bug: 1824877
    Change-Id: Ifa262212d67e096cc29131e5738aec0365ed9893
    Signed-off-by: Scott Little <email address hidden>