CI: extremely long times of overcloud deploy in multinode jobs

[Sagi (Sergey) Shnaidman]

Overcloud deploy and undercloud install sometimes takes abnormally long times in multinode jobs.

Overcloud deploy takes about 1 hour 10 min:
http://logs.openstack.org/62/487362/5/check/gate-tripleo-ci-centos-7-scenario004-multinode-oooq-puppet/fd67c98/console.html#_2017-08-08_09_29_25_855991

and undercloud takes about 30 min:
http://logs.openstack.org/62/487362/5/check/gate-tripleo-ci-centos-7-scenario004-multinode-oooq-puppet/fd67c98/console.html#_2017-08-08_08_57_14_978656

It causes timeouts and job failure.
Possibly related to slow infra clouds (acc. to Paul Belanger)

[Emilien Macchi (emilienm)]

We probably need more details on this one...

[Paul Belanger (pabelanger)]

This job ran on infracloud-vanilla, which was at capacity yesterday. Were monitoring the issues now to see if changes to nodepool will help.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/492289

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/492289
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=84e0168e49fce2f3d79559866ebea8f7050eef51
Submitter: Jenkins
Branch: master

commit 84e0168e49fce2f3d79559866ebea8f7050eef51
Author: Alex Schultz <email address hidden>
Date: Wed Aug 9 14:55:03 2017 -0600

    Disable existing repositories in tripleo-ci

    The images that are provided via infra have repositories configured on
    them without the local mirrors configured. This change adds a command to
    disable all existing repositories on the node before setting up the
    defined repos.

    Change-Id: Ie055a95f6d5ca8ca704d8b30407b99f3948dad20
    Related-Bug: #1709327

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/493383

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/494056

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/493383
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=6c45808360757dfda6b2cec344fa439f695f5b10
Submitter: Jenkins
Branch: master

commit 6c45808360757dfda6b2cec344fa439f695f5b10
Author: Steve Baker <email address hidden>
Date: Mon Aug 14 10:48:46 2017 +1200

    Avoid overcloud validate timeout on stack failure

    Currently the wait_for call on the validate stack does not included a
    --fail-match argument. This means that if the validate heat stack goes
    to CREATE_FAILED, wait_for will continue polling until --walltime.
    Related-Bug: #1709327

    Change-Id: I473df0dc267134d472d69ab43459ffbbd872ffa0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/494056
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=5389b4c44b1e2c1dc30848359f8a415481cc86d1
Submitter: Jenkins
Branch: master

commit 5389b4c44b1e2c1dc30848359f8a415481cc86d1
Author: John Fulton <email address hidden>
Date: Tue Aug 15 22:29:54 2017 -0400

    Rename centos repositories created by quickstart

    Quickstart disables the existing CentOS repositories without
    local mirrors provided by infra and creates a similarly named
    mirrored repository to replace it. This change renames those
    mirrored repositories, so they do not conflict with the existing
    CentOS repositories, by prefacing them with "quickstart-".

    Change-Id: I99a9b73eaf1a32193adf0f719f205e004f4e233d
    Related-Bug: #1709327
    Closes-Bug: #1711006