Comment 1 for bug 1869842

The above ovb job failure seems related to mirror issue.

- Issue faced during overcloud image modify[0].
- Repo setup script was executed but some task failed[1] during repo setup.
- Task[2] (Install mock package) didn't finished
- Future tasks for disabling epel and removing epel didn't ran.
- Later puppet hiera from epel got installed and caused that NTP Error[3], because block[4] was not executed due to puppet error so it reported ntp failure.

Expectations/ Possible Improvements:-

* It was expected that repo_setup.sh should fail if it failed to install some packages and all-nodes.sh[3] shouldn't have been executed.

* epel repo removal should not be skipped.

* Using infra mirrors instead of fedora mirror should improve the situation.

[0] https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/modify-image/tasks/manual.yml#L58-L59

[1]

https://logserver.rdoproject.org/98/714598/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/08be6c9/logs/undercloud/home/zuul/repo_setup.sh.1585621037.log.txt.gz

 ~~~
 2020-03-31 02:18:06 | Error: Error downloading packages:
 2020-03-31 02:18:06 | Status code: 503 for https://mirrors.fedoraproject.org/metalink?repo=epel-8&arch=x86_64&infra=genclo&content=cento
 ~~~

More details here:-

* https://logserver.rdoproject.org/98/714598/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/08be6c9/logs/overcloud-controller-0/var/log/dnf.log.txt.gz

[2] https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-8/master.yml#L39

[3] https://logserver.rdoproject.org/98/714598/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/08be6c9/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

~~~
2020-03-31 03:32:38 | fatal: [overcloud-controller-0]: FAILED! => changed=true
2020-03-31 03:32:38 | msg: non-zero return code
2020-03-31 03:32:38 | rc: 1
2020-03-31 03:32:38 | stderr: |-
2020-03-31 03:32:38 | Shared connection to 192.168.24.8 closed.
2020-03-31 03:32:38 | stderr_lines: <omitted>
2020-03-31 03:32:38 | stdout: |-
2020-03-31 03:32:38 | Trying to ping default gateway 10.0.0.1...Ping to 10.0.0.1 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Trying to ping 192.168.24.8 for local network 192.168.24.0/24.
2020-03-31 03:32:38 | Ping to 192.168.24.8 succeeded.
2020-03-31 03:32:38 | SUCCESS
2020-03-31 03:32:38 | Failed to start Hiera: RuntimeError: Config file /etc/puppetlabs/puppet/hiera.yaml not found
~~~

[4] https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/validation-scripts/all-nodes.sh#L107