tripleo-ci-centos-8-standalone-on-multinode-ipa is failing sporadically on 'install tls dependencies'

[Ronelle Landy]

tripleo-ci-centos-8-standalone-on-multinode-ipa is failing sporadically on 'install tls dependencies' task. The repo that fails to install is not always the same but is usually python3-pyusb. An example error and logs are included below:

2020-06-22 11:41:24.655424 | primary | TASK [ipa-multinode : install tls dependencies] ********************************
2020-06-22 11:41:24.655585 | primary | Monday 22 June 2020 11:41:24 +0000 (0:00:00.592) 0:13:12.241 ***********
2020-06-22 11:41:27.348926 | primary | fatal: [undercloud]: FAILED! => {
2020-06-22 11:41:27.349003 | primary | "changed": false,
2020-06-22 11:41:27.349031 | primary | "results": []
2020-06-22 11:41:27.349061 | primary | }
2020-06-22 11:41:27.349099 | primary |
2020-06-22 11:41:27.349132 | primary | MSG:
2020-06-22 11:41:27.349157 | primary |
2020-06-22 11:41:27.349181 | primary | Failed to download packages: Cannot download Packages/python3-pyusb-1.0.0-9.module_el8.2.0+370+b142e101.noarch.rpm: All mirrors were tried

https://773bfd9c27e0a22edfee-9129bbe5bc5ca2743983500bca1995c4.ssl.cf5.rackcdn.com/737058/1/check/tripleo-ci-centos-8-standalone-on-multinode-ipa/9cc8342/job-output.txt

https://07b92d57b5f1a612a74b-4cebd80e32fd7e7586df4103c9478cce.ssl.cf5.rackcdn.com/737058/1/check/tripleo-ci-centos-8-standalone-on-multinode-ipa/c7d95cc/job-output.txt

https://027cb57af33b4dad80bb-9476297ff63d6e8659419c3d29e9b66c.ssl.cf1.rackcdn.com/735815/1/gate/tripleo-ci-centos-8-standalone-on-multinode-ipa/a717cc9/job-output.txt

The job history is:
https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-on-multinode-ipa

Note that there is another (unrelated) multinode IPA error under investigation - see https://bugs.launchpad.net/tripleo/+bug/1884287

[Ronelle Landy (rlandy)]

The following rpms are installed in the failing step:

https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/ipa-multinode/defaults/main.yml#L22

tls_packages:
  - ipa-client
  - python3-pip
  - openssl-perl

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/737323

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ronelle Landy (<email address hidden>) on branch: master
Review: https://review.opendev.org/737323

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/737323
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=b91c3376e345c22a2d05e430fcd54c3442f55051
Submitter: Zuul
Branch: master

commit b91c3376e345c22a2d05e430fcd54c3442f55051
Author: Ronelle Landy <email address hidden>
Date: Mon Jun 22 13:07:18 2020 -0400

    Remove the tls deps install task - dependency of the tripleo-ipa

    The tls dep install task is failing sporadically on some clouds.
    This task skould not need to install all the deps as they
    should be installed as a dependency of the tripleo-ipa package.

    Change-Id: I6bda4f9f9208013d5e19e75ab63a443a571fc7f9
    Closes-Bug: #1884570

[Ronelle Landy (rlandy)]

Reopening this - still not fixed - now openssl has install issues.
Not all mirrors provide these rpms.
We need a reliable way to install them.

Making the job non-voting until we figure this out

[Ronelle Landy (rlandy)]

https://review.opendev.org/#/c/737998/ - to make job non-voting until the deps install is worked out

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/739135

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/739135
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=7fea87335557b3b07e61a9f5800b25ac9497541d
Submitter: Zuul
Branch: master

commit 7fea87335557b3b07e61a9f5800b25ac9497541d
Author: Ronelle Landy <email address hidden>
Date: Thu Jul 2 15:19:34 2020 -0400

    Move tls deps install before DNS changes

    Ensure that required TLS dependencies are installed
    before standard DNS entries are removed and the new TLS setup
    has not yet completed.

    Change-Id: Ie61065107ec1aed207e9207a0d59db706e96941a
    Related-Bug: #1884570

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/741035

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741035
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=d4f462dc8d344665d074eae192ef9caf5f3cafcc
Submitter: Zuul
Branch: master

commit d4f462dc8d344665d074eae192ef9caf5f3cafcc
Author: Ronelle Landy <email address hidden>
Date: Tue Jul 14 15:12:44 2020 -0400

    Set install of ipa packages to false - use local forwarder

    multinode-ipa job has been failing spordically on package
    installs during the run of the undercloud-ipa-install.yaml
    playbook. These package installs should not be required as
    dependencies for TLS are installed earlier.
    This patch sets the package installs to false to avoid
    the sporadic failures.

    This package also gets the local cloud DNS forwarder
    rather than using 1.1.1.1 for all cases.

    Change-Id: I729982744a029e16b210c3db9021ccbfe583cbbb
    Related-Bug: #1884570