ipa job mirror issue tracker for master and wallaby

[chandan kumar]

FS064, FS039 and standalone-ipa jobs fails with following error during ipa preparation on master and wallaby branch.
```
2022-07-20 18:52:03 | + sudo dnf install -yq ansible-core ansible-tripleo-ipa
2022-07-20 18:52:06 | Error: Error downloading packages:
2022-07-20 18:52:06 | ansible-tripleo-ipa-0.2.3-0.20220301014811.6b0ed82.el9.noarch: Cannot download, all mirrors were already tried without success
2022-07-20 18:52:06 |
```
or
```
2022-07-20 18:48:28 | + sudo dnf install -yq ansible-core ansible-tripleo-ipa
2022-07-20 18:48:31 | Error: Error downloading packages:
2022-07-20 18:48:31 | ansible-core-2.13.1-2.el9.x86_64: Cannot download, all mirrors were already tried without success
2022-07-20 18:48:31 |
```
We have seen either of these abive two error at the same task.
```
TASK [freeipa-setup : Run the tripleo-ipa preparation script] ******************
2022-07-20 18:52:01.985847 | primary | Wednesday 20 July 2022 18:52:01 -0400 (0:00:02.803) 0:19:21.086 ********
2022-07-20 18:52:06.616031 | primary | fatal: [supplemental]: FAILED! => {"changed": true, "cmd": "set -o pipefail && ~cloud-user/ipa_prep.sh 2>&1 | awk '{ print strftime(\"%Y-%m-%d %H:%M:%S |\"), $0; fflush(); }' > ~cloud-user/ipa_prep.sh.log\n", "delta": "0:00:03.294997", "end": "2022-07-20 18:52:06.427693", "msg": "non-zero return code", "rc": 1, "start": "2022-07-20 18:52:03.132696", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
```

Note: It is a tracker bug for the same.

Logs:
[1]. https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/481c0be/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz

[2]. https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/b5f2114/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz

[Alfredo Moralejo (amoralej)]

My understanding of what's happening:

1. IPA deployment is working fine, packages installed and ipa configured:

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/481c0be/logs/supplemental/home/cloud-user/deploy_freeipa.log.txt.gz

2. In the next package installation attempt (part of the tripleo-ipa preparation script, it fails to reach the package mirrors:

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/481c0be/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz

I'd say step 1 is the root cause, i guess it's changing dns configuration or something like that. Would it be possible to hold a node for that job to inspect the status of the server after the failure?

[Ananya Banerjee (frenzyfriday)]

I have rerun the job, will hold the node and pass the IP

[Ananya Banerjee (frenzyfriday)]

We dont see it anymore. Looks like this is solved

[Marios Andreou (marios-b)]

moving this to fix released as we aren't seeing it in the last few days