[9-stream][master][yoga] ovs/ovn fips jobs fails randomly with DNS issue

[yatin]

These FIPS jobs running on centos-9-stream fails randomly at different points due to DNS issue.
In these jobs after configuring fips, node get's rebooted. The unbound service may take some time to be ready after the reboot and until then any DNS resolution fails.

Fails like below:-
2022-06-16 02:36:58.829027 | controller | + ./stack.sh:_install_rdo:314 : sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo http://trunk.rdoproject.org/centos9-master/delorean-deps.repo
2022-06-16 02:36:58.940595 | controller | % Total % Received % Xferd Average Speed Time Time Time Current
2022-06-16 02:36:58.940689 | controller | Dload Upload Total Spent Left Speed
2022-06-16 02:36:58.941657 | controller |
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: trunk.rdoproject.org
2022-06-16 02:36:58.946989 | controller | + ./stack.sh:_install_rdo:316 : sudo dnf -y update
2022-06-16 02:36:59.649717 | controller | CentOS Stream 9 - BaseOS 0.0 B/s | 0 B 00:00
2022-06-16 02:36:59.650078 | controller | Errors during downloading metadata for repository 'baseos':
2022-06-16 02:36:59.650117 | controller | - Curl error (6): Couldn't resolve host name for https://mirror-int.ord.rax.opendev.org/centos-stream/9-stream/BaseOS/x86_64/os/repodata/repomd.xml [Could not resolve host: mirror-int.ord.rax.opendev.org]

2022-06-16 02:40:30.411317 | controller | + functions-common:sudo_with_proxies:2384 : sudo http_proxy= https_proxy= no_proxy= dnf install -y rabbitmq-server
2022-06-16 02:40:31.225502 | controller | Last metadata expiration check: 0:03:30 ago on Thu 16 Jun 2022 02:37:01 AM UTC.
2022-06-16 02:40:31.278762 | controller | No match for argument: rabbitmq-server
2022-06-16 02:40:31.286118 | controller | Error: Unable to find a match: rabbitmq-server

Job builds:-
- https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovn-tempest-ovs-release-fips&project=openstack/neutron
- https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovs-tempest-fips&project=openstack/neutron

[Miguel Lavalle (minsel)]

Proposed fix https://review.opendev.org/c/openstack/neutron/+/846001

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/846001
Committed: https://opendev.org/openstack/neutron/commit/51360b5a08230716a237317c4493ddcd9172cde0
Submitter: "Zuul (22348)"
Branch: master

commit 51360b5a08230716a237317c4493ddcd9172cde0
Author: yatinkarel <email address hidden>
Date: Wed Jun 15 20:12:51 2022 +0530

    Set nslookup_target in FIPS jobs

    FIPS jobs reboot nodes after enabling FIPS,
    but unbound service takes some time to
    be ready and until then DNS resolution fails.
    [1] added support to check DNS is ready after
    reboot, in order to utilize it we need to
    set "nslookup_target" job var.

    [1] https://review.opendev.org/c/zuul/zuul-jobs/+/844704

    Closes-Bug: #1978938
    Change-Id: Ia39753f7cc533ba9959136b95bc3401d2f34cad4

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/847995

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/847995
Committed: https://opendev.org/openstack/neutron/commit/6d07d23f8d919e430ae02323185567e98453dc37
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 6d07d23f8d919e430ae02323185567e98453dc37
Author: yatinkarel <email address hidden>
Date: Wed Jun 15 20:12:51 2022 +0530

    Set nslookup_target in FIPS jobs

    FIPS jobs reboot nodes after enabling FIPS,
    but unbound service takes some time to
    be ready and until then DNS resolution fails.
    [1] added support to check DNS is ready after
    reboot, in order to utilize it we need to
    set "nslookup_target" job var.

    [1] https://review.opendev.org/c/zuul/zuul-jobs/+/844704

    Closes-Bug: #1978938
    Change-Id: Ia39753f7cc533ba9959136b95bc3401d2f34cad4
    (cherry picked from commit 51360b5a08230716a237317c4493ddcd9172cde0)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.0.0.0rc1 release candidate.