multinode-ipa job is not resolving the NTP hosts on some providers

[Ronelle Landy]

multinode-ipa job is failing the standalone deploy on some upstream providers with the following error:

2022-01-11 08:12:35.752196 | fa163e36-e1ec-94b4-ed17-000000000664 | TASK | Ensure system is NTP time synced
0000000, correction: 0.000000000, skew: 0.000\ntry: 26, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 27, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 28, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 29, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 30, refid: 00000000, correction: 0.000000000, skew: 0.000", "stdout_lines": ["try: 1, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 2, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 3, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 4, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 5, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 6, refid: 00000000, correction: 0.000000000, skew: 0.000", ....

Providers most impacted: rax-ord and ovh-gra1

It seems that the NTP hosts cannot be resolved by IPA server dns.

Example deploy log: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7e2/820535/10/check/tripleo-ci-centos-8-standalone-on-multinode-ipa/7e29cd9/logs/undercloud/home/zuul/standalone_deploy.log

This fails check and gate sporadically - depending on the provider:

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-standalone-on-multinode-ipa

[Ronelle Landy (rlandy)]

Debug patches:

https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/824067 Enable IPA debug logging

https://review.opendev.org/c/openstack/tripleo-heat-templates/+/824069 [DNM] Troubleshooting ntp & ipa stuff

[Ronelle Landy (rlandy)]

https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/824181 - Removing default NTP servers when not defined in job

[Ronelle Landy (rlandy)]

https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/824061/ DNM: Add entries to /etc/hosts - pool.ntp.org

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/824157

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/824208

[Ronelle Landy (rlandy)]

https://review.opendev.org/c/openstack/tripleo-ci/+/824157 - patch to disable voting/gating

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/824233

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/824248

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ronelle Landy <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/824233

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/824157
Committed: https://opendev.org/openstack/tripleo-ci/commit/30f2552c086b0e0311a9a89f190e276f6d7994e4
Submitter: "Zuul (22348)"
Branch: master

commit 30f2552c086b0e0311a9a89f190e276f6d7994e4
Author: Alex Schultz <email address hidden>
Date: Tue Jan 11 15:42:08 2022 +0000

    Revert "Revert "Make standalone-ipa job non-voting during debug""

    This reverts commit 70a392b842cf6dda153895b49693709ae1dac757.

    Reason for revert: We merged the NTP validation again and need to
    troubleshoot why it's failing in some clouds.

    Change-Id: I74347477079a297bee09c754f9ed633c6ec2628e
    Related-Bug: #1957083

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/824996

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/824996
Committed: https://opendev.org/openstack/tripleo-quickstart-extras/commit/4cb5112398ed55c8f2dc02bdc8f9028937823159
Submitter: "Zuul (22348)"
Branch: master

commit 4cb5112398ed55c8f2dc02bdc8f9028937823159
Author: Alex Schultz <email address hidden>
Date: Mon Jan 17 10:19:18 2022 -0700

    Add ipa named recursion back into the config

    In 8.2, recursion was disallowed by default in the IPA server
    configuration because it's generally not a good idea. However in CI
    we're using the IPA server as the primary name resolver for the job so
    we need to allow it because some clouds use external IPs only. This is
    a CI specific fix.

    Change-Id: I29d576d63e794a7a783fc8f954eb15f20977dd86
    Related-Bug: #1957083

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Alex Schultz <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/824248
Reason: while this works, we probably don't need it. let's skip this for now

[Marios Andreou (marios-b)]

looks like we're just waiting for that now to make jobs voting again https://review.opendev.org/c/openstack/tripleo-ci/+/826124

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/824208
Committed: https://opendev.org/openstack/ansible-role-collect-logs/commit/ae4d1c7913a4f3672c99459d7c3fc156a42cab5b
Submitter: "Zuul (22348)"
Branch: master

commit ae4d1c7913a4f3672c99459d7c3fc156a42cab5b
Author: Alex Schultz <email address hidden>
Date: Tue Jan 11 09:24:22 2022 -0700

    Add ipa command log collection

    If IPA is installed, let's capture the config and dns configuration to
    help troubleshoot ipa related things.

    Change-Id: Ic45d32c46dbac2e81abde1a9116d34ffad14bf18
    Related-Bug: #1957083

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ghanshyam <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/824061
Reason: TrieplO project is retiring now, for details, please see https://review.opendev.org/c/openstack/governance/+/905145 or reach out to OpenStack TC.