tripleo-ci-centos-9-standalone-on-multinode-ipa is failing ping test

[Ronelle Landy]

tripleo-ci-centos-9-standalone-on-multinode-ipa is now failing the ping test for redhat.com with the following trace:

2022-04-11 18:41:06.085346 | primary | TASK [ipa-multinode : Check DNS works] *****************************************
2022-04-11 18:41:06.085406 | primary | Monday 11 April 2022 18:41:06 +0000 (0:00:00.053) 0:11:11.186 **********
2022-04-11 18:41:21.952153 | primary | fatal: [subnode-1]: FAILED! => {"changed": true, "cmd": "ping -c 5 redhat.com\n", "delta": "0:00:14.128341", "end": "2022-04-11 18:41:21.604243", "msg": "non-zero return code", "rc": 1, "start": "2022-04-11 18:41:07.475902", "stderr": "", "stderr_lines": [], "stdout": "PING redhat.com (34.235.198.240) 56(84) bytes of data.\n\n--- redhat.com ping statistics ---\n5 packets transmitted, 0 received, 100% packet loss, time 4107ms", "stdout_lines": ["PING redhat.com (34.235.198.240) 56(84) bytes of data.", "", "--- redhat.com ping statistics ---", "5 packets transmitted, 0 received, 100% packet loss, time 4107ms"]}
2022-04-11 18:41:21.956280 | primary |

It is failing on periodic as well:

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-wallaby

ttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master

[Ronelle Landy (rlandy)]

The failing step is here:

https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/ipa-multinode/tasks/ipaserver-undercloud-setup.yml#L282

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/837410

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/837410
Committed: https://opendev.org/openstack/tripleo-quickstart-extras/commit/6fc79351fd7a3347e0dab9b78720bc566679b578
Submitter: "Zuul (22348)"
Branch: master

commit 6fc79351fd7a3347e0dab9b78720bc566679b578
Author: Ronelle Landy <email address hidden>
Date: Mon Apr 11 16:24:33 2022 -0400

    Switch test DNS resolution

    Original test site is not resolving at the moment.
    Switching test sites to avoid the failure.

    Change-Id: I4099383d9fe4a0303a218e1ae61df2545934b41e
    Related-Bug: #1968615

[Jakob Meng (jm1337)]

Nowadays, redhat.com resolves to 34.235.198.240 and 52.200.142.250. Both IPs cannot be pinged, they do not respond to icmp requests. Previously redhat.com was resolved to 209.132.183.105 which sent out icmp responses.

Red Hat's Support denied my request to allow icmp traffic to redhat.com again with:

> These IPs [redhat.com] are for a redirect service that redirects to
> www.redhat.com. I don't know what TripleO CI jobs intend to do, but
> you most likely want to be pinging www.redhat.com servers instead.

[Alan Pevec (apevec)]

What is this ping test trying to validate?
Task is "Check DNS works" so it should not do both nameresolver config test AND ICMP which is different networking layer!

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/839094

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/839094
Committed: https://opendev.org/openstack/tripleo-quickstart-extras/commit/91d707f4c618a5b75d29096b2c5712af5fa7204f
Submitter: "Zuul (22348)"
Branch: master

commit 91d707f4c618a5b75d29096b2c5712af5fa7204f
Author: Rafael Castillo <email address hidden>
Date: Fri Apr 22 11:53:36 2022 -0700

    Change DNS test to use dig instead of ping

    The ping test would fail if icmp requests were blocked by the host, even
    if dns was working correctly. Using dig ensures we don't try to talk to
    he resolved host at all.

    Also restores the host tested to the one used before the workaround to
    the related bug.

    Related-Bug: #1968615
    Change-Id: I121f49edb0ee8fedc085e6c6fc94f548b95c8713