cs8 train: os_tempest ping router ip address task failing with Destination Host Unreachable

[Bhagyashri Shewale]

periodic-tripleo-ci-centos-8-scenario007-standalone-train (non-voting) is failing with blow error:

Error logs:

2023-08-30 10:21:25.008998 | primary | TASK [os_tempest : Ping router ip address] *************************************
2023-08-30 10:21:25.009006 | primary | Wednesday 30 August 2023 10:21:25 +0000 (0:00:00.090) 0:23:31.740 ******
2023-08-30 10:21:29.631453 | primary | FAILED - RETRYING: Ping router ip address (5 retries left).
2023-08-30 10:21:43.997530 | primary | FAILED - RETRYING: Ping router ip address (4 retries left).
2023-08-30 10:21:58.528763 | primary | FAILED - RETRYING: Ping router ip address (3 retries left).
2023-08-30 10:22:13.247635 | primary | FAILED - RETRYING: Ping router ip address (2 retries left).
2023-08-30 10:22:27.770777 | primary | FAILED - RETRYING: Ping router ip address (1 retries left).
2023-08-30 10:22:42.141586 | primary | fatal: [undercloud]: FAILED! => {"attempts": 5, "changed": true, "cmd": "set -e\nping -c2 \"192.168.24.173\"\n", "delta": "0:00:03.060747", "end": "2023-08-30 10:22:41.951456", "msg": "non-zero return code", "rc": 1, "start": "2023-08-30 10:22:38.890709", "stderr": "", "stderr_lines": [], "stdout": "PING 192.168.24.173 (192.168.24.173) 56(84) bytes of data.\nFrom 192.168.24.1 icmp_seq=1 Destination Host Unreachable\nFrom 192.168.24.1 icmp_seq=2 Destination Host Unreachable\n\n--- 192.168.24.173 ping statistics ---\n2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1006ms\npipe 2", "stdout_lines": ["PING 192.168.24.173 (192.168.24.173) 56(84) bytes of data.", "From 192.168.24.1 icmp_seq=1 Destination Host Unreachable", "From 192.168.24.1 icmp_seq=2 Destination Host Unreachable", "", "--- 192.168.24.173 ping statistics ---", "2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1006ms", "pipe 2"]}
2023-08-30 10:22:42.141707 | primary |
2023-08-30 10:22:42.141715 | primary | NO MORE HOSTS LEFT *************************************************************
2023-08-30 10:22:42.145585 | primary |
2023-08-30 10:22:42.145607 | primary | PLAY RECAP *********************************************************************
2023-08-30 10:22:42.145615 | primary | undercloud : ok=130 changed=48 unreachable=0 failed=1 skipped=168 rescued=0 ignored=0

[1]: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario007-standalone-train&skip=0
[2]: https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario007-standalone-train/af697cc/job-output.txt

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/893512

[Douglas Viroel (dviroel)]

The nftable major update might be causing this:

-nftables-0.9.3-26.el8.x86_64
+nftables-1.0.4-2.el8.x86_64

Error from iptables-manager:

2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info [-] Failed to process floating IPs.: neutron_lib.exceptions.l3.IpTablesApplyException: Failure applying iptables rules
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/linux/iptables_manager.py", line 433, in defer_apply
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info self.defer_apply_off()
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/linux/iptables_manager.py", line 447, in defer_apply_off
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info self._apply()
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/linux/iptables_manager.py", line 464, in _apply
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info first = self._apply_synchronized()
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/linux/iptables_manager.py", line 624, in _apply_synchronized
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info raise err
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/linux/iptables_manager.py", line 516, in _do_run_restore
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info run_as_root=True, **kwargs)
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info File "/usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 148, in execute
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info returncode=returncode)
2023-08-29 10:21:23.220 ERROR /var/log/containers/neutron/l3-agent.log: 114802 ERROR neutron.agent.l3.router_info neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Stdin: # Generated by iptables_manager

https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario007-standalone-train/d98ae11/logs/undercloud/var/log/extra/big-errors.txt

[Douglas Viroel (dviroel)]

These errors are actually related to iptables 1.8.5-8, according with:

https://bugzilla.redhat.com/show_bug.cgi?id=2236501

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/893512
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/e08e44b88f387fd23471dca5c05283029374ae79
Submitter: "Zuul (22348)"
Branch: master

commit e08e44b88f387fd23471dca5c05283029374ae79
Author: Douglas Viroel <email address hidden>
Date: Fri Sep 1 10:13:40 2023 -0300

    Block iptables updates on Centos-8 train

    This patch excludes newer versions of iptables that might me breaking
    some CS-8 stable/train jobs. It also pins the rpm to the latest
    knwon good version.

    Related-Bug: #2033644

    Change-Id: I7d22932ab32bea52b302a1d0274e11b4d86525c3

[Alan Pevec (apevec)]

IMO this workaround is fine for the remaining (short) life of TripleO stable/train.