Centos8 train missing some needed Iptables rules - Timeout exception waiting for the logger. Please check connectivity to [<IP>:19885]

[Sandeep Yadav]

Centos8 train is missing some needed Iptables rules which is causing below issues:-

https://89efc0a3fde3c09ac7e0-e8eca875c493e144e603ac9a9d6a27d8.ssl.cf2.rackcdn.com/737682/2/check/tripleo-ci-centos-8-containers-multinode-train/22bf85e/job-output.txt

* Below tasks failed because zuul executor unable to connect to port 19885 of undercloud(primary node).
~~~
2020-07-08 06:37:28.576207 | TASK [Check script existence]
2020-07-08 06:37:33.381977 | Timeout exception waiting for the logger. Please check connectivity to [198.101.251.41:19885]
2020-07-08 06:37:33.389729 | primary | ok
2020-07-08 06:37:33.587819 |
2020-07-08 06:37:33.588398 | TASK [Run ansible playbook to collect logs]
2020-07-08 06:37:38.702989 | Timeout exception waiting for the logger. Please check connectivity to [198.101.251.41:19885]
2020-07-08 06:49:07.816263 | primary | ok: Runtime: 0:11:33.098212
~~~

* Below task failed while zuul executor trying to collect logs from secondary node because it cannot ssh to port 22 of secondary node:-

~~~
2020-07-08 06:49:35.560497 | LOOP [fetch-output : Collect logs, artifacts and docs]
2020-07-08 06:49:37.407568 | primary | changed: .d..tp..... ./
2020-07-08 06:49:37.408400 | primary | changed: All items complete
2020-07-08 06:49:37.408541 |
2020-07-08 06:49:38.565070 | primary | changed: .d..tp..... ./
2020-07-08 06:49:39.725616 | primary | changed: .d..tp..... ./
2020-07-08 06:56:09.835836 |
.
.
2020-07-08 06:56:14.644328 | PLAY RECAP
2020-07-08 06:56:14.644483 | primary | ok: 5 changed: 4 unreachable: 0 failed: 0 skipped: 1 rescued: 0 ignored: 0
2020-07-08 06:56:14.644887 | secondary | ok: 2 changed: 1 unreachable: 0 failed: 1 skipped: 1 rescued: 0 ignored: 0
~~~

* Comparing tripleo-ci-centos-8-containers-multinode-train[1] vs tripleo-ci-centos-7-containers-multinode-train[2] we noticed some iptables rules are missing:-

~~~
-A openstack-INPUT -i lo -j ACCEPT
-A openstack-INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
-A openstack-INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A openstack-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A openstack-INPUT -p tcp -m state --state NEW -m tcp --dport 19885 -j ACCEPT
-A openstack-INPUT -s 172.24.4.0/23 -p udp -m udp --dport 69 -j ACCEPT
-A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 6385 -j ACCEPT
-A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 80 -j ACCEPT
-A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 8000 -j ACCEPT
-A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 8003 -j ACCEPT
-A openstack-INPUT -s 172.24.4.0/23 -p tcp -m tcp --dport 8004 -j ACCEPT
-A openstack-INPUT -m limit --limit 2/min -j LOG --log-prefix "iptables dropped: "
-A openstack-INPUT -j REJECT --reject-with icmp-host-prohibited
~~~

[1] https://89efc0a3fde3c09ac7e0-e8eca875c493e144e603ac9a9d6a27d8.ssl.cf2.rackcdn.com/737682/2/check/tripleo-ci-centos-8-containers-multinode-train/22bf85e/logs/undercloud/var/log/extra/network.txt

[2] https://56d2313a0e09771ea5c4-ce557708b87a3b26463b6947241b4c8b.ssl.cf2.rackcdn.com/740324/1/check/tripleo-ci-centos-7-containers-multinode-train/dbaa0a4/logs/undercloud/var/log/extra/network.txt

This seems to be caused because /etc/sysconfig/iptables is emptied in tripleo-bootstrap [3] and and then nft flush ruleset[4] removes in memory iptables rules.

[3] https://github.com/openstack/tripleo-ansible/blob/stable/train/tripleo_ansible/roles/tripleo-bootstrap/tasks/main.yml#L81-L89

[4] https://github.com/openstack/tripleo-heat-templates/blob/stable/train/deployment/tripleo-firewall/tripleo-firewall-baremetal-puppet.yaml#L72-L82

[Marios Andreou (marios-b)]

the bug at https://bugs.launchpad.net/tripleo/+bug/1886953 seems to share the same root cause as this - added https://bugs.launchpad.net/tripleo/+bug/1886953/comments/1 with details

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/739963
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9db0d36614c5abdb0ccb02ff6b300bc0d277f672
Submitter: Zuul
Branch: stable/train

commit 9db0d36614c5abdb0ccb02ff6b300bc0d277f672
Author: yatinkarel <email address hidden>
Date: Fri Jul 10 11:22:18 2020 +0530

    Revert "[train/backport] Prevent nftables to interfere with tripleo firewall"

    This reverts commit c3b24599d7055c4dead110d25d7479eca7a557fe.

    Revert "Do not fail if /usr/sbin/nft is not present"

    This reverts commit eedb679db95b281b2be0199d48876b8af64ea3a0.

    The nftables rules are not enabled now after [1] so reverting
    the original changes which are not needed.

    [1] https://git.centos.org/rpms/nftables/c/3730f48

    Related-Bug: #1870095
    Related-Bug: #1869166
    Closes-Bug: #1887112
    Change-Id: Ib3309cbbd6f2ca300ec205528402a3836a6f34df

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.

[Marios Andreou (marios-b)]

Bug status has been set to 'Fix-Released' based on the discussion and/or patches above. If you disagree please re-set 'Triaged' and reach out to us on freenode #tripleo thank you!