Issue when using a chain policy in iptables on master

[Cédric Jeanneret]

Hello there,

Context:
We're wanting to push our custom tripleo firewall rules in a dedicated chain, and redirect from INPUT to that new TRIPLEO_INPUT chain.
The patch is here: https://review.opendev.org/c/openstack/tripleo-ansible/+/839163

The reason is: make things clearer, and easier to manage, as well as to avoid any lockout when applying the rules.

In order to make the whole thing more customizable and not really depend on the actual order before hitting a final DROP, we're switching to a chain policy (namely, -P INPUT DROP) instead of the final '-A INPUT -m conntrack --ctstate NEW -m comment --comment "999 drop all ipv4" -j DROP'.

While it seems to be pretty the same, it isn't: something isn't properly allowed, apparently linked to ARP (nftables covers both l2 and l3, apparently).

As a consequence, there are issues within the RDO jobs, for instance this one:
periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master

It can deploy both UC and OC, but during the tempest phase, it tries to ping a virtual router, and fails:
TASK [os_tempest : Ping router ip address] *************************************
Friday 06 May 2022 12:42:51 -0400 (0:00:00.087) 2:03:29.990 ************
FAILED - RETRYING: Ping router ip address (5 retries left).
FAILED - RETRYING: Ping router ip address (4 retries left).
FAILED - RETRYING: Ping router ip address (3 retries left).
FAILED - RETRYING: Ping router ip address (2 retries left).
FAILED - RETRYING: Ping router ip address (1 retries left).
fatal: [undercloud]: FAILED! => {"attempts": 5, "changed": true, "cmd": "set -e\nping -c2 \"10.0.0.184\"\n", "delta": "0:00:03.118020", "end": "2022-05-06 12:44:10.409759", "msg": "non-zero return code", "rc": 1, "start": "2022-05-06 12:44:07.291739", "stderr": "", "stderr_lines": [], "stdout": "PING 10.0.0.184 (10.0.0.184) 56(84) bytes of data.\nFrom 10.0.0.1 icmp_seq=1 Destination Host Unreachable\nFrom 10.0.0.1 icmp_seq=2 Destination Host Unreachable\n\n--- 10.0.0.184 ping statistics ---\n2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1061ms\npipe 2", "stdout_lines": ["PING 10.0.0.184 (10.0.0.184) 56(84) bytes of data.", "From 10.0.0.1 icmp_seq=1 Destination Host Unreachable", "From 10.0.0.1 icmp_seq=2 Destination Host Unreachable", "", "--- 10.0.0.184 ping statistics ---", "2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1061ms", "pipe 2"]}

This is fully reproducible in any lab with the mentioned patch - you can get a small env, with one UC and one Controller, create the networks and router, get the router IP, and it will fail to ping.

Fun things:
- If you add an IP in the router network on the right interface of the UC, as well as ensure the route table contains the right line for that same network, it will ping; once you remove this IP and related route, it will STILL ping
- That router IP doesn't answer to ping from the Controller either with the patch, while it does answer with plain master
- While the reproducer is easy to get, you'll need to drop everything (both UC and OC) in order to do more tests once it pings

It really, really looks like an l2 issues rather than l3. Pretty sure the "drop" policy within nftables chains is also applied to ARP, which may be the issue - and, if this is the case, we'll need to find a way to tell nftables to allow ARP on some interfaces.

I'm able to provide 2 envs in parallel, one patched, one unpatched, if needed.

Thank you for your help!

[Cédric Jeanneret (cjeanner)]

Some more links:
here's a testproject job where we add the policy without playing with other chains:
https://review.rdoproject.org/r/c/testproject/+/42613 (check commit message for tripleo-ansible patch)

here's a testproject with the actual TRIPLEO_INPUT patch: https://review.rdoproject.org/r/c/testproject/+/42344

[Cédric Jeanneret (cjeanner)]

Some more info:

- policy set to ACCEPT, create router, ping it, it's fine. Stop ping
- switch policy to DROP, ping again, it's fine. Stop ping
- remove router, create a new one (hence: new MAC, new IP), ping - doesn't work. Stop ping
- switch policy to ACCEPT, ping again, it's still failing.

Would be interesting testing with iptables-legacy.

[Cédric Jeanneret (cjeanner)]

Tested with iptables-legacy:

dnf install epel-release
[disable epel repositories]
dnf install --enablerepo epel -y iptables-legacy
update-alternatives --config iptables [select iptables-legacy]
reboot

Notes:
- INPUT policy was ACCEPT in the restore file
- checking the iptables content after the reboot showed the rules, and the ACCEPT policy

Then:
- ping the already-existing router: OK
- switched policy to DROP, ping OK
- replaced the router (delete, create), ping OK

This is probably a good proof the issue isn't at L3 level, but deeper in the network, like L2....

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841259

[Slawek Kaplonski (slaweq)]

After long investigation and help from the RHEL iptables expert, Phil Sutter, I found out that the problem is with the geneve tunnel which is marked to be not tracked in the raw table but later in the filter table we have rule to allow such traffic with conntrack state "NEW" which didn't match. With default policy ACCEPT it all works just because geneve traffic is accepted by that default policy. We need to have rule to allow geneve traffic which has conntrack state UNTRACKED instead of NEW and that should solve the problem.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841259
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/d6e9f84fb46e421708d47a742e31dddebab1592b
Submitter: "Zuul (22348)"
Branch: master

commit d6e9f84fb46e421708d47a742e31dddebab1592b
Author: Slawek Kaplonski <email address hidden>
Date: Tue May 10 14:24:26 2022 +0200

    Iptables rule for geneve traffic should match UNTRACKED state

    In the raw table we are explicity set Geneve traffic to be not tracked
    by conntrack. Because of that we should later in the filter table match
    on the UNTRACKED geneve traffic instead of NEW.

    Closes-Bug: #1972283
    Change-Id: If744e35d4c0286a50ccf0220c122f8355d5833f2

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 17.0.0 release.