pod communication issues due to calico interaction with iptables 1.8

[Ghada Khalil]

Brief Description
-----------------
The iptables package was recently updated to v1.8.4: https://review.opendev.org/c/starlingx/integ/+/815907
Since, we've experienced reports of pod communication issues. This appears to be an interaction issue with calico and the new version of iptables as described here: https://github.com/projectcalico/calico/issues/2322

Severity
--------
Major: Various issues w/ pod communications

Steps to Reproduce
------------------
- Fresh install
- Apply application pods

Expected Behavior
------------------
- pods report issues during deployment

Actual Behavior
----------------
- pods should deploy successfully

Reproducibility
---------------
Intermittent

System Configuration
--------------------
multi-node systems

Branch/Pull Time/Commit
-----------------------
stx master after 2021-10-31

Last Pass
---------
Before 2021-10-31

Timestamp/Logs
--------------
Steve Webster has seen the issue in the lab

Test Activity
-------------
Regression Testing

Workaround
----------
Unknown

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/819894

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/ansible-playbooks/+/819894
Committed: https://opendev.org/starlingx/ansible-playbooks/commit/43af74a27f3173700d7958fc52a23f3ec96150c2
Submitter: "Zuul (22348)"
Branch: master

commit 43af74a27f3173700d7958fc52a23f3ec96150c2
Author: Steven Webster <email address hidden>
Date: Thu Nov 25 18:37:06 2021 -0500

    Force calico to use NFT iptables backend

    This commit sets a calico environment variable to force the use of
    iptables-nft in the calico-node pod.

    StarlingX recently uprevved iptables on the host to 1.8.4. An
    implication of this change is that iptables is now using the nftables
    kernel API and infrastructure to store firewall / traffic rules.
    For example, running iptables --version on the latest release will
    show:

    iptables v1.8.4 (nf_tables)

    This indicates that by default, nftables is being used, rather than
    the older iptables-legacy infrastructure.
    In fact, StarlingX does not even provide the user-space mechanism to
    see the iptables-legacy tables anymore.
    It can however be seen via a pod with the iptable-legacy binary,
    such as via the calico-node pod.

    nf_tables is the right way to move forward for us, as it has a
    number of benefits over the older iptables (legacy) infrastructure.

    One thing to note is that it's entirely possible that a
    pod/container can have it's own version of the user-space iptables.
    That is, a pod based on an older iptables may only be programming
    the legacy tables, while a newer one may be using the iptables-nft
    version. This seems to be causing the mismatch, although
    theoretically the underlying xtables datapath _should_ be the same
    between the two.

    It has been seen that the move to iptables 1.8.4 has caused an
    occasional loss of connectivity between pods.
    Specifically, the traffic seems to be caught up in the FORWARD chain
    of the iptables filter table. Because of a missing cali-FORWARD rule,
    the traffic is dropped as per the default policy of the chain. the
    cali-FORWARD rule was found to be in the legacy table, rather than the
    nft table.

    Currently there are three k8s components that modify iptables -
    kubelet, kube-proxy, and calico-node. They all seem to share the same
    algorigthm for determining whether to use iptables-nft, or
    iptables-legacy. Basically, the number of rules from each table is
    taken, and whichever is higher wins. This seems to be working for k8s
    (host), and kube-proxy. However, we've seen cases where Calico is
    using the iptables-legacy backend.

    This commit tells calico explicitly to use the NFT backend.

    Note 2: The issue was actually only seen with k8s 1.18, which has
    Calico 3.12. Later versions of Calico did not exhibit the same
    behaviour, but I'm changing the k8s 1.19+ versions to explicitly set
    the iptables backend in Calico as well.
    Since StarlingX does not even have a mechanism for manipulating
    the iptables-legacy tables from user-space anymore, I think it's
    safer moving forward to set the backend to "NFT...