Comment 1 for bug 1978806

Revision history for this message
Trent Lloyd (lathiat) wrote :

The problem here is that because OVN loads the nf_conntrack module - it starts tracking connections for all interfaces on the host even though there are 0 iptables/nftables rules. These connections thus take up space in the tables that is needed by OVN.

For normal host connections it is not normally a big problem however for both GENEVE and VXLAN overlayed packets they set the SOURCE port of every packet to a different port in the range 1-65534.

It assigns a static port to each "tunelled flow" by hashing the tunneled packets (src_ip, dst_ip, src_port, dst_port) tuple with the intent that the GENEVE/VXLAN packets will get distributed over multiple links of an LACP or bonded link - however the same tunneled flow will always have the port and thus take the same path and preserve network packet ordering. If they used the same source/destination port then all traffic between the same 2 HVs would not get distributed over LACP and if the port was random it would not reserve network packet order as packets of the same connection may go over multiple links.

Reference: https://datatracker.ietf.org/doc/html/draft-gross-geneve-01/#section-3.3

Thus when you have a very busy cloud you potentially have (NUMBER_OF_HYPERVISORS ^ 2) * 65534 potential conntrack entries per hypervisor. FOr 40 hypervisors that is about 2.5 million entries. The default nf_conntrack_max in ovn-chassis 1,000,000.

In practice in production we saw hundreds of thousands of conntrack entries normally and in excess of 1 million in some cases leading to dropped connections as the nf_conntrack_max was exceeded. This is notably worse in a public internet facing application which speaks to many different remote Internet IPs as well.

There is precedence for this change in other projects:
https://bugs.launchpad.net/tripleo/+bug/1885551
https://github.com/antrea-io/antrea/issues/1133
https://bugzilla.redhat.com/show_bug.cgi?id=1985336