The mac table size of neutron bridges (br-tun, br-int, br-*) is too small by default and eventually makes openvswitch explode

[Slawek Kaplonski]

Description of problem:

the CPU utilization of ovs-vswitchd is high without DPDK enabled

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1512 root 10 -10 4352840 793864 12008 R 1101 0.3 15810:26 ovs-vswitchd

at the same time we were observing failures to send packets (ICMP) over VXLAN tunnel, we think this might be related to high CPU usage.

--- Reproducer and analysis on ovs side done by Jiri Benc:

Reproducer:

Create an ovs bridge:

------
ovs-vsctl add-br ovs0
ip l s ovs0 up
------

Save this to a file named "reproducer.py":

------
#!/usr/bin/python
from scapy.all import *

data = [(str(RandMAC()), str(RandIP())) for i in range(int(sys.argv[1]))]

s = conf.L2socket(iface="ovs0")
while True:
    for mac, ip in data:
        p = Ether(src=mac, dst=mac)/IP(src=ip, dst=ip)
        s.send(p)
------

Run the reproducer:

./reproducer.py 5000

----
The problem is how flow revalidation works in ovs. There are several 'revalidator' threads launched. They should normally sleep (modulo waking every 0.5 second just to do nothing) and they wake if anything of interest happens (udpif_revalidator => poll_block). On every wake up, each revalidator thread checks whether flow revalidation is needed and if it is, it does the revalidation.

The revalidation is very costly with high number of flows. I also suspect there's a lot of contention between the revalidator threads.

The flow revalidation is triggered by many things. What is of interest for us is that any eviction of a MAC learning table entry triggers revalidation.

The reproducer script repeatedly sends the same 5000 packets, all of them with a different MAC address. This causes constant overflows of the MAC learning table and constant revalidation. The revalidator threads are being immediately woken up and are busy looping the revalidation.

Which is exactly the pattern from the customers' data: there are 16000+ flows and the packet capture shows that the packets are repeating every second.

A quick fix is to increase the MAC learning table size:

ovs-vsctl set bridge <bridge> other-config:mac-table-size=50000

This should lower the CPU usage down substantially; allow a few seconds for things to settle down.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/573696

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/573696
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1f8378e0ac4b8c3fc4670144e6efc51940d796ad
Submitter: Zuul
Branch: master

commit 1f8378e0ac4b8c3fc4670144e6efc51940d796ad
Author: Slawek Kaplonski <email address hidden>
Date: Fri Jun 8 15:37:39 2018 +0200

    [OVS] Add mac-table-size to be set on each ovs bridge

    By default number of MAC addresses which ovs stores in memory
    is quite low - 2048.

    Any eviction of a MAC learning table entry triggers revalidation.
    Such revalidation is very costly so it cause high CPU usage by
    ovs-vswitchd process.

    To workaround this problem, higher value of mac-table-size
    option can be set for bridge. Then this revalidation will happen
    less often and CPU usage will be lower.
    This patch adds config option for neutron-openvswitch-agent to allow
    users tune this setting in bridges managed by agent.
    By default this value is set to 50000 which should be enough for most
    systems.

    Change-Id: If628f52d75c2b5fec87ad61e0219b3286423468c
    Closes-Bug: #1775797

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/575765

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/575773

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/575774

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/575765
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e3be3f913c6f402648691c5ea6a278592adc6810
Submitter: Zuul
Branch: stable/queens

commit e3be3f913c6f402648691c5ea6a278592adc6810
Author: Slawek Kaplonski <email address hidden>
Date: Fri Jun 8 15:37:39 2018 +0200

    [OVS] Add mac-table-size to be set on each ovs bridge

    By default number of MAC addresses which ovs stores in memory
    is quite low - 2048.

    Any eviction of a MAC learning table entry triggers revalidation.
    Such revalidation is very costly so it cause high CPU usage by
    ovs-vswitchd process.

    To workaround this problem, higher value of mac-table-size
    option can be set for bridge. Then this revalidation will happen
    less often and CPU usage will be lower.
    This patch adds config option for neutron-openvswitch-agent to allow
    users tune this setting in bridges managed by agent.
    By default this value is set to 50000 which should be enough for most
    systems.

    Change-Id: If628f52d75c2b5fec87ad61e0219b3286423468c
    Closes-Bug: #1775797
    (cherry picked from commit 1f8378e0ac4b8c3fc4670144e6efc51940d796ad)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/575773
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=44b159aa9fabedd3c8a2989649761bdda1917ae0
Submitter: Zuul
Branch: stable/pike

commit 44b159aa9fabedd3c8a2989649761bdda1917ae0
Author: Slawek Kaplonski <email address hidden>
Date: Fri Jun 8 15:37:39 2018 +0200

    [OVS] Add mac-table-size to be set on each ovs bridge

    By default number of MAC addresses which ovs stores in memory
    is quite low - 2048.

    Any eviction of a MAC learning table entry triggers revalidation.
    Such revalidation is very costly so it cause high CPU usage by
    ovs-vswitchd process.

    To workaround this problem, higher value of mac-table-size
    option can be set for bridge. Then this revalidation will happen
    less often and CPU usage will be lower.
    This patch adds config option for neutron-openvswitch-agent to allow
    users tune this setting in bridges managed by agent.
    By default this value is set to 50000 which should be enough for most
    systems.

    Change-Id: If628f52d75c2b5fec87ad61e0219b3286423468c
    Closes-Bug: #1775797
    (cherry picked from commit 1f8378e0ac4b8c3fc4670144e6efc51940d796ad)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/575774
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=33e259f921c47ac341f55147e3ade78485d1eb93
Submitter: Zuul
Branch: stable/ocata

commit 33e259f921c47ac341f55147e3ade78485d1eb93
Author: Slawek Kaplonski <email address hidden>
Date: Fri Jun 8 15:37:39 2018 +0200

    [OVS] Add mac-table-size to be set on each ovs bridge

    By default number of MAC addresses which ovs stores in memory
    is quite low - 2048.

    Any eviction of a MAC learning table entry triggers revalidation.
    Such revalidation is very costly so it cause high CPU usage by
    ovs-vswitchd process.

    To workaround this problem, higher value of mac-table-size
    option can be set for bridge. Then this revalidation will happen
    less often and CPU usage will be lower.
    This patch adds config option for neutron-openvswitch-agent to allow
    users tune this setting in bridges managed by agent.
    By default this value is set to 50000 which should be enough for most
    systems.

    Change-Id: If628f52d75c2b5fec87ad61e0219b3286423468c
    Closes-Bug: #1775797
    (cherry picked from commit 1f8378e0ac4b8c3fc4670144e6efc51940d796ad)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.6 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ocata-eol release.