Failures to connect to localhost due to arp table overflow disrupt whole cluster

[Bogdan Dobrelya]

When network controller host a router that is connecting networks with many VMs, node's arp table gets overflowed and local connections start to fail.

This affects OCF monitoring where some scripts use client libraries which send requests to localhost.

[Bogdan Dobrelya (bogdando)]

The UX for this issue is a critical.

[Bogdan Dobrelya (bogdando)]

And for the MOS perspective, neutron agents code should be inspected for possible flaws causing so many connections (?) which make conntrack to behave so unstable.

[Eugene Nikanorov (enikanorov)]

Indeed it's hardly a neutron bug.
We need to test these conditions on our scale lab.
Right now it's not clear what we can do at neutron side.

Meanwhile the solution might be to configure conntrack on controllers and increase limits.

I'm putting this to Incomplete until we test it.

[Eugene Nikanorov (enikanorov)]

Repro should involve creating the same amount of resources/connections as on the node that has hit the issue:

netstat -apn | wc -l
~1000

conntrack -L | wc -l
~4000

neutron floatingip-list | wc -l
~600

[Eugene Nikanorov (enikanorov)]

Additional details:
router namespace has

conntrack -L | wc -l
430000

Reaching such load requires kernel parameters to be adjusted.
But then it seems that it's possible to reproduce the issue with less amount of connections in default settings

[Bogdan Dobrelya (bogdando)]

Update: the arp neighbour cache may be undersized on nodes with many namespaces resulting in thrashing and garbage collection of the ARP table.
The following potential fix should be tested for the 'ping: sendmsg: Invalid argument' and 'net_ratelimit' warning signs:
sysctl net.ipv4.neigh.default.gc_thresh1=1024
sysctl net.ipv4.neigh.default.gc_thresh2=2048
sysctl net.ipv4.neigh.default.gc_thresh3=4096

while current out-of-box values are:
net.ipv4.neigh.default.gc_thresh1 = 128
net.ipv4.neigh.default.gc_thresh2 = 512
net.ipv4.neigh.default.gc_thresh3 = 1024

[Eugene Nikanorov (enikanorov)]

Reproduced by lowering net.ipv4.neigh.default.gc_thresh* down to 4-8-16

[Bogdan Dobrelya (bogdando)]

thank you, @Eugene, for the great job done

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/218204

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/218204
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=6d32ad9e3e2fefa7819bfa7207623aa6667bf8bb
Submitter: Jenkins
Branch: master

commit 6d32ad9e3e2fefa7819bfa7207623aa6667bf8bb
Author: Sergey Kolekonov <email address hidden>
Date: Fri Aug 28 13:54:43 2015 +0300

    Avoid neighbour table overflow problem

    “Neighbour table overflow” error which occurs in large networks when there are
    two many ARP requests which the server is not able to reply.

    This problem can occur when there're a lot of connections in Neutron routers'
    namespaces. Tune systcl to avoid this problem.

    Closes-bug: #1488938
    Co-Authored-By: Eugene Nikanorov <email address hidden>
    Co-Authored-By: Bogdan Dobrelya <email address hidden>
    Change-Id: I00e954c4792fd4d7993fd5e36fef9be86af22196

[Andrew Woodward (xarses)]

+ Customer found on 6.0 added series for 6.0 and 6.1. Please backport

[Vitaly Sedelnik (vsedelnik)]

Won't Fix for 6.0-updates as the fix affects fuel-librarys - will document as known issue. Nominated for MU3 for 6.1

[Alexander Saprykin (cutwater)]

Verified on ISO 288

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/223069

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/223069
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=35012e035389b4f3ccb8387133bf30117d4b7821
Submitter: Jenkins
Branch: stable/6.1

commit 35012e035389b4f3ccb8387133bf30117d4b7821
Author: Sergey Kolekonov <email address hidden>
Date: Fri Aug 28 13:54:43 2015 +0300

    Avoid neighbour table overflow problem

    “Neighbour table overflow” error which occurs in large networks when there are
    two many ARP requests which the server is not able to reply.

    This problem can occur when there're a lot of connections in Neutron routers'
    namespaces. Tune systcl to avoid this problem.

    Closes-bug: #1488938
    Co-Authored-By: Eugene Nikanorov <email address hidden>
    Co-Authored-By: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit 6d32ad9e3e2fefa7819bfa7207623aa6667bf8bb)
    Change-Id: I00e954c4792fd4d7993fd5e36fef9be86af22196

[Alexey Stupnikov (astupnikov)]

Verified on Fuel-6.1

[Alexander Bozhenko (alexbozhenko)]

Could you please add the same for ipv6? There is a customer, who uses it in production.