Duplicate entries in FDB table
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
neutron |
Won't Fix
|
Undecided
|
Unassigned |
Bug Description
Posting here, because I'm not sure of a better place at the moment.
Environment: Juno
OS: Ubuntu 14.04 LTS
Plugin: ML2/LinuxBridge
root@infra01_
bridge utility, 0.0
root@infra01_
ip utility, iproute2-ss131122
root@infra01_
Linux infra01_
We recently discovered that across the environment (5 controller, 50+ compute) there are (tens of) thousands of duplicate entries in the FDB table, but only for the 00:00:00:00:00:00 broadcast entries. This is in an environment of ~1600 instances, ~4,100 ports, and 80 networks.
In this example, the number of duplicate FDB entries for this particular VTEP jumps wildly:
root@infra01_
1429
root@infra01_
81057
root@infra01_
25806
root@infra01_
473141
root@infra01_
225472
That behavior can be observed for all other VTEPs. We're seeing over 13 million total FDB entries on this node:
root@infra01_
root@infra01_
13554258
We're also seeing the wild counts on compute nodes. These were run within 1 second of the previous completion:
root@compute032:~# bridge fdb show | wc -l
898981
root@compute032:~# bridge fdb show | wc -l
734916
root@compute032:~# bridge fdb show | wc -l
1483081
root@compute032:~# bridge fdb show | wc -l
508811
root@compute032:~# bridge fdb show | wc -l
2349221
On this node, you can see over 28,000 duplicates for each of the entries:
root@compute032:~# bridge fdb show | sort | uniq -c | sort -nr
28871 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.39 self permanent
28871 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.38 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.243.252 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.243.157 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.243.133 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.242.66 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.242.193 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.60 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.59 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.58 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.57 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.55 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.54 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.53 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.51 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.50 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.49 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.48 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.47 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.46 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.45 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.44 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.43 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.42 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.40 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.37 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.36 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.35 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.34 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.33 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.32 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.31 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.30 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.29 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.28 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.27 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.26 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.25 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.24 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.23 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.22 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.21 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.137 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.132 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.131 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.130 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.129 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.128 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.127 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.107 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.106 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.105 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.104 self permanent
28870 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.103 self permanent
28869 00:00:00:00:00:00 dev vxlan-15 dst 172.29.240.136 self permanent
All other entries for other VXLAN networks on this node have 2 duplicates per VTEP, but it varies wildly across the environment.
Using the 'bridge monitor fdb' command, I am unable to see this behavior in action. Nor is there anything wild in the syslog other than messages like this:
2016-01-
2016-01-
2016-01-
2016-01-
2016-01-
2016-01-
2016-01-
2016-01-
If additional info is needed please let me know.
tags: | added: linuxbridge |
Changed in neutron: | |
assignee: | nobody → John Perkins (john-d-perkins) |
Changed in neutron: | |
assignee: | John Perkins (john-d-perkins) → nobody |
Changed in neutron: | |
status: | In Progress → Confirmed |
Turned on debug for a brief moment and captured what I could. Last count was > 12 million FDB entries. Linked here is a gist with a subset of the log that I could capture:
https:/ /gist.github. com/busterswt/ f80db135400623d 92919
Hope it helps. Please let me know if you need any other info.