After contrail-api restart, arp packets from BMS dropped on TSN

[Vedamurthy Joshi]

R2.1 Build 36 Ubuntu 12.04.3 Icehouse

This is a scale setup with 2 config nodes (nodei34, nodei35)

On a VN vn1 with 10.1.0.0/16 subnet, there are about 4K ports and ~400 real endpoints (netns instances acting as BMS on openvswitch)
When initally setup, things were fine.
After restarting neutron-server and contrail-api on both the nodes, all flooded traffic from BMS to TSN is getting dropped by vrouter as "Invalid VNID"

env.roledefs = {
    'all': [host2, host3, host4, host5, host6],
    'cfgm': [host2, host3],
    'openstack': [host2],
    'webui': [host3],
    'control': [host3, host4],
    'compute': [host5, host6],
    'collector': [host2, host3],
    'database': [host2, host3, host4],
    'toragent': [host6],
    'tsn': [host6],
    'build': [host_build],
}

env.hostnames = {
    'all': ['nodei34', 'nodei35', 'nodei36', 'nodei37', 'nodei38']
}

root@nodei38:~# dropstats | grep -v " 0"

Discards 2

Invalid NH 1

Invalid VNID 193
Jumbo Mcast Pkt with DF Bit 0

root@nodei38:~#

Sample packets :

root@nodei38:~# tcpdump -ni eth3 udp port 4789 -xx -vv
tcpdump: WARNING: eth3: no IPv4 address assigned
tcpdump: listening on eth3, link-type EN10MB (Ethernet), capture size 65535 bytes
01:02:44.096061 IP (tos 0x0, ttl 64, id 35871, offset 0, flags [DF], proto UDP (17), length 378)
    192.168.1.166.35491 > 192.168.1.6.4789: [no cksum] UDP, length 350
        0x0000: 0025 90e4 0993 5254 0001 0127 0800 4500
        0x0010: 017a 8c1f 4000 4011 2957 c0a8 01a6 c0a8
        0x0020: 0106 8aa3 12b5 0166 0000 0800 0000 0000
        0x0030: 0400 ffff ffff ffff 0001 0000 0ee1 0800
        0x0040: 4510 0148 0000 0000 8011 3996 0000 0000
        0x0050: ffff ffff 0044 0043 0134 3e2f 0101 0600
        0x0060: a419 5a57 008e 0000 0000 0000 0000 0000
        0x0070: 0000 0000 0000 0000 0001 0000 0ee1 0000
        0x0080: 0000 0000 0000 0000 0000 0000 0000 0000
        0x0090: 0000 0000 0000 0000 0000 0000 0000 0000
        0x00a0: 0000 0000 0000 0000 0000 0000 0000 0000
        0x00b0: 0000 0000 0000 0000 0000 0000 0000 0000
        0x00c0: 0000 0000 0000 0000 0000 0000 0000 0000
        0x00d0: 0000 0000 0000 0000 0000 0000 0000 0000
        0x00e0: 0000 0000 0000 0000 0000 0000 0000 0000
        0x00f0: 0000 0000 0000 0000 0000 0000 0000 0000
        0x0100: 0000 0000 0000 0000 0000 0000 0000 0000
        0x0110: 0000 0000 0000 0000 0000 0000 0000 0000
        0x0120: 0000 0000 0000 0000 0000 0000 0000 0000
        0x0130: 0000 0000 0000 0000 0000 0000 0000 0000
        0x0140: 0000 0000 0000 0000 6382 5363 3501 0132
        0x0150: 040a 0101 7f0c 086f 7673 2d76 6d33 3937
        0x0160: 0d01 1c02 030f 0677 0c2c 2f1a 792a ff00
        0x0170: 0000 0000 0000 0000 0000 0000 0000 0000
        0x0180: 0000 0000 0000 0000

01:15:52.830221 IP (tos 0x0, ttl 64, id 36016, offset 0, flags [DF], proto UDP (17), length 78)
    192.168.1.128.43433 > 192.168.1.6.4789: [no cksum] UDP, length 50
        0x0000: 0025 90e4 0993 5254 0001 0101 0800 4500
        0x0010: 004e 8cb0 4000 4011 2a18 c0a8 0180 c0a8
        0x0020: 0106 a9a9 12b5 003a 0000 0800 0000 0000
        0x0030: 0400 ffff ffff ffff 0001 0000 0009 0806
        0x0040: 0001 0800 0604 0001 0001 0000 0009 0a01
        0x0050: 0049 0000 0000 0000 0a01 0564
01:15:53.828104 IP (tos 0x0, ttl 64, id 36017, offset 0, flags [DF], proto UDP (17), length 78)
    192.168.1.128.43433 > 192.168.1.6.4789: [no cksum] UDP, length 50
        0x0000: 0025 90e4 0993 5254 0001 0101 0800 4500
        0x0010: 004e 8cb1 4000 4011 2a17 c0a8 0180 c0a8
        0x0020: 0106 a9a9 12b5 003a 0000 0800 0000 0000
        0x0030: 0400 ffff ffff ffff 0001 0000 0009 0806
        0x0040: 0001 0800 0604 0001 0001 0000 0009 0a01
        0x0050: 0049 0000 0000 0000 0a01 0564

[Hari Prasad Killi (haripk)]

Vedu retested using build42 and didnt see this issue. Issue seems to be resolved with the fixes done for scaling.