flow from BMS to KVM VM kept in hold state causing flow-queue-limit drops

[Vedamurthy Joshi]

R2.1 Build 6 Ubuntu 14.04 Multinode setup

BMS : 10.1.1.5
KVM VM on nodeh5 : 10.1.1.3

Pings from 10.1.1.5 to 10.1.1.3 failed.

env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6, host7],
    'cfgm': [host1,host2,host3],
    'openstack': [host2],
    'control': [host1,host2,host3],
    'compute': [host4,host5, host6, host7],
    'collector': [host1,host2,host3],
    'webui': [host1],
    'database': [host1,host2,host3],
    'toragent': [host6],
    'tsn': [host6],
    'build': [host_build],
}

env.hostnames = {
    'all': ['nodec22', 'nodeg30', 'nodeg29', 'nodeh4', 'nodeh5', 'nodeg11', 'nodeh3-compute-vm1']
}

root@nodeh5:~# dropstats | grep -v " 0"

Flow Action Drop 98
Flow Queue Limit Exceeded 28830 <<<<<<<<<<

Cloned Original 3577

Invalid NH 2
Invalid Label 4

Duplicated 114711

Misc 9
Jumbo Mcast Pkt with DF Bit 0

root@nodeh5:~#

root@nodeh5:~# flow -l
Flow table

 Index Source:Port Destination:Port Proto(V)
-------------------------------------------------------------------------
218096 10.1.1.3:9082 10.1.1.5:0 1 (1)
        (K(nh):17, Action:F, S(nh):20, Statistics:0/0 UdpSrcPort 54039)

352928 10.1.1.5:4991 10.1.1.3:0 1 (1)
        (K(nh):17, Action:H, S(nh):0, Statistics:28692/2811816 UdpSrcPort 0)

396264 10.1.1.3:4991 10.1.1.5:0 1 (1)
        (K(nh):17, Action:F, S(nh):20, Statistics:0/0 UdpSrcPort 53187)

419132 10.1.1.5:9082 10.1.1.3:0 1 (1)
        (K(nh):17, Action:H, S(nh):0, Statistics:11/1078 UdpSrcPort 0)

root@nodeh5:~#

Agent trace :

2015-01-20 01:00:42.827 FlowTrace: operation = Add info= [ flow_index = 4294967295 source_ip = 20.1.1.6 source_port = 1342 destination_ip = 10.1.1.3 destination_port = 0 protocol = 1 vrf = 1 mirror_l= [ [ ] ] mirror_vrf = 65535 implicit_deny = 0 short_flow = 1 source_vn = __UNKNOWN__ dest_vn = default-domain:admin:vn1-4c3239e6-c8f7-4f0c-a05f-5c0140d46c3f source_sg_id_l= [ [ ] ] dest_sg_id_l= [ [ (*_iter55) = 1, ] ] nh_id = 17 vrf_assign = l3_flow = 1 smac = 00:00:5e:00:01:00 dmac = 02:fe:90:81:78:1d ] file = controller/src/vnsw/agent/pkt/flow_table.cc line = 843
2015-01-20 01:00:42.827 FlowTrace: operation = Add info= [ flow_index = 313224 source_ip = 10.1.1.3 source_port = 1342 destination_ip = 20.1.1.6 destination_port = 0 protocol = 1 vrf = 1 mirror_l= [ [ ] ] mirror_vrf = 65535 implicit_deny = 0 short_flow = 1 source_vn = default-domain:admin:vn1-4c3239e6-c8f7-4f0c-a05f-5c0140d46c3f dest_vn = __UNKNOWN__ source_sg_id_l= [ [ (*_iter54) = 1, ] ] dest_sg_id_l= [ [ ] ] nh_id = 17 vrf_assign = l3_flow = 1 smac = 02:fe:90:81:78:1d dmac = 00:00:5e:00:01:00 ] file = controller/src/vnsw/agent/pkt/flow_table.cc line = 843
2015-01-20 01:00:43.113 FlowTrace: operation = Delete info= [ flow_index = 313224 source_ip = 10.1.1.3 source_port = 1342 destination_ip = 20.1.1.6 destination_port = 0 protocol = 1 vrf = 1 mirror_l= [ [ ] ] mirror_vrf = 65535 implicit_deny = 0 short_flow = 1 source_vn = default-domain:admin:vn1-4c3239e6-c8f7-4f0c-a05f-5c0140d46c3f dest_vn = __UNKNOWN__ source_sg_id_l= [ [ (*_iter54) = 1, ] ] dest_sg_id_l= [ [ ] ] nh_id = 17 vrf_assign = l3_flow = 1 smac = 02:fe:90:81:78:1d dmac = 00:00:5e:00:01:00 ] file = controller/src/vnsw/agent/pkt/flow_table.cc line = 1519
2015-01-20 01:00:43.113 FlowTrace: operation = Delete info= [ flow_index = 73252 source_ip = 20.1.1.6 source_port = 1342 destination_ip = 10.1.1.3 destination_port = 0 protocol = 1 vrf = 1 mirror_l= [ [ ] ] mirror_vrf = 65535 implicit_deny = 0 short_flow = 1 source_vn = __UNKNOWN__ dest_vn = default-domain:admin:vn1-4c3239e6-c8f7-4f0c-a05f-5c0140d46c3f source_sg_id_l= [ [ ] ] dest_sg_id_l= [ [ (*_iter55) = 1, ] ] nh_id = 17 vrf_assign = l3_flow = 1 smac = 00:00:5e:00:01:00 dmac = 02:fe:90:81:78:1d ] file = controller/src/vnsw/agent/pkt/flow_table.cc line = 1519
2015-01-20 01:00:43.835 FlowDetailErr: 313224 3 1 167837955 335610118 Flow : No route for Dst-IP 1 controller/src/vnsw/agent/pkt/pkt_flow_info.cc 41

[Manish Singh (manishs)]

Fixed under:
https://review.opencontrail.org/#/c/6392/

[Hari Prasad Killi (haripk)]

mainline : https://github.com/Juniper/contrail-controller/commit/f219a4e58f2209009a33707b510ed72ec803f276