With repeated hping3 having quick tcp setup/teardown, some flows' pkts are dropped

[Vedamurthy Joshi]

R3.0 Ubuntu 14.04 Kilo multi-node setup

20.1.1.3 and 10.1.1.3 are on two separate VNs connected via policy to allow all traffic with SI(si1, si2) (in-network)

On repeatedly running this below, it seen that sometimes, few pkts get dropped.

hping3 -S -p 22 10.1.1.3 -s 10000 -c 1000 -i u1000

===========
len=44 ip=10.1.1.3 ttl=59 DF id=0 sport=22 flags=SA seq=998 win=14600 rtt=1.7 ms
len=44 ip=10.1.1.3 ttl=59 DF id=0 sport=22 flags=SA seq=999 win=14600 rtt=1.6 ms

--- 10.1.1.3 hping statistic ---
1000 packets transmitted, 996 packets received, 1% packet loss
===========

flow -l suggests that vrouter is dropping it..

root@nodek1:/var/log/contrail# flow -l
Flow table(size 269484032, entries 2105344)

Entries: Created 36078 Added 36072 Processed 36078 Used Overflow entries 0
(Created Flows/CPU: 18071 4639 1114 669 1023 887 1150 566 3 112 18 18 68 10 201 7 1200 577 108 1214 618 583 1595 1437 0 2 4 5 0 0 162 17)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified
TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

    Index Source:Port/Destination:Port Proto(V)
-----------------------------------------------------------------------------------
   267300<=>1444256 20.1.1.3:10640 6 (7)
                         10.1.1.3:22
    (K(nh):66, Action:D(NoSrcRt), Flags:, TCP:S, S(nh):2, Statistics:1/54 UdpSrcPort 59949

  1145268<=>1986676 10.204.216.221:42079 6 (0->3)
                         169.254.0.3:22
    (K(nh):5, Action:N(SD), Flags:, TCP:SSrEEr, S(nh):5, Statistics:24965/1323013 UdpSrcPort 61223

  1281640<=>2072452 20.1.1.3:10640 6 (3->2)
                         10.1.1.3:22
    (K(nh):21, Action:F, Flags:, TCP:S, S(nh):21, Statistics:1/54 UdpSrcPort 52021

  1444256<=>267300 10.1.1.3:22 6 (7)
                         20.1.1.3:10640
    (K(nh):66, Action:D(NoSrcRt), Flags:, TCP:Sr, S(nh):16, Statistics:0/0 UdpSrcPort 49851

  1986676<=>1145268 20.1.1.3:22 6 (3->0)
                         20.1.1.2:42079
    (K(nh):21, Action:N(SD), Flags:, TCP:SSrEEr, S(nh):21, Statistics:40094/9613611 UdpSrcPort 63912

  2072452<=>1281640 10.1.1.3:22 6 (3->2)
                         20.1.1.3:10640
    (K(nh):71, Action:F, Flags:, TCP:Sr, S(nh):71, Statistics:0/0 UdpSrcPort 57363

root@nodek1:/var/log/contrail#

============================
We also see that some flows involving SI (vrf translated flows ) dont get evicted as the corresponding vn1 flow was marked to drop (Ipv4Dis)

default-domain:admin:vn1:service-7365e1af-ce95-4f40-9c73-7608262bef69-default-domain_admin_si1 : vrf 6
default-domain:admin:vn1:service-7365e1af-ce95-4f40-9c73-7608262bef69-default-domain_admin_si2 : vrf 2
default-domain:admin:vn1:vn1 : vrf 1
default-domain:admin:vn2:service-7365e1af-ce95-4f40-9c73-7608262bef69-default-domain_admin_si1 : vrf 4
default-domain:admin:vn2:service-7365e1af-ce95-4f40-9c73-7608262bef69-default-domain_admin_si2 : vrf 5
default-domain:admin:vn2:vn2 : vrf 3

flow -l :

   183284<=>1955480 20.1.1.3:10953 6 (3->5)
                         10.1.1.3:22
    (K(nh):44, Action:F, Flags:, TCP:S, S(nh):44, Statistics:2/108 UdpSrcPort 64709

  1664652 20.1.1.3:10953 6 (1)
                         10.1.1.3:22
    (K(nh):49, Action:D(Ipv4Dis), Flags:, TCP:S, S(nh):49, Statistics:2/108 UdpSrcPort 64028

  1955480<=>183284 10.1.1.3:22 6 (3->5)
                         20.1.1.3:10953
    (K(nh):54, Action:F, Flags:, TCP:Sr, S(nh):54, Statistics:0/0 UdpSrcPort 52798

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/16957
Submitter: Praveen K V (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/16957
Committed: http://github.org/Juniper/contrail-controller/commit/be0553bf98696093427ba14fd7da521d07c6c482
Submitter: Zuul
Branch: master

commit be0553bf98696093427ba14fd7da521d07c6c482
Author: Praveen K V <email address hidden>
Date: Sun Feb 7 10:56:16 2016 +0530

Remove the static route-key members used in FlowEntry

Two static members inet4_route_key_ and inet6_route_key_ were used for
flow optimization. With multi-thread processing of flows, the static
members cannot be used anymore.

Change-Id: Icc35a96fe6197665b2f49a60bf3626f8b8b248da
Closes-Bug: #1542654