Stale flows left behind after long running tcp test

[Vinod Nair]

Stale flows left behind after long running tcp traffic test.

2016-08-10 10:36:39 -0700
Flow Statistics
---------------
    Total Entries --- Total = 10765, new = 0
    Active Entries --- Total = 10765, new = 0
    Hold Entries --- Total = 0, new = 0
    Fwd flow Entries - Total = 10765
    drop flow Entries - Total = 0
    NAT flow Entries - Total = 0

    Rate of change of Active Entries
    --------------------------------
        current rate = 0
        Avg setup rate = 20943
        Avg teardown rate = 0
    Rate of change of Flow Entries
    ------------------------------
        current rate = 0

2016-08-10 10:45:53 -0700
Flow Statistics
---------------
    Total Entries --- Total = 10765, new = 0
    Active Entries --- Total = 10765, new = 0
    Hold Entries --- Total = 0, new = 0
    Fwd flow Entries - Total = 10765
    drop flow Entries - Total = 0
    NAT flow Entries - Total = 0

    Rate of change of Active Entries
    --------------------------------
        current rate = 0
        Avg setup rate = 21107
        Avg teardown rate = 0
    Rate of change of Flow Entries
    ------------------------------
        current rate = 0

Also flow -l has some entries as below . flow -l output attached,
cat /tmp/flow.txt | more
Flow table(size 80609280, entries 629760)

Entries: Created 395527966 Added 395495716 Processed 395527966 Used Overflow entries 10763
(Created Flows/CPU: 0 0 0 0 0 0 0 0 0 0 63053703 131778222 69209480 131486561)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified Dm=Delete Marked
TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

    Index Source:Port/Destination:Port Proto(V)
-----------------------------------------------------------------------------------
   151
     Oflow entries:
         613598

   483
     Oflow entries:
         530339

   551
     Oflow entries:
         613343

   651
     Oflow entries:
         544667

   691
     Oflow entries:
         596988

Version : 3.1 Build 19 Mitaka

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23208
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23208
Committed: http://github.org/Juniper/contrail-controller/commit/52ffcdbdde5b32a7c868c59dd370b13e827ea96f
Submitter: Zuul
Branch: R3.1

commit 52ffcdbdde5b32a7c868c59dd370b13e827ea96f
Author: Prabhjot Singh Sethi <email address hidden>
Date: Thu Aug 11 16:35:52 2016 +0530

Fix flows stuck in vrouter for ever

Issue:
------
Stats collector and flow processing run in parallel
so flow handle and gen id not being the key can change
while stats collector aging process, due to which it
can run eviction validation using old flow handle and
gen id, while enqueue an eviction request using new
flow handle and gen id, resulting in skipping delete
from vrouter and causing lingering flows

Fix:
----
Lock and fetch flow handle and gen idat the start of
eviction processing and use the same while enqueuing
eviction request

Closes-Bug: 1611881
Change-Id: I9c7aa4bfee0b82334e92d5a748d1b2045e3976df

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23621
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23622
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23623
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23621
Committed: http://github.org/Juniper/contrail-controller/commit/a82202465a83c774cc1dbf0e31d32f963b8febfa
Submitter: Zuul
Branch: master

commit a82202465a83c774cc1dbf0e31d32f963b8febfa
Author: Prabhjot Singh Sethi <email address hidden>
Date: Thu Aug 11 16:35:52 2016 +0530

Fix flows stuck in vrouter for ever

Issue:
------
Stats collector and flow processing run in parallel
so flow handle and gen id not being the key can change
while stats collector aging process, due to which it
can run eviction validation using old flow handle and
gen id, while enqueue an eviction request using new
flow handle and gen id, resulting in skipping delete
from vrouter and causing lingering flows

Fix:
----
Lock and fetch flow handle and gen idat the start of
eviction processing and use the same while enqueuing
eviction request

Closes-Bug: 1611881
Change-Id: I9c7aa4bfee0b82334e92d5a748d1b2045e3976df
(cherry picked from commit 52ffcdbdde5b32a7c868c59dd370b13e827ea96f)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23622
Committed: http://github.org/Juniper/contrail-controller/commit/68b8416638340ac90a50417d895e4c65709bd9d3
Submitter: Zuul
Branch: R3.0

commit 68b8416638340ac90a50417d895e4c65709bd9d3
Author: Prabhjot Singh Sethi <email address hidden>
Date: Thu Aug 11 16:35:52 2016 +0530

Fix flows stuck in vrouter for ever

Issue:
------
Stats collector and flow processing run in parallel
so flow handle and gen id not being the key can change
while stats collector aging process, due to which it
can run eviction validation using old flow handle and
gen id, while enqueue an eviction request using new
flow handle and gen id, resulting in skipping delete
from vrouter and causing lingering flows

Fix:
----
Lock and fetch flow handle and gen idat the start of
eviction processing and use the same while enqueuing
eviction request

Closes-Bug: 1611881
Change-Id: I9c7aa4bfee0b82334e92d5a748d1b2045e3976df
(cherry picked from commit 52ffcdbdde5b32a7c868c59dd370b13e827ea96f)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23623
Committed: http://github.org/Juniper/contrail-controller/commit/3e2e00190ee1c4e3a00b784b17d1d21cdd82762a
Submitter: Zuul
Branch: R3.0.2.x

commit 3e2e00190ee1c4e3a00b784b17d1d21cdd82762a
Author: Prabhjot Singh Sethi <email address hidden>
Date: Thu Aug 11 16:35:52 2016 +0530

Fix flows stuck in vrouter for ever

Issue:
------
Stats collector and flow processing run in parallel
so flow handle and gen id not being the key can change
while stats collector aging process, due to which it
can run eviction validation using old flow handle and
gen id, while enqueue an eviction request using new
flow handle and gen id, resulting in skipping delete
from vrouter and causing lingering flows

Fix:
----
Lock and fetch flow handle and gen idat the start of
eviction processing and use the same while enqueuing
eviction request

Closes-Bug: 1611881
Change-Id: I9c7aa4bfee0b82334e92d5a748d1b2045e3976df
(cherry picked from commit 52ffcdbdde5b32a7c868c59dd370b13e827ea96f)