few tcp flow entries not deleted forever

[Vedamurthy Joshi]

R2.20 Build 111 Ubuntu 14.04 Juno multi-node setup

Between two vms on the two compute nodes, i was doing continuous hping3 traffic to 1k ports (in a loop)
After the traffic was stopped, one flow was left undeleted (stays forever)

root@nodek1:~# flow -l
Flow table(size 537919488, entries 4202496)

Entries: Created 6187365 Added 6187263 Processed 6187365 Used Overflow entries 0
(Created Flows/CPU: 0 0 0 1 0 0 0 0 718206 714381 680628 679650 725306 716998 719565 720477 0 2 0 0 0 0 0 0 64125 62305 61468 61247 68315 70846 63842 60003)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop, M=Mirror Index
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified
TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

 Index Source:Port Destination:Port Proto(V)
-------------------------------------------------------------------------
369352<=>144172 219.14.75.4:31336 219.14.75.3:2200 6 (6)
(K(nh):51, Action:F, Flags:EcM, TCP:SRD, S(nh):51, Stats:1/54, SPort:53652)

[Vedamurthy Joshi (vedujoshi)]

Another flow

3364564<=>3543304 219.14.75.3:2200 219.14.75.4:42112 6 (6)
(K(nh):51, Action:F, Flags:, TCP:Sr, S(nh):14, Stats:22/1188, SPort:56455)

----------------

root@nodek1:~# flow -l
Flow table(size 537919488, entries 4202496)

Entries: Created 6187385 Added 6187283 Processed 6187385 Used Overflow entries 0
(Created Flows/CPU: 0 0 0 3 0 0 0 0 718206 714383 680628 679651 725307 717001 719566 720479 0 2 0 0 0 0 0 0 64125 62308 61469 61247 68315 70848 63842 60005)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop, M=Mirror Index
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified
TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

 Index Source:Port Destination:Port Proto(V)
-------------------------------------------------------------------------
369352<=>144172 219.14.75.4:31336 219.14.75.3:2200 6 (6)
(K(nh):51, Action:F, Flags:EcM, TCP:SRD, S(nh):51, Stats:1/54, SPort:53652)

1696148<=>3502704 10.204.216.221:54370 169.254.0.4:22 6 (0->6)
(K(nh):5, Action:N(SD), Flags:, TCP:SSrEEr, S(nh):5, Stats:5405/300325, SPort:51206)

3364564<=>3543304 219.14.75.3:2200 219.14.75.4:42112 6 (6)
(K(nh):51, Action:F, Flags:, TCP:Sr, S(nh):14, Stats:22/1188, SPort:56455)

3502704<=>1696148 219.14.75.4:22 169.254.169.254:54370 6 (6->0)
(K(nh):51, Action:N(SD), Flags:, TCP:SSrEEr, S(nh):51, Stats:5063/506877, SPort:52656)

root@nodek1:~#

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/15639
Submitter: Anand H. Krishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/15639
Committed: http://github.org/Juniper/contrail-vrouter/commit/7e8b3550ba4ba410e95f18e9df5650f9ab38b626
Submitter: Zuul
Branch: R2.20

commit 7e8b3550ba4ba410e95f18e9df5650f9ab38b626
Author: Anand H. Krishnan <email address hidden>
Date: Mon Dec 7 11:21:47 2015 +0530

Before initiating eviction, set M(odified) flag

Currently, before evicting the forward flow, we get to the reverse
flow and mark it for eviction. Then we go on to mark forward flow
for eviction and schedule the eviction at a later time. It is in
this deferred functionality that is scheduled that the forward and
the reverse flow gets evicted. We set the M(odified) flag at the
time of 'eviction-marking' in each of the flow entries so that
subsequent changes to the flow entry is not honoured.

It is possible that the forward flow can undergo a change in which
its reverse flow is unlinked (or changed) while the reverse flow
was in the process of getting marked for eviction. If that happens,
at the time of actual eviction, the deferred call will not be able
to find the reverse flow from the forward flow and the reverse flow
will remain in the 'Eviction-Candidate' state forever.

To fix this race, mark the forward flow as M(odified) before
initiating the eviction process, making sure that the mapping between
the forward and the reverse flow is untouched once eviction process
is initiated.

Change-Id: I088ac3abd5c3bfc7bf77efc697bbe1386254358f
Closes-BUG: 1521496

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/15825
Submitter: Anand H. Krishnan (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/15825
Committed: http://github.org/Juniper/contrail-vrouter/commit/48ae1731f8a9354f0115a836d9a37d767c814060
Submitter: Zuul
Branch: master

commit 48ae1731f8a9354f0115a836d9a37d767c814060
Author: Anand H. Krishnan <email address hidden>
Date: Mon Dec 7 11:21:47 2015 +0530

Before initiating eviction, set M(odified) flag

Currently, before evicting the forward flow, we get to the reverse
flow and mark it for eviction. Then we go on to mark forward flow
for eviction and schedule the eviction at a later time. It is in
this deferred functionality that is scheduled that the forward and
the reverse flow gets evicted. We set the M(odified) flag at the
time of 'eviction-marking' in each of the flow entries so that
subsequent changes to the flow entry is not honoured.

It is possible that the forward flow can undergo a change in which
its reverse flow is unlinked (or changed) while the reverse flow
was in the process of getting marked for eviction. If that happens,
at the time of actual eviction, the deferred call will not be able
to find the reverse flow from the forward flow and the reverse flow
will remain in the 'Eviction-Candidate' state forever.

To fix this race, mark the forward flow as M(odified) before
initiating the eviction process, making sure that the mapping between
the forward and the reverse flow is untouched once eviction process
is initiated.

Change-Id: I088ac3abd5c3bfc7bf77efc697bbe1386254358f
Closes-BUG: 1521496