Packets are getting dropped for a flow between delete marking and deletion

Bug #1569786 reported by Divakar Dharanalakota
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Fix Committed
Medium
Divakar Dharanalakota
R2.21.x
Fix Committed
Medium
Divakar Dharanalakota
R2.22.x
Fix Committed
Medium
Divakar Dharanalakota
R3.0
Fix Committed
Medium
Divakar Dharanalakota
Trunk
Fix Committed
Medium
Divakar Dharanalakota

Bug Description

After agent marking the flow for delete, if there are any packets hitting that flow entry, all those packets are getting dropped for a considerable time.

Tags: vrouter
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/19265
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/19265
Committed: http://github.org/Juniper/contrail-vrouter/commit/0f8fb1c44c7526f0d742bb92ef8cd03f25f2028a
Submitter: Zuul
Branch: R3.0

commit 0f8fb1c44c7526f0d742bb92ef8cd03f25f2028a
Author: Divakar <email address hidden>
Date: Wed Apr 13 15:06:30 2016 +0530

Create a new flow if a packet hits a delete marked flow

When Agent deletes a flow, the flow entry is not immediately deleted in
Vrouter. Vrouter uses work queues and rcu callbacks to streamline the
deletion. After these asynchronous callbacks, flow entry is treated as
deleted. If there are any packets hitting this flow entry before marking
as "not active" those packets are dropped. On a loaded system with huge
number of flows, these call backs take a considerable time(in the worst
case as much as 30 ms). Packets hitting this flow entry will be dropped
for all this time.

To avoid this issue, when Agent deletes the flow entry, entry is "marked
deleted". A delete marked entry is no more treated as active entry,
hence flow lookup does not return this entry. As a result, a new flow
entry gets created. All the subsequent packets hit the new entry. "flow"
utility displays this particular entry as "Dm".

Change-Id: I0e150bbeb4b3fe3bf3fa1f367e074586363786f2
closes-bug: #1569786

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/19517
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/19572
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/19573
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.21.x

Review in progress for https://review.opencontrail.org/19639
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/19572
Committed: http://github.org/Juniper/contrail-vrouter/commit/d4af42c9c12be657b506b323891f7792a2d390c3
Submitter: Zuul
Branch: R2.20

commit d4af42c9c12be657b506b323891f7792a2d390c3
Author: Divakar <email address hidden>
Date: Wed Apr 13 15:06:30 2016 +0530

Create a new flow if a packet hits a delete marked flow

When Agent deletes a flow, the flow entry is not immediately deleted in
Vrouter. Vrouter uses work queues and rcu callbacks to streamline the
deletion. After these asynchronous callbacks, flow entry is treated as
deleted. If there are any packets hitting this flow entry before marking
as "not active" those packets are dropped. On a loaded system with huge
number of flows, these call backs take a considerable time(in the worst
case as much as 30 ms). Packets hitting this flow entry will be dropped
for all this time.

To avoid this issue, when Agent deletes the flow entry, entry is "marked
deleted". A delete marked entry is no more treated as active entry,
hence flow lookup does not return this entry. As a result, a new flow
entry gets created. All the subsequent packets hit the new entry. "flow"
utility displays this particular entry as "Dm".

closes-bug: #1569786

Conflicts:
 dp-core/vr_flow.c
 utils/flow.c

Change-Id: I0e150bbeb4b3fe3bf3fa1f367e074586363786f2

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/19639
Committed: http://github.org/Juniper/contrail-vrouter/commit/0a59b1c3f2018f1e2d5b6ed8dddf22ed2e55e162
Submitter: Zuul
Branch: R2.21.x

commit 0a59b1c3f2018f1e2d5b6ed8dddf22ed2e55e162
Author: Divakar <email address hidden>
Date: Tue Apr 26 22:11:16 2016 +0530

Create a new flow if a packet hits a delete marked flow

When Agent deletes a flow, the flow entry is not immediately deleted in
Vrouter. Vrouter uses work queues and rcu callbacks to streamline the
deletion. After these asynchronous callbacks, flow entry is treated as
deleted. If there are any packets hitting this flow entry before marking
as "not active" those packets are dropped. On a loaded system with huge
number of flows, these call backs take a considerable time(in the worst
case as much as 30 ms). Packets hitting this flow entry will be dropped
for all this time.

To avoid this issue, when Agent deletes the flow entry, entry is "marked
deleted". A delete marked entry is no more treated as active entry,
hence flow lookup does not return this entry. As a result, a new flow
entry gets created. All the subsequent packets hit the new entry. "flow"
utility displays this particular entry as "M".

Change-Id: Ia40c5a943ea2d06fa01fa3a5a60d52cba1ecfedb
closes-bug: #1569786

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/19517
Committed: http://github.org/Juniper/contrail-vrouter/commit/4c2a100fe1c69bd9db760e803635b8a7e68134e9
Submitter: Zuul
Branch: master

commit 4c2a100fe1c69bd9db760e803635b8a7e68134e9
Author: Divakar <email address hidden>
Date: Wed Apr 13 15:06:30 2016 +0530

Create a new flow if a packet hits a delete marked flow

When Agent deletes a flow, the flow entry is not immediately deleted in
Vrouter. Vrouter uses work queues and rcu callbacks to streamline the
deletion. After these asynchronous callbacks, flow entry is treated as
deleted. If there are any packets hitting this flow entry before marking
as "not active" those packets are dropped. On a loaded system with huge
number of flows, these call backs take a considerable time(in the worst
case as much as 30 ms). Packets hitting this flow entry will be dropped
for all this time.

To avoid this issue, when Agent deletes the flow entry, entry is "marked
deleted". A delete marked entry is no more treated as active entry,
hence flow lookup does not return this entry. As a result, a new flow
entry gets created. All the subsequent packets hit the new entry. "flow"
utility displays this particular entry as "Dm".

Change-Id: I0e150bbeb4b3fe3bf3fa1f367e074586363786f2
closes-bug: #1569786

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/19573
Committed: http://github.org/Juniper/contrail-vrouter/commit/b62fa8d8d73f22f9f6ecde2d84711ffa91888e4b
Submitter: Zuul
Branch: R2.22.x

commit b62fa8d8d73f22f9f6ecde2d84711ffa91888e4b
Author: Divakar <email address hidden>
Date: Wed Apr 13 15:06:30 2016 +0530

Create a new flow if a packet hits a delete marked flow

When Agent deletes a flow, the flow entry is not immediately deleted in
Vrouter. Vrouter uses work queues and rcu callbacks to streamline the
deletion. After these asynchronous callbacks, flow entry is treated as
deleted. If there are any packets hitting this flow entry before marking
as "not active" those packets are dropped. On a loaded system with huge
number of flows, these call backs take a considerable time(in the worst
case as much as 30 ms). Packets hitting this flow entry will be dropped
for all this time.

To avoid this issue, when Agent deletes the flow entry, entry is "marked
deleted". A delete marked entry is no more treated as active entry,
hence flow lookup does not return this entry. As a result, a new flow
entry gets created. All the subsequent packets hit the new entry. "flow"
utility displays this particular entry as "Dm".

closes-bug: #1569786

Conflicts:
 dp-core/vr_flow.c
 utils/flow.c

Change-Id: I0e150bbeb4b3fe3bf3fa1f367e074586363786f2

information type: Proprietary → Public
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.