3.1.2.0-62:TSN sends BUM traffic to the originating QFX.

Bug #1691647 reported by Sandeep Sridhar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.1
Fix Committed
High
Divakar Dharanalakota
R3.2
Fix Committed
High
Divakar Dharanalakota
R4.0
Fix Committed
High
Divakar Dharanalakota
Trunk
Fix Committed
High
Divakar Dharanalakota

Bug Description

Contrail version: 3.1.2.0-62
QFX version:14.1X53-D33.2

During scale tests, we have discovered that TSN sends BUM traffic back to the originating QFX when the following conditions are met:

1. 100K MAC addresses on ethernet table.
2. All learnt on ovsdb Mac tables.
3. clear ethernet tables on QFX
4. During the re-learning phase, TSN tends to put BUM traffic back to originating QFX. This happens only for certain VNIs though.

I have replicated this issue in my lab.

-Sandeep.

Tags: bms vrouter nttc
information type: Proprietary → Public
tags: added: nttc
Revision history for this message
Sandeep Sridhar (ssandeep) wrote :

It was concluded that we need to implement Split Horizon (do not forward traffic back to the SW where we learnt the dmac through ovs). Manish will fix it by implementing RPF check for unicast lookup. Please let me know what is the target release for this implementation.

Greetings,
Sandeep.

Revision history for this message
vivekananda shenoy (vshenoy83) wrote :

Hi Manish,

Any updates on this fix you are planning to implement ? Can you please provide an ETA for this fix ?

Regards,
Vivek

Revision history for this message
Manish Singh (manishs) wrote :

Needs fix in vrouter to do rpf check for unicast traffic.

Revision history for this message
vivekananda shenoy (vshenoy83) wrote :

Hi Manish,

Any updates on this bug ?

Regards,
Vivek

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/33127
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.1

Review in progress for https://review.opencontrail.org/33161
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R4.0

Review in progress for https://review.opencontrail.org/33166
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2

Review in progress for https://review.opencontrail.org/33167
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/33161
Committed: http://github.com/Juniper/contrail-vrouter/commit/6ee6e1bcb91af4bf0d34a1b73f77eb670341b086
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 6ee6e1bcb91af4bf0d34a1b73f77eb670341b086
Author: Divakar D <email address hidden>
Date: Fri Jun 23 14:34:30 2017 +0530

Dont forward a packet back to same tunnel

In the following transient condition in Vrouter packet is
transmitted back to same Tunnel from which it is received.

Tor bridge entry times out, but this Evpn update is not yet received
by TSN Vrouter. As mac entry is timed out in Tor, any packet destined to
this Mac becomes unknown unicast in Tor and flooded to TSN(along with
other ports of Tor - on which the destination Mac is really lying). In TSN after
decapsulating the Vxlan packet, the bridge lookup potentially point
back to same Tor (as Vxlan tunnel NH), as Evpn delete update is not yet
received by Vrouter. This potentially can cause a loop.

As a fix, the outer tunnel source IP from which the packet is received
is validated against destination tunnel IP to which this packet is about to be
forwarded. If same, packet is dropped by incrementing a new counter.
"dropstats" has been updated to display this new counter. The fix has
been applied to Mpls tunnels as well to make it generic.

Conflicts:
 dp-core/vr_nexthop.c
 dp-core/vr_stats.c
 include/vr_packet.h
 sandesh/vr.sandesh
 utils/vr_util.c

Change-Id: Ia75097376b741d0e75cdf682075cd929003e4994
closes-bug: #1691647

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/33127
Committed: http://github.com/Juniper/contrail-vrouter/commit/58c2ae341bd32e934c74c6f087b56ba66d6110d9
Submitter: Zuul (<email address hidden>)
Branch: master

commit 58c2ae341bd32e934c74c6f087b56ba66d6110d9
Author: Divakar D <email address hidden>
Date: Fri Jun 23 14:34:30 2017 +0530

Dont forward a packet back to same tunnel

In the following transient condition in Vrouter packet is
transmitted back to same Tunnel from which it is received.

Tor bridge entry times out, but this Evpn update is not yet received
by TSN Vrouter. As mac entry is timed out in Tor, any packet destined to
this Mac becomes unknown unicast in Tor and flooded to TSN(along with
other ports of Tor - on which the destination Mac is really lying). In TSN after
decapsulating the Vxlan packet, the bridge lookup potentially point
back to same Tor (as Vxlan tunnel NH), as Evpn delete update is not yet
received by Vrouter. This potentially can cause a loop.

As a fix, the outer tunnel source IP from which the packet is received
is validated against destination tunnel IP to which this packet is about to be
forwarded. If same, packet is dropped by incrementing a new counter.
"dropstats" has been updated to display this new counter. The fix has
been applied to Mpls tunnels as well to make it generic.

Change-Id: Ia75097376b741d0e75cdf682075cd929003e4994
closes-bug: #1691647

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/33166
Committed: http://github.com/Juniper/contrail-vrouter/commit/3d49d324122b0d9306f97e30194f23407f92c680
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 3d49d324122b0d9306f97e30194f23407f92c680
Author: Divakar D <email address hidden>
Date: Fri Jun 23 14:34:30 2017 +0530

Dont forward a packet back to same tunnel

In the following transient condition in Vrouter packet is
transmitted back to same Tunnel from which it is received.

Tor bridge entry times out, but this Evpn update is not yet received
by TSN Vrouter. As mac entry is timed out in Tor, any packet destined to
this Mac becomes unknown unicast in Tor and flooded to TSN(along with
other ports of Tor - on which the destination Mac is really lying). In TSN after
decapsulating the Vxlan packet, the bridge lookup potentially point
back to same Tor (as Vxlan tunnel NH), as Evpn delete update is not yet
received by Vrouter. This potentially can cause a loop.

As a fix, the outer tunnel source IP from which the packet is received
is validated against destination tunnel IP to which this packet is about to be
forwarded. If same, packet is dropped by incrementing a new counter.
"dropstats" has been updated to display this new counter. The fix has
been applied to Mpls tunnels as well to make it generic.

Change-Id: Ia75097376b741d0e75cdf682075cd929003e4994
closes-bug: #1691647

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/33167
Committed: http://github.com/Juniper/contrail-vrouter/commit/f8cd5ea3127c8e6b410805b563ec710d1c6f9720
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit f8cd5ea3127c8e6b410805b563ec710d1c6f9720
Author: Divakar D <email address hidden>
Date: Fri Jun 23 14:34:30 2017 +0530

Dont forward a packet back to same tunnel

In the following transient condition in Vrouter packet is
transmitted back to same Tunnel from which it is received.

Tor bridge entry times out, but this Evpn update is not yet received
by TSN Vrouter. As mac entry is timed out in Tor, any packet destined to
this Mac becomes unknown unicast in Tor and flooded to TSN(along with
other ports of Tor - on which the destination Mac is really lying). In TSN after
decapsulating the Vxlan packet, the bridge lookup potentially point
back to same Tor (as Vxlan tunnel NH), as Evpn delete update is not yet
received by Vrouter. This potentially can cause a loop.

As a fix, the outer tunnel source IP from which the packet is received
is validated against destination tunnel IP to which this packet is about to be
forwarded. If same, packet is dropped by incrementing a new counter.
"dropstats" has been updated to display this new counter. The fix has
been applied to Mpls tunnels as well to make it generic.

Conflicts:
 dp-core/vr_stats.c
 include/vr_packet.h
 sandesh/vr.sandesh
 utils/dropstats.c

Change-Id: Ia75097376b741d0e75cdf682075cd929003e4994
closes-bug: #1691647

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.