OpenContrail

(VxLAN OVSDB): Arp is not getting resolved across certain VNIs due to TSN not forwarding the vxaln encap packets to end vtep.

Bug #1483401 reported by Ranjit patro on 2015-08-10

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R2.20	Fix Committed	High	Manish Singh	Juniper Openstack r2.21
Trunk	Fix Committed	High	Manish Singh	Juniper Openstack r3.0-fcs
OpenContrail	Fix Committed	High	Manish Singh

Bug Description

   CONTROL NODE IP: 10.94.189.54
   GUI LOGIN: https://10.94.189.54:8143/#p=config_net_ports
   TSN: 10.94.189.88

                                     (VNI 2300, 2700)
H1=====st-pdt-opus06(97.97.97.97)========================pdt-elit-01(85.85.85.85)====H2
            |
            |
           TSN(12.1.1.2)

  ARP is getting resolved for VNI2300 but is not getting resolved for VNI2700.
  When arp request is initiated from H2-->H1, arp request encapsulated with SRC IP=85.85.85.85 and DST-IP=12.1.1.2 is sent out to TSN.
  tcpdump on TSN i can see the packets coming in. But TSN is not forwarding the packets back to 97.97.97.97.
  For VNI 2300, tcpdump on TSN shows vxlan encap packets correctly but for VNI 2700 i see TSN is not encapsualting the packets backto remote vtep 97.97.97.97 as pasted below.

  BELOW IS OUTPUTS FROM TSN(10.94.189.88) after arp request was initiated from H2--H1 FOR VNI2700
   AS CAN BE SEEN BELOW TSN IS NOT ENCAPSULATING THE PACKET BACK TO 97.97.97.97 AFTER RECIVING BCAST PACKET FROM 85.85.85.85
   FOR VNI 2700.

root@cd-st-lnxserver16:~# tcpdump -ni p1p2 | grep -i 2700
tcpdump: WARNING: p1p2: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on p1p2, link-type EN10MB (Ethernet), capture size 65535 bytes
13:29:36.835379 IP 85.85.85.85.5912 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835395 IP 85.85.85.85.3845 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835408 IP 85.85.85.85.20707 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835439 IP 85.85.85.85.15580 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835470 IP 85.85.85.85.19194 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835503 IP 85.85.85.85.7392 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835538 IP 85.85.85.85.5167 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835571 IP 85.85.85.85.26342 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835573 IP 85.85.85.85.23994 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835608 IP 85.85.85.85.25056 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835696 IP 85.85.85.85.2403 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835735 IP 85.85.85.85.21709 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835783 IP 85.85.85.85.26239 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835787 IP 85.85.85.85.5916 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835815 IP 85.85.85.85.32486 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835855 IP 85.85.85.85.25994 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835912 IP 85.85.85.85.29257 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835955 IP 85.85.85.85.31526 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.835998 IP 85.85.85.85.27746 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:36.836043 IP 85.85.85.85.26348 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:38.836142 IP 85.85.85.85.5912 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:38.836169 IP 85.85.85.85.3845 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:38.836202 IP 85.85.85.85.20707 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:38.836230 IP 85.85.85.85.15580 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700
13:29:38.836260 IP 85.85.85.85.19194 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2700

^C994 packets captured
998 packets received by filter
0 packets dropped by kernel

root@cd-st-lnxserver16:~#

BELOW IS OUTPUTS FROM TSN(10.94.189.88) after arp request was initiated from H2--H1 FOR VNI2300
AS CAN BE SEEN BELOW TSN IS ENCAPSULATING THE PACKET BACK TO 97.97.97.97
====================================================================================

root@cd-st-lnxserver16:~# tcpdump -ni p1p2 | grep -i 2300
tcpdump: WARNING: p1p2: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on p1p2, link-type EN10MB (Ethernet), capture size 65535 bytes
13:22:21.899463 IP 85.85.85.85.21226 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899485 IP 12.1.1.2.63299 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899491 IP 85.85.85.85.30780 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899496 IP 12.1.1.2.61876 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899502 IP 85.85.85.85.22882 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899507 IP 12.1.1.2.50166 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899523 IP 85.85.85.85.19212 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899527 IP 12.1.1.2.51163 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899572 IP 85.85.85.85.27748 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899580 IP 12.1.1.2.62838 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899583 IP 85.85.85.85.25094 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899587 IP 12.1.1.2.53670 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899617 IP 85.85.85.85.1253 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899624 IP 12.1.1.2.62006 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899642 IP 85.85.85.85.7284 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899646 IP 12.1.1.2.60054 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899678 IP 85.85.85.85.16349 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899681 IP 12.1.1.2.63654 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899697 IP 85.85.85.85.16634 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899700 IP 12.1.1.2.59702 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899726 IP 85.85.85.85.15408 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899729 IP 12.1.1.2.55402 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899754 IP 85.85.85.85.19195 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899757 IP 12.1.1.2.49749 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899783 IP 85.85.85.85.30604 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899786 IP 12.1.1.2.57292 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899811 IP 85.85.85.85.3324 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899813 IP 12.1.1.2.56205 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899839 IP 85.85.85.85.28362 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899842 IP 12.1.1.2.62744 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899901 IP 85.85.85.85.25610 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899903 IP 12.1.1.2.51194 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899933 IP 85.85.85.85.12298 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899936 IP 12.1.1.2.52515 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899961 IP 85.85.85.85.581 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899964 IP 12.1.1.2.65297 > 97.97.97.97.4789: VXLAN, flags [I] (0x08), vni 2300
13:22:21.899991 IP 85.85.85.85.20517 > 12.1.1.2.4789: VXLAN, flags [I] (0x08), vni 2300

root@cd-st-lnxserver16:~#

root@cd-st-lnxserver16:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-tor-agent-1 active
contrail-tor-agent-2 active
contrail-tor-agent-3 active
contrail-vrouter-agent active
contrail-vrouter-nodemgr active

========Run time service failures=============
/var/crashes/core.contrail-vroute.12194.cd-st-lnxserver16.1438089702
/var/crashes/core.contrail-vroute.2154.cd-st-lnxserver16.1438739145
/var/crashes/core.contrail-vroute.7206.cd-st-lnxserver16.1438770395
/var/crashes/core.contrail-vroute.2150.cd-st-lnxserver16.1437999440
/var/crashes/core.contrail-vroute.6335.cd-st-lnxserver16.1438109983
/var/crashes/core.contrail-tor-ag.2151.cd-st-lnxserver16.1438023139
/var/crashes/core.contrail-vroute.24050.cd-st-lnxserver16.1438095047
/var/crashes/core.contrail-vroute.18873.cd-st-lnxserver16.1438755472
/var/crashes/core.contrail-vroute.24639.cd-st-lnxserver16.1438783974
/var/crashes/core.contrail-vroute.12406.cd-st-lnxserver16.1438024479
/var/crashes/core.contrail-vroute.16164.cd-st-lnxserver16.1438039358
/var/crashes/core.contrail-vroute.2214.cd-st-lnxserver16.1438045914
/var/crashes/core.contrail-vroute.5746.cd-st-lnxserver16.1438378955
/var/crashes/core.contrail-vroute.6870.cd-st-lnxserver16.1438032845
root@cd-st-lnxserver16:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-fabric-utils 2.20-74 74
contrail-install-packages 2.20-74~icehouse 74
contrail-lib 2.20-74 74
contrail-nodemgr 2.20-74 74
contrail-nova-vif 2.20-74 74
contrail-setup 2.20-74 74
contrail-utils 2.20-74 74
contrail-vrouter-agent 2.20-74 74
contrail-vrouter-common 2.20-74 74
contrail-vrouter-dkms 2.20-74 74
contrail-vrouter-init 2.20-74 74
contrail-vrouter-utils 2.20-74 74
python-contrail 2.20-74 74
python-contrail-vrouter-api 2.20-74 74
python-neutronclient 2:2.3.4-0ubuntu1.2contrail 74
python-nova 1:2014.1.3-0ubuntu1~cloud0.3contrail74
python-opencontrail-vrouter-netns 2.20-74 74
root@cd-st-lnxserver16:~#

Tags:

Revision history for this message

Senthilnathan Murugappan (msenthil) wrote on 2015-08-10:

http://cd-st-lnxserver16.dcbg.juniper.net:8085/Snh_Layer2RouteReq?x=2682
Agent has the configs

however dont see the vxlan id on vrouter
root@cd-st-lnxserver16:/var/log/contrail# vxlan --get 2700
VXLAN Table

VNID NextHop
----------------
Error: No such file or directory

Revision history for this message

Ranjit patro (rpatro) wrote on 2015-08-10:

VNI2700 is not seen in the TSN even though I see the VNI in the Controller UI.
Also all the QFX TORs have the VNI programmed correctly.

root@cd-st-lnxserver16:~# vxlan --get 2300
VXLAN Table

VNID NextHop
----------------
2300 7134

root@cd-st-lnxserver16:~# vxlan --get 2700 >>>>>>>>not present
VXLAN Table

VNID NextHop
----------------
Error: No such file or directory
root@cd-st-lnxserver16:~#

{master:0}
regress@pdt-elit-01> show ovsdb logical-switch Contrail-7a00e61b-2f14-46d7-a4ad-b40e07256564
Logical switch information:
Logical Switch Name: Contrail-7a00e61b-2f14-46d7-a4ad-b40e07256564
Flags: Created by both >>>>>>>>>>>>>>>>>
VNI: 2700
Num of Remote MAC: 1
Num of Local MAC: 1

{master:0}
regress@pdt-elit-01> show vlans Contrail-7a00e61b-2f14-46d7-a4ad-b40e07256564

Routing instance VLAN name Tag Interfaces
default-switch Contrail-7a00e61b-2f14-46d7-a4ad-b40e07256564 NA
                                                           et-0/0/25.2700*
                                                           vtep.32769*
                                                           xe-0/0/6:0.2700*

{master:0}
regress@pdt-elit-01>

{master:0}
regress@st-pdt-opus06> show ovsdb logical-switch Contrail-7a00e61b-2f14-46d7-a4ad-b40e07256564
Logical switch information:
Logical Switch Name: Contrail-7a00e61b-2f14-46d7-a4ad-b40e07256564
Flags: Created by both >>>>>>>>>>
VNI: 2700
Num of Remote MAC: 1
Num of Local MAC: 1

{master:0}
regress@st-pdt-opus06>

Revision history for this message

Manish Singh (manishs) wrote on 2015-08-11:

Agent as well does not have 2700 vxlan id.
This vxlan id is used as automatically allotted VN id for VN(vn-tor-3080) and configured Vxlan for vn-tor-2700.
Probably vxlan mode change or VN update has caused this issue in agent.
Still debugging.

Nagabhushana R (bhushana) on 2015-08-11

Changed in juniperopenstack:
importance:	Undecided → High
Changed in opencontrail:
importance:	Undecided → High
Changed in juniperopenstack:
assignee:	nobody → Manish Singh (manishs)
Changed in opencontrail:
assignee:	nobody → Manish Singh (manishs)
tags:	added: vrouter removed: qfx

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2015-08-13: [Review update] R2.20

Review in progress for https://review.opencontrail.org/13024
Submitter: Manish Singh (<email address hidden>)

Hari Prasad Killi (haripk) on 2015-09-10

tags:

added: releasenote

Revision history for this message

Manish Singh (manishs) wrote on 2015-09-14:

Update: Till date problem was not seen again.

Ashish Ranjan (aranjan-n) on 2015-09-23

Changed in opencontrail:
status:	New → Fix Committed

Revision history for this message

Hari Prasad Killi (haripk) wrote on 2016-02-10:

The change mentioned above is already present in mainline.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.