5.0.1: Dynamic mirroring without Juniper header does not work

[Shashikiran H]

queens_5_0_176

This was found as a part of regression runs on 9th Aug 2018.
I have port mirroring dynamic without Juniper header enabled. The packets are not getting mirrored to the analyser. Below is the mirror details

(vrouter-agent)[root@nodem5 /]$ mirror -b
Mirror Table

Flags:D=Dynamic Mirroring, Hw=NIC Assisted Mirroring

Index NextHop Flags VNI Vlan
------------------------------------------------
    0 44 15 0
(vrouter-agent)[root@nodem5 /]$ nh --get 44
Id:44 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:7 Vrf:0
              Flags:Valid, Vxlan, Etree Root,
              Oif:0 Len:14 Data:0c c4 7a dc 42 d7 0c c4 7a dc 44 6b 08 00
              Sip:10.10.11.5 Dip:10.10.11.19

I can see the packet come up on mirror destination compute's proto 17:
for source 222.203.45.3, dst 16.191.195.3, mirror 121.74.8.3:

19:37:26.010705 IP 10.10.11.5.64868 > 10.10.11.19.4789: VXLAN, flags [I] (0x08), vni 15
IP 222.203.45.3 > 16.191.195.3: ICMP echo request, id 6464, seq 3, length 64
        0x0000: 0cc4 7adc 42d7 0cc4 7adc 446b 0800 4500
        0x0010: 0086 1a50 0000 4011 35ec 0a0a 0b05 0a0a
        0x0020: 0b13 fd64 12b5 0072 0000 0800 0000 0000
        0x0030: 0f00 0000 5e00 0100 02df 9553 9f49 0800
        0x0040: 4500 0054 1a50 4000 4001 40c8 decb 2d03
        0x0050: 10bf c303 0800 48a9 1940 0003 1d9c 6d5b
        0x0060: 0000 0000 4249 0a00 0000 0000 1011 1213
        0x0070: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223
        0x0080: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233
        0x0090: 3435 3637

[N Anand Rao (anandrao79)]

Updating the findings based on debugging with Shashi just now.

1) When encap type is VXLAN, the mirrored pkt is reaching the compute node of the analyser vm but not reaching the analyser VM.

23:11:34.579559 0c:c4:7a:dc:44:6b > 0c:c4:7a:dc:42:d7, ethertype IPv4 (0x0800), length 216: 10.10.11.5.54784 > 10.10.11.19.4789: VXLAN, flags [I] (0x08), vni 7
02:5c:95:74:c5:e5 > 00:00:5e:00:01:00, ethertype IPv4 (0x0800), length 166: 2.168.119.3.ssh > 2.168.119.2.60988: Flags [P.], seq 800:900, ack 1, win 588, options [nop,nop,TS val 111106 ecr 41912808], length 100
23:11:35.578872 0c:c4:7a:dc:44:6b > 0c:c4:7a:dc:42:d7, ethertype IPv4 (0x0800), length 148: 10.10.11.5.59136 > 10.10.11.19.4789: VXLAN, flags [I] (0x08), vni 7
02:5c:95:74:c5:e5 > 00:00:5e:00:01:00, ethertype IPv4 (0x0800), length 98: 2.168.119.3 > 89.239.240.3: ICMP echo request, id 2854, seq 55, length 64

Looking at the L2 table, I don’t see a proper NH for the pkt to be sent to the analyser VM.
I am not sure if the dest mac highlighted above should be the analyser VM’s mac for L2 lookup to succeed.

2) It works fine when the encap type is changed to MPLSoUDP. This is because the pkt is sent with correct label 36 which corresponds to the analyser VM tap interface. Hence the mirrored pkts are able to reach the analyser VM.

We need to analyse more why it is not working with VXLAN encap.

[Shashikiran H (skiranh)]

Both working and non-working setups have been provided. Haji and Saurabh are working on this.

[haji mohamed ashraf ali (hajim)]

Nexthop is not programmed properly from agent side, I have given the analysis below.
So request somebody from agent team to triage the issue.

From: Haji Mohamed Ashraf Ali <email address hidden>
Date: Friday, 31 August 2018 at 10:46 AM
To: Sangarshan Pillareddy <email address hidden>, Ashok Singh R <email address hidden>, Nagendra E S <email address hidden>, Harsh Kumar <email address hidden>
Cc: Sivakumar Ganapathy <email address hidden>, "Saurabh Gupta (FT)" <email address hidden>, Anand Narayanan Rao <email address hidden>
Subject: Re: 1786487

++ Harsh kumar

From: Sangarshan Pillareddy <email address hidden>
Date: Tuesday, 21 August 2018 at 3:27 PM
To: Ashok Singh R <email address hidden>, Nagendra E S <email address hidden>
Cc: Sivakumar Ganapathy <email address hidden>, "Saurabh Gupta (FT)" <email address hidden>, Anand Narayanan Rao <email address hidden>, Haji Mohamed Ashraf Ali <email address hidden>
Subject: Re: 1786487

I will take a look,

Regards,
Sangarshan

From: Ashok Singh R <email address hidden>
Date: Tuesday, 21 August 2018 at 2:06 PM
To: Nagendra E S <email address hidden>, Sangarshan Pillareddy <email address hidden>
Cc: Sivakumar Ganapathy <email address hidden>, "Saurabh Gupta (FT)" <email address hidden>, Anand Narayanan Rao <email address hidden>, Haji Mohamed Ashraf Ali <email address hidden>
Subject: Re: 1786487

Nagendra/Sangarsh,

Can one of you help? I don’t have background on this feature. But Haji has both working and non-working setup. So we should start from that.

Regards,
Ashok

From: Haji Mohamed Ashraf Ali <email address hidden>
Date: Tuesday, August 21, 2018 at 1:02 PM
To: Ashok Singh R <email address hidden>
Cc: Sivakumar Ganapathy <email address hidden>, "Saurabh Gupta (FT)" <email address hidden>, Anand Narayanan Rao <email address hidden>, Nagendra E S <email address hidden>
Subject: Bug:1786487

Hi Ashok,

Need your support for the bug: 1786487 analysis are given below

Port mirroring configured as dynamic without juniper enabled and packets are not getting mirrored to the Analyzer.

In Working case(Image: 4.0.2 ):

root@nodeg29:/sys/kernel/debug/tracing# vxlan --dump
VXLAN Table

 VNID NextHop
----------------
      2 16
      6 30
root@nodeg29:/sys/kernel/debug/tracing# nh --get 30
Id:30 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:2
              Flags:Valid, Encap, Etree Root,
              Sub NH(label): 28(23)

Id:28 Type:Encap Fmly:AF_BRIDGE Rid:0 Ref_cnt:4 Vrf:2
              Flags:Valid, Etree Root,
              EncapFmly:0806 Oif:3 Len:14
              Encap Data: 02 7b 29 9c e7 48 00 00 5e 00 01 00 08 00
Setup details:
Src 9.1.1.3 c46
Mirror is 9.1.1.4 g29
Dest 9.1.1.5 c46

In Not working case(Latest Image):

(vrouter-agent)[root@nodem19 /]$ nh --get 42
Id:42 Type:Composite Fmly:AF_BRIDGE Rid:0 Ref_cnt:2 Vrf:2
              Flags:Valid, Encap, Etree Root,
              Sub NH(label): -1
Setup Details:
Ui config nodec22
Control c22, c11, c15
Compute g23, c31
Project test-TestIntfMirror-92859040
Src 80.255.58.3
Dat 148.46.249.3
Mirror 78.86.114.3

From our understanding, Nexth...

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/46726
Submitter: Kumar Harsh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/46728
Submitter: Kumar Harsh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/46726
Submitter: Kumar Harsh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/46728
Submitter: Kumar Harsh (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/46726
Committed: http://github.com/Juniper/contrail-controller/commit/74ece23d689382ea1874dc8c155f97fa199074fe
Submitter: Zuul v3 CI (<email address hidden>)
Branch: master

commit 74ece23d689382ea1874dc8c155f97fa199074fe
Author: Kumar Harsh <email address hidden>
Date: Thu Oct 4 21:33:36 2018 +0530

Composite next hop for mirror destination is having zero component
next hops this because of passing incorrect vm interface mac address
for component next hop key construction.

Fix: derive mac address key from vm_mac() method which is used for
creating interface next hops .

Change-Id: Ic0d0a70903ae3e70a746b81f46d7a1c61f9de51e
Closes-Bug: #1786487

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/46728
Committed: http://github.com/Juniper/contrail-controller/commit/b2601b1ba40c51e3aec9943580a42373103971c9
Submitter: Zuul v3 CI (<email address hidden>)
Branch: R5.0

commit b2601b1ba40c51e3aec9943580a42373103971c9
Author: Kumar Harsh <email address hidden>
Date: Thu Oct 4 21:33:36 2018 +0530

Composite next hop for mirror destination is having zero component
next hops, this because of passing incorrect vm interface mac address
for component next hop key construction.
Fix: derive mac address key from vm_mac() method which is used for
creating interface next hops .

Change-Id: Ic0d0a70903ae3e70a746b81f46d7a1c61f9de51e
Closes-Bug: #1786487

[Shashikiran H (skiranh)]

Works in the recent 5.0 builds.