R4.1 build 15 Netronome build 47: Port mirroring is not working

[Ankit Jain]

Following is the scenario:

Compute/VM: SRC: nodei8 / 199.78.177.3, -> DST: nodel9 / 69.189.97.3 => ANALYZER: nodel9 / 42.51.87.3

Steps:

1) Ping 69.189.97.3 from 199.78.177.3: count =5
2) Enable port mirroring on the port 199.78.177.3

Observation : Expecting 10 packets, received only 2

[ubuntu@169.254.0.3] sudo: ping -s 1200 -c 5 -W 1 69.189.97.3
[ubuntu@169.254.0.3] out: PING 69.189.97.3 (69.189.97.3) 1200(1228) bytes of data.
[ubuntu@169.254.0.3] out: 1208 bytes from 69.189.97.3: icmp_seq=1 ttl=63 time=1.45 ms
[ubuntu@169.254.0.3] out: 1208 bytes from 69.189.97.3: icmp_seq=2 ttl=63 time=0.522 ms
[ubuntu@169.254.0.3] out: 1208 bytes from 69.189.97.3: icmp_seq=3 ttl=63 time=0.550 ms
[ubuntu@169.254.0.3] out: 1208 bytes from 69.189.97.3: icmp_seq=4 ttl=63 time=0.671 ms
[ubuntu@169.254.0.3] out: 1208 bytes from 69.189.97.3: icmp_seq=5 ttl=63 time=0.671 ms
[ubuntu@169.254.0.3] out:

root@ctest-analyzer-vm-80878757:/home/ubuntu#
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
19:08:50.524659 IP 199.78.177.3.61310 > 42.51.87.3.8099: UDP, length 1581
19:08:52.623990 IP 199.78.177.3.50506 > 42.51.87.3.8099: UDP, length 1309

Counters before and after:

root@nodel9:/home/ubuntu# /opt/netronome/libexec/vr_vif_cntrs -v 3
Obtaining VIF Counters for each VIF entry
VIF index 3
 VIF Input bytes=131239
 VIF Input packets=292
 VIF Input errors=10
 VIF Output bytes=3585182
 VIF Output packets=31305
 VIF Output errors=0
root@nodel9:/home/ubuntu#
root@nodel9:/home/ubuntu#
root@nodel9:/home/ubuntu#
root@nodel9:/home/ubuntu#
root@nodel9:/home/ubuntu# /opt/netronome/libexec/vr_vif_cntrs -v 3
Obtaining VIF Counters for each VIF entry
VIF index 3
 VIF Input bytes=137449
 VIF Input packets=297
 VIF Input errors=10
 VIF Output bytes=3593390
 VIF Output packets=31329
 VIF Output errors=0

Netronome team needs to debug it further. Without Netronome acceleration the mirroring works as expected.

[Pieter Malan (pmalan)]

I am not able to reproduce the mirroring issue locally.

We are seeing VRCNTR_WIRE_DROP_RPF_FAIL increments on the Ankit's setup.

            rpf_result = rpf_check(input_vif_index, VIF_TYPE_PHYSICAL);
            if (rpf_result == NH_SOURCE_SEND_TO_HOST) {
                vr_stats_system_out(VRCNTR_WIRE_FBCK_RPF_TO_HOST);
                goto send_to_vr;
            } else if (rpf_result == NH_SOURCE_INVALID) {
                /* Drop the packet. */
                vr_stats_system_out(VRCNTR_WIRE_DROP_RPF_FAIL);
                goto drop;
            }

input_vif_index is used to determine the rpf_result, this issue may be related to https://bugs.launchpad.net/juniperopenstack/+bug/1797358, where the input_vif_index is not valid.

[Ankit Jain (ankitja)]

Seeing the same issue even with nfp_p0 intf.

[Pieter Malan (pmalan)]

Observation is that traffic is mirrored but the source IP is the underlay IP and not the IP of the port mirrored:

Expected Packet:
07:05:19.139958 IP 10.2.0.5.64266 > 10.2.0.6.8099: UDP, length 127

Received Packet:
07:04:11.445307 IP 10.0.109.17.64266 > 10.2.0.6.8099: UDP, length 100

We are busy investigating it internally and working on a fix.

[Jeba Paulaiyan (jebap)]

Hi Pieter,

The bug 1797216 is not seen in the latest Netronome build 82.

I’ve run the following tests to validate the same:

1) TestIntfMirror.test_intf_mirror_src_cn1vn1_dst_cn2vn2_analyzer_cn3vn3
2018-10-25 15:34:59,930 - INFO - END TEST : test_intf_mirror_src_cn1vn1_dst_cn2vn2_analyzer_cn3vn3 : PASSED[0:02:57]

2) TestIntfMirror.test_intf_mirror_with_subintf_src_cn1vn1_dst_cn2vn2_analyzer_cn3vn3
2018-10-255:31:37,749 - INFO - END TEST : test_intf_mirror_with_subintf_src_cn1vn1_dst_cn2vn2_analyzer_cn3vn3 : PASSED[0:03:04]

3) TestIntfMirror.test_intf_mirroring_disable_enable_scenarios
2018-10-25 15:26:41,529 - INFO - END TEST : test_intf_mirroring_disable_enable_scenarios : PASSED[0:06:52]

4) TestIntfMirror.test_juniper_header
2018-10-25 15:38:04,656 - INFO - END TEST : test_juniper_header : PASSED[0:04:43]

5) TestSVCV2MirrorIPv6.test_svc_v2_mirroring
2018-10-25 15:40:42,006 - INFO - END TEST : test_svc_v2_mirroring : PASSED[0:02:20]

6) TestSVCV2Mirror.test_svc_v2_mirroring
2018-10-25 15:43:49,733 - INFO - END TEST : test_svc_v2_mirroring : PASSED[0:02:11]

Regards,
Ankit

[Jeba Paulaiyan (jebap)]

Fixed in Netronome build 82