[ R3.1 build 71 mitaka] Svc mirroring : Not all icmp response pkts mirrored

[Ankit Jain]

Issue :

Svc mirrroring v1 transparent : Not all icmp response pkts getting mirrored, ping count set to 5

total pkts mirrored 11, expected 20, only 1 icmp response pkt mirrored

src vm and dst vm (and analyzer on the CN same as of dst vm) on different Compute nodes, ping count set to 5.

The problem could be seen when one of src or dst vm is on the compute same as of analyzer ( In this case, src on CN1, dst on CN2, analyzer on CN2)

script : test_svc_mirroring failing due to this issue.

pkt cap on Src vm:
root@ctest-in-network-vm1-47516529:/home/ubuntu# tcpdump -ni eth0 icmp

07:51:27.078104 IP 93.211.251.3 > 155.188.99.3: ICMP echo request, id 9505, seq 1, length 64
07:51:27.079649 IP 155.188.99.3 > 93.211.251.3: ICMP echo reply, id 9505, seq 1, length 64
07:51:28.079944 IP 93.211.251.3 > 155.188.99.3: ICMP echo request, id 9505, seq 2, length 64
07:51:28.080246 IP 155.188.99.3 > 93.211.251.3: ICMP echo reply, id 9505, seq 2, length 64
07:51:29.078948 IP 93.211.251.3 > 155.188.99.3: ICMP echo request, id 9505, seq 3, length 64
07:51:29.079154 IP 155.188.99.3 > 93.211.251.3: ICMP echo reply, id 9505, seq 3, length 64
07:51:30.078244 IP 93.211.251.3 > 155.188.99.3: ICMP echo request, id 9505, seq 4, length 64
07:51:30.078457 IP 155.188.99.3 > 93.211.251.3: ICMP echo reply, id 9505, seq 4, length 64
07:51:31.078240 IP 93.211.251.3 > 155.188.99.3: ICMP echo request, id 9505, seq 5, length 64
07:51:31.078443 IP 155.188.99.3 > 93.211.251.3: ICMP echo reply, id 9505, seq 5, length 64

Ping from 93.211.251.3 to 155.188.99.3 with count = 5

Pkt cap nn Analyzer:

ubuntu@ctest-mirror-si-19622030-1001:~$ tcpdump -ni eth0 -U udp port 8099

07:51:27.246921 IP 155.188.99.3.28616 > 10.250.2.252.8099: UDP, length 262
07:51:27.246975 IP 93.211.251.3.29643 > 10.250.2.252.8099: UDP, length 262
07:51:27.247559 IP 93.211.251.3.59486 > 10.250.2.252.8099: UDP, length 262
07:51:28.247626 IP 93.211.251.3.59486 > 10.250.2.252.8099: UDP, length 262
07:51:28.248115 IP 93.211.251.3.61644 > 10.250.2.252.8099: UDP, length 262
07:51:29.247275 IP 93.211.251.3.61644 > 10.250.2.252.8099: UDP, length 262
07:51:29.247372 IP 93.211.251.3.59486 > 10.250.2.252.8099: UDP, length 262
07:51:30.246563 IP 93.211.251.3.61644 > 10.250.2.252.8099: UDP, length 262
07:51:30.246666 IP 93.211.251.3.59486 > 10.250.2.252.8099: UDP, length 262
07:51:31.246713 IP 93.211.251.3.61644 > 10.250.2.252.8099: UDP, length 262
07:51:31.246797 IP 93.211.251.3.59486 > 10.250.2.252.8099: UDP, length 262

Service Instance Details
Instance Name ctest-mirror_si-07277868_1
Display Name ctest-mirror_si-07277868_1
UUID 99bd503c-090f-40aa-a62a-b1e3e4f05a58
Template ctest-st1-05792197 (transparent, version 1)
# Instance(s) 1
Networks Left: Automatic
Image ubuntu-traffic
Flavor contrail_flavor_small
Availability Zone ANY:ANY
Instance Status
Virtual Machine Status Power State Networks
ctest-mirror_si-07277868_1001 ACTIVE RUNNING svc-vn-left:10.250.2.252
View Console
Interface Status
Interface Status Health Status IP Address
default-domain__ctest-TestSVCMirror-88652948__ctest-mirror_si-07277868_1__1__left__1 Active - 10.250.2.252

Permissions
Owner 60dad8575b0f40ac98b2ca2071ff55b7
Owner Permissions Read, Write, Refer
Global Permissions -
Shared List -

Service Templates

ctest-st1-05792197TransparentAnalyzer / v1Leftubuntu-traffic / contrail_flavor_small
Details
Name ctest-st1-05792197
Display Name ctest-st1-05792197
UUID a6a605cc-97b7-425c-b2f4-0aa3373ca03d
Version 1
Mode Transparent
Type Analyzer / v1
Scaling Disabled
Availability Zone Disabled
Interface Type (s) Left (Shared IP)
Image ubuntu-traffic
Flavor contrail_flavor_small
Service Instances ctest-TestSVCMirror-88652948:ctest-mirror_si-07277868_1

Permissions
Owner 60dad8575b0f40ac98b2ca2071ff55b7
Owner Permissions Read, Write, Refer
Global Permissions -
Shared List
Project Permissions
domain:468f568a-59fd-4262-b84f-6cc1f1a47436 Read, Refer

[Ankit Jain (ankitja)]

The issue is seen for port mirroring also for the following scenario:

src_vm on cn1vn1, dst_vm on cn2vn1 and analyzer vm on cn1vn1

step1) port mirroring enabled on the port of src vm
step2) ping dst vm from src vm with count = 5

Observation : AssertionError: 1 ICMP Packets mirrored to the analyzer VM ctest-analyzer_vm-57143999,Expected 10 packets

When promiscuous mode was enabled: The script passed as expected
10 ICMP packets are mirrored to the analyzer service VM 'ctest-analyzer_vm-29179249'

[Gadi Leshem (gleshem)]

Can you please provide troubleshoot ?

/opt/netronome/bin/vrouter_troubleshoot.sh -V

[Ankit Jain (ankitja)]

Attached the troubleshoot.

The same issue is seen with svc v2 and port mirroring.

[Jon Hickman (jhickman8x3)]

We have replicated the BUG and have identified where the issue is. @e are working on a fix.

Our offload vif->mtu value for a virtual interface is not being populated like the physical interfaces.

[Johan Moraal (jmoraal)]

This bug has been fixed and committed in the Netronome repos. It will be included in Netronome's next release to Juniper