vrouter: First VM-VM ping packet dropped when size is > 3000 bytes

[Vinod Nair]

On sending pkts of size > 6K looks like not all fragments are not sent

ping 22.1.0.4 -s 6000 -c 1
PING 22.1.0.4 (22.1.0.4) 6000(6028) bytes of data.

--- 22.1.0.4 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

Sending End VIF

root@cmbu-starwars-01:/sv02/dpdkvm/tso# vifdump 3 -nevv host 22.1.0.4
tcpdump: WARNING: mon3: no IPv4 address assigned
tcpdump: listening on mon3, link-type EN10MB (Ethernet), capture size 65535 bytes
00:23:30.196802 02:9f:c4:bd:3d:c4 > 02:bb:ff:a2:b1:48, ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 64, id 37041, offset 0, flags [+], proto ICMP (1), length 1500)
    22.1.0.3 > 22.1.0.4: ICMP echo request, id 1481, seq 1, length 1480
00:23:30.196802 02:9f:c4:bd:3d:c4 > 02:bb:ff:a2:b1:48, ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 64, id 37041, offset 1480, flags [+], proto ICMP (1), length 1500)
    22.1.0.3 > 22.1.0.4: ip-proto-1
00:23:30.196803 02:9f:c4:bd:3d:c4 > 02:bb:ff:a2:b1:48, ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 64, id 37041, offset 2960, flags [+], proto ICMP (1), length 1500)
    22.1.0.3 > 22.1.0.4: ip-proto-1
00:23:30.196804 02:9f:c4:bd:3d:c4 > 02:bb:ff:a2:b1:48, ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 64, id 37041, offset 4440, flags [+], proto ICMP (1), length 1500)
    22.1.0.3 > 22.1.0.4: ip-proto-1
00:23:30.196804 02:9f:c4:bd:3d:c4 > 02:bb:ff:a2:b1:48, ethertype IPv4 (0x0800), length 122: (tos 0x0, ttl 64, id 37041, offset 5920, flags [none], proto ICMP (1), length 108)
    22.1.0.3 > 22.1.0.4: ip-proto-1

00:23:35.208802 02:9f:c4:bd:3d:c4 > 02:bb:ff:a2:b1:48, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 22.1.0.4 tell 22.1.0.3, length 28
00:23:35.208802 02:bb:ff:a2:b1:48 > 02:9f:c4:bd:3d:c4, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Reply 22.1.0.4 is-at 02:bb:ff:a2:b1:48, length 28
00:24:00.260824 02:bb:ff:a2:b1:48 > 02:9f:c4:bd:3d:c4, ethertype IPv4 (0x0800), length 590: (tos 0xc0, ttl 64, id 45313, offset 0, flags [none], proto ICMP (1), length 576)
    22.1.0.4 > 22.1.0.3: ICMP ip reassembly time exceeded, length 556
        (tos 0x0, ttl 64, id 37041, offset 0, flags [+], proto ICMP (1), length 1500)
    22.1.0.3 > 22.1.0.4: ICMP echo request, id 1481, seq 1, length 1480

root@cmbu-starwars-01:/sv02/dpdkvm/tso# vif --get 3
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload, Df=Drop New Flows, L=MAC Learning Enabled
       Proxy=MAC Requests Proxied Always, Er=Etree Root

vif0/3 PMD: tap9fc4bd3d-c4
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
            Vrf:1 Flags:PL3L2DEr QOS:-1 Ref:15
            RX port packets:9076378 errors:0 syscalls:2
            RX queue packets:9054547 errors:8
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 8
            RX packets:9076370 bytes:1998598102 errors:0
            TX packets:5205749 bytes:90195743191 errors:0
            Drops:165013
            TX port packets:9948331 errors:316888 syscalls:7162779

Receiver VM:
==================

00:23:27.973029 02:9f:c4:bd:3d:c4 > 02:bb:ff:a2:b1:48, ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 64, id 37041, offset 0, flags [+], proto ICMP (1), length 1500)
    22.1.0.3 > 22.1.0.4: ICMP echo request, id 1481, seq 1, length 1480
00:23:27.973046 02:9f:c4:bd:3d:c4 > 02:bb:ff:a2:b1:48, ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 64, id 37041, offset 2960, flags [+], proto ICMP (1), length 1500)
    22.1.0.3 > 22.1.0.4: ip-proto-1
00:23:27.973048 02:9f:c4:bd:3d:c4 > 02:bb:ff:a2:b1:48, ethertype IPv4 (0x0800), length 1514: (tos 0x0, ttl 64, id 37041, offset 1480, flags [+], proto ICMP (1), length 1500)
    22.1.0.3 > 22.1.0.4: ip-proto-1

00:23:58.035959 02:bb:ff:a2:b1:48 > 02:9f:c4:bd:3d:c4, ethertype IPv4 (0x0800), length 590: (tos 0xc0, ttl 64, id 45313, offset 0, flags [none], proto ICMP (1), length 576)
    22.1.0.4 > 22.1.0.3: ICMP ip reassembly time exceeded, length 556
        (tos 0x0, ttl 64, id 37041, offset 0, flags [+], proto ICMP (1), length 1500)
    22.1.0.3 > 22.1.0.4: ICMP echo request, id 1481, seq 1, length 1480

Version: 4.0.0.0-3054~mitaka

[Kiran (kiran-kn80)]

It works for me. Can you pl recreate and provide the setup.

[Vinod Nair (vinodnair)]

Still seen in 4.0 Build 10 Also

sender
ping 22.1.0.3 -s 9000 -c 1
PING 22.1.0.3 (22.1.0.3) 9000(9028) bytes of data.

--- 22.1.0.3 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

Receice compute

vif0
23:18:27.100920 00:25:90:e5:ae:90 > 00:25:90:e5:b8:54, ethertype IPv4 (0x0800), length 1506: (tos 0x0, ttl 64, id 52658, offset 0, flags [none], proto UDP (17), length 1492)
    8.0.0.3.52909 > 8.0.0.2.6635: [no cksum] UDP, length 1464
23:18:27.100920 00:25:90:e5:ae:90 > 00:25:90:e5:b8:54, ethertype IPv4 (0x0800), length 106: (tos 0x0, ttl 64, id 52658, offset 0, flags [none], proto UDP (17), length 92)
    8.0.0.3.52909 > 8.0.0.2.6635: [no cksum] UDP, length 64
23:18:27.100920 00:25:90:e5:ae:90 > 00:25:90:e5:b8:54, ethertype IPv4 (0x0800), length 106: (tos 0x0, ttl 64, id 52658, offset 0, flags [none], proto UDP (17), length 92)
    8.0.0.3.52909 > 8.0.0.2.6635: [no cksum] UDP, length 64
23:18:27.100921 00:25:90:e5:ae:90 > 00:25:90:e5:b8:54, ethertype IPv4 (0x0800), length 1506: (tos 0x0, ttl 64, id 52658, offset 0, flags [none], proto UDP (17), length 1492)
    8.0.0.3.52909 > 8.0.0.2.6635: [no cksum] UDP, length 1464
23:18:27.100921 00:25:90:e5:ae:90 > 00:25:90:e5:b8:54, ethertype IPv4 (0x0800), length 106: (tos 0x0, ttl 64, id 52658, offset 0, flags [none], proto UDP (17), length 92)
    8.0.0.3.52909 > 8.0.0.2.6635: [no cksum] UDP, length 64
23:18:27.100921 00:25:90:e5:ae:90 > 00:25:90:e5:b8:54, ethertype IPv4 (0x0800), length 1506: (tos 0x0, ttl 64, id 52658, offset 0, flags [none], proto UDP (17), length 1492)
    8.0.0.3.52909 > 8.0.0.2.6635: [no cksum] UDP, length 1464
23:18:29.064906 00:25:90:e5:b8:54 > 00:25:90:e5:ae:90, ethertype 802.1Q (0x8100), length 626: vlan 2003, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 26, offset 0, flags [none], proto UDP (17), length 608, bad cksum 0 (->686f)!)
    8.0.0.2.49839 > 8.0.0.3.6635: [no cksum] UDP, length 580

In receive VM
================
23:18:26.606186 00:00:5e:00:01:00 > 02:36:fa:27:98:ef, ethertype IPv4 (0x0800), length 1474: (tos 0x0, ttl 63, id 52658, offset 0, flags [+], proto ICMP (1), length 1460)
    22.2.0.3 > 22.1.0.3: ICMP echo request, id 1504, seq 1, length 1440
23:18:26.606193 00:00:5e:00:01:00 > 02:36:fa:27:98:ef, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 52658, offset 1440, flags [+], proto ICMP (1), length 60)
    22.2.0.3 > 22.1.0.3: ip-proto-1
23:18:26.606195 00:00:5e:00:01:00 > 02:36:fa:27:98:ef, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 52658, offset 2920, flags [+], proto ICMP (1), length 60)
    22.2.0.3 > 22.1.0.3: ip-proto-1
23:18:28.570769 02:36:fa:27:98:ef > 00:00:5e:00:01:00, ethertype IPv4 (0x0800), length 590: (tos 0xc0, ttl 64, id 13965, offset 0, flags [none], proto ICMP (1), length 576)
    22.1.0.3 > 22.2.0.3: ICMP ip reassembly time exceeded, length 556
        (tos 0x0, ttl 63, id 46303, offset 0, flags [+], proto ICMP (1), length 1460)
    22.2.0.3 > 22.1.0.3: ICMP echo request, id 1503, seq 1, length 1440

[Raja Sivaramakrishnan (raja-u)]

Release note: When the DPDK vrouter needs to fragment packets before sending them on the wire, the reassembly of the fragments on the receiver may timeout in some cases.

[Kiran (kiran-kn80)]

This is happening in both kernel vrouter and DPDK vrouter. I see the error counter "Flow Queue Limit Exceeded" incrementing for the first ping packet. Subsequent ping packets go through fine.

[Kiran (kiran-kn80)]

Hari/Divakar: If I incrase the VR_MAX_FLOW_QUEUE_ENTRIES to say 10, this issue goes away. It happens for large size ping packets only.

[Hari Prasad Killi (haripk)]

Vrouter only stores upto three fragments while flow setup is happening. When VM sends more than three fragments, the additional fragments are dropped and that is the problem seen. Since this will not be seen in TCP & UDP cases (due to offload) and only seen for big ICMP packets, I propose not to do anything for this case.