Juniper Openstack

DPDK: vhost0 with jumbo mtu traffic across / to computes stall

  1. Series trunk
  2. Bug #1547782
Bug #1547782 reported by Vinod Nair
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.0
Fix Committed
Critical
Raja Sivaramakrishnan
Juniper Openstack r3.0.1.0
Trunk
Fix Committed
Critical
Raja Sivaramakrishnan
Juniper Openstack r3.1.0.0-fcs

Bug Description

On A DPDK cluster with jumbo mtu set , image copy across computes stall
vhost 0 is set to 9K mtu

root@cs-scale-7:~# ifconfig vhost0
vhost0 Link encap:Ethernet HWaddr 90:e2:ba:4c:63:f0
          inet addr:13.1.0.7 Bcast:13.1.0.255 Mask:255.255.255.0
          inet6 addr: fe80::92e2:baff:fe4c:63f0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
          RX packets:959358 errors:0 dropped:0 overruns:0 frame:0
          TX packets:208715 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1268164858 (1.2 GB) TX bytes:192388121 (192.3 MB)

cs-scale-4:/store# scp server.qcow2 root@13.1.0.7:/tmp/
server.qcow2 0% 2112KB 317.0KB/s - stalled -^server.qcow2 0% 2112KB 285.3KB/s - stalled -roo

Tcpdump from the compute is attached

root@cs-scale-7:~# vif --list ; sleep 60; vif --list
Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/0 PCI: 0:0:0.0 (Speed 10000, Duplex 1)
            Type:Physical HWaddr:90:e2:ba:4c:63:f0 IPaddr:0
            Vrf:0 Flags:L3L2Vp MTU:1514 Ref:21
            RX device packets:0 bytes:0 errors:10
            RX port packets:5143389927 errors:0
            RX queue packets:5143357509 errors:10
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 10
            RX packets:5143389917 bytes:1153171663200 errors:0
            TX packets:5142452783 bytes:1152055991047 errors:0
            TX port packets:5142452799 errors:7

vif0/1 Virtual: vhost0
            Type:Host HWaddr:90:e2:ba:4c:63:f0 IPaddr:0
            Vrf:0 Flags:L3L2 MTU:1514 Ref:10
            RX port packets:213184 errors:0
            RX queue packets:213123 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:213184 bytes:194334458 errors:0
            TX packets:968373 bytes:1269237344 errors:0
            TX queue packets:935965 errors:0
            TX port packets:968373 errors:0

vif0/2 Socket: unix
            Type:Agent HWaddr:00:00:5e:00:01:00 IPaddr:0
            Vrf:65535 Flags:L3 MTU:1514 Ref:2
            RX port packets:143 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:143 bytes:28514 errors:0
            TX packets:22423 bytes:2513813 errors:0
            TX queue packets:22423 errors:0
            TX port packets:22423 errors:0 syscalls:22427

vif0/5 PMD: tap4fc7dd3e-83
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
            Vrf:2 Flags:PL3L2D MTU:9160 Ref:13
            RX port packets:2571178439 errors:0 syscalls:1
            RX queue packets:2571178438 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:2571178439 bytes:503950977350 errors:0
            TX packets:2571163312 bytes:503948011652 errors:0
            TX port packets:2571129611 errors:33710 syscalls:37

vif0/6 PMD: tapd6c079b5-42
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
            Vrf:1 Flags:PL3L2D MTU:9160 Ref:13
            RX port packets:2571123399 errors:0 syscalls:1
            RX queue packets:2571123398 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:2571123399 bytes:503940189510 errors:0
            TX packets:2571182287 bytes:503951727336 errors:0
            TX port packets:2571181836 errors:459 syscalls:35

sleep60Vrouter Interface Table

Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
       Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
       D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
       Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
       Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload

vif0/0 PCI: 0:0:0.0 (Speed 10000, Duplex 1)
            Type:Physical HWaddr:90:e2:ba:4c:63:f0 IPaddr:0
            Vrf:0 Flags:L3L2Vp MTU:1514 Ref:21
            RX device packets:0 bytes:0 errors:10
            RX port packets:5143390881 errors:0
            RX queue packets:5143358024 errors:10
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 10
            RX packets:5143390871 bytes:1153171789557 errors:0
            TX packets:5142453272 bytes:1152056188295 errors:0
            TX port packets:5142453288 errors:7

vif0/1 Virtual: vhost0
            Type:Host HWaddr:90:e2:ba:4c:63:f0 IPaddr:0
            Vrf:0 Flags:L3L2 MTU:1514 Ref:10
            RX port packets:213673 errors:0
            RX queue packets:213611 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:213673 bytes:194531706 errors:0
            TX packets:969327 bytes:1269363701 errors:0
            TX queue packets:936480 errors:0
            TX port packets:969327 errors:0

vif0/2 Socket: unix
            Type:Agent HWaddr:00:00:5e:00:01:00 IPaddr:0
            Vrf:65535 Flags:L3 MTU:1514 Ref:2
            RX port packets:143 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:143 bytes:28514 errors:0
            TX packets:22718 bytes:2543313 errors:0
            TX queue packets:22718 errors:0
            TX port packets:22718 errors:0 syscalls:22722

vif0/5 PMD: tap4fc7dd3e-83
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
            Vrf:2 Flags:PL3L2D MTU:9160 Ref:13
            RX port packets:2571178439 errors:0 syscalls:1
            RX queue packets:2571178438 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:2571178439 bytes:503950977350 errors:0
            TX packets:2571163312 bytes:503948011652 errors:0
            TX port packets:2571129611 errors:33710 syscalls:37

vif0/6 PMD: tapd6c079b5-42
            Type:Virtual HWaddr:00:00:5e:00:01:00 IPaddr:0
            Vrf:1 Flags:PL3L2D MTU:9160 Ref:13
            RX port packets:2571123399 errors:0 syscalls:1
            RX queue packets:2571123398 errors:0
            RX queue errors to lcore 0 0 0 0 0 0 0 0 0 0 0 0 0 0
            RX packets:2571123399 bytes:503940189510 errors:0
            TX packets:2571182287 bytes:503951727336 errors:0
            TX port packets:2571181836 errors:459 syscalls:35

root@cs-scale-7:~#
root@cs-scale-7:~#
root@cs-scale-7:~#
root@cs-scale-7:~#
root@cs-scale-7:~#

Build 2714 Kilo

See original description
Revision history for this message
Vinod Nair (vinodnair) wrote :
Changed in juniperopenstack:
milestone: none → r3.0-fcs
tags: added: blocker
summary: - DPDK: vhost0 with jumo mtu traffic across computes stall
+ DPDK: vhost0 with jumbo mtu traffic across computes stall
summary: - DPDK: vhost0 with jumbo mtu traffic across computes stall
+ DPDK: vhost0 with jumbo mtu traffic across / to computes stall
Vinod Nair (vinodnair)
description: updated
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/17833
Submitter: Andriy Berestovskyy (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Review in progress for https://review.opencontrail.org/17835
Submitter: Wojciech Zmuda (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/17836
Submitter: Wojciech Zmuda (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/17833
Submitter: Andriy Berestovskyy (<email address hidden>)

Revision history for this message
Raja Sivaramakrishnan (raja-u) wrote :

Release note: With DPDK enabled, the MTU on the vhost0 interface cannot be set higher than 8K

tags: added: releasenote
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/17835
Committed: http://github.org/Juniper/contrail-vrouter/commit/f6ec55cb2e4f454976dd7406ed1173d562766cc4
Submitter: Zuul
Branch: master

commit f6ec55cb2e4f454976dd7406ed1173d562766cc4
Author: Wojciech Zmuda <email address hidden>
Date: Thu Feb 25 17:06:49 2016 +0100

DPDK: increase max packet size from 9160B to 9kB.

Packet size is passed to the ixgbe driver as a buffer
size. Then it is calculated to get the buffer size in kilobytes
using right bitshift by 10. 9160 >> 10 is 8. This leads to truncating
incoming 9kB frames to 8kB. Changing packet size to integer multiple
of 1kB solves the problem.

Change-Id: I202c7f9430c5952b5e89b1ef9fff101ed36bfe82
Closes-bug: #1547782

Changed in juniperopenstack:
milestone: none → r3.1.0.0-fcs
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/17836
Submitter: Wojciech Zmuda (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/17836
Committed: http://github.org/Juniper/contrail-vrouter/commit/aa66037b411ca5c7f88c27a10aed6d5d623af0cf
Submitter: Zuul
Branch: R3.0

commit aa66037b411ca5c7f88c27a10aed6d5d623af0cf
Author: Wojciech Zmuda <email address hidden>
Date: Thu Feb 25 17:06:49 2016 +0100

DPDK: increase max packet size from 9160B to 9kB.

Packet size is passed to the ixgbe driver as a buffer
size. Then it is calculated to get the buffer size in kilobytes
using right bitshift by 10. 9160 >> 10 is 8. This leads to truncating
incoming 9kB frames to 8kB. Changing packet size to integer multiple
of 1kB solves the problem.

Change-Id: I202c7f9430c5952b5e89b1ef9fff101ed36bfe82
Closes-bug: #1547782

Revision history for this message
Vinod Nair (vinodnair) wrote :

Tried with Build 3.0 Build 2729 and still see issues

HOST:
Set mtu of 9160 on vhost interface and tried to copy a file , traffic rate is way low and stalls .. But sometimes it does work

scp root@8.0.0.1:/store/traffic* .
root@8.0.0.1's password:
traffic1.qcow2 0% 0 0.0KB/s --:-- ETA

traffic1.qcow2 0% 0 0.0KB/s - stalled -

traffic1.qcow2 0% 0 0.0KB/s - stalled -
traffic1.qcow2 0% 0 0.0KB/s - stalled

VROUTER: The mtu seems to be capped at 1468

Trace path from a VM
root@sw1-traffic1:~# ifconfig eth0 mtu 9160
root@sw1-traffic1:~# tracepath 140.1.0.3
 1?: [LOCALHOST] pmtu 9160
 1: 8.0.0.3 0.763ms
 1: 8.0.0.3 0.994ms
 2: 140.2.0.1 1.092ms pmtu 1468
 2: no reply

Revision history for this message
Vinod Nair (vinodnair) wrote :

My bad .. There was some stray traffic , see no issues with host to host traffic
Tried with mtu up to 9160 using 3.0 build 2729

Nagabhushana R (bhushana)
information type: Proprietary → Public
Ashish Ranjan (aranjan-n)
no longer affects: juniperopenstack/r3.1
Revision history for this message
Greg (gregsmith) wrote :

What is the symptom of this bug?
That is, what happens when the MTU is set too high or to a "wrong" value?

Thanks,

Greg S

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.