Low iperf throughput VM to VM

[Billey O'Mahoney]

Detailed bug description:
 Using two ubuntu14.04 VMs on one compute node I see very low TCP performance between the two VMs.
Steps to reproduce:
 VM2# iperf -s
 VM1# iperf <ip_addr VM1>
Expected results:
 Would expect results of 4-5Gbps
Actual result:
 Seeing results of < 1Gbps
Reproducibility:
 <put your information here>
Workaround:
 By replicating the basics of the setup on a separate
Impact:
 Very poor network performance for VMs
Description of the environment:
* Fuel@OPNFV Colorado SR3.0.
* DPDK-enabled private network NICs on compute node and OVS-DPDK plugin enabled (configuration DPDK=selected, NSH=not selected)
* VMs are ubuntu 14.04
* VLAN network segmentation
* Compute node 28 core 2 Numa Node Intel(R) Xeon(R) CPU E5-2695 v3 @ 2.30GHz (or similar).
Additional information:
 After recreating on a development environment - ie using OVS2.6.1 and QEMU2.5 it was discovered that if OVS was compiled with DPDK16.11 instead of 16.07 the iperf TCP performance was 4-5Gbps.

[Oleksiy Molchanov (omolchanov)]

Hi Billey, what version of Fuel do you use?

[Billey O'Mahoney (billey)]

Hi Oleksiy,

Fuel9 - I'm not sure if it's a point release but it's which ever version is use by Fuel@OPNFV Colorado SR3.0. I can find out.

The Fuel@OPNFV guys (smelikyan) are working on a fix - by including DPDK 16.11 - at the moment.

[Billey O'Mahoney (billey)]

From unit testing with qemu, ovs & dpdk I found that moving to dpdk 16.11 give much better performance provided that INDIRECT_DESC feature is not disabled in dpdk's vhostuser configuration. There is a patch in review to ensure this feature is enabled in future OVS versions. https://mail.openvswitch.org/pipermail/ovs-dev/2017-March/329228.html

Also I came across this bug https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1668829 which seems to indicate that it could be something in qemu that is the root of the issue in the way it affects the distribution of the transmitting VMs packets – making them more bursty which then overflows the relatively small virtio queues to the other VM.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 10.0/newton
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/32321

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/30635
Submitter: Pkgs Jenkins <email address hidden>
Branch: master

Commit: 847038c8ea40ee2f399e67f0a7d9fd8b31618998
Author: Ivan Suzdal <email address hidden>
Date: Fri Mar 17 10:18:28 2017

Dpdk 16.11

Sources from [0]

[0] http://packages.ubuntu.com/zesty/dpdk

Closes-Bug: #1663332

Change-Id: Ic1ec9896108708a9f138347e57a02a88dad39d85

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/32321
Submitter: Pkgs Jenkins <email address hidden>
Branch: 10.0/newton

Commit: e87421a93b48ba7e47a6223824e64c56f7bbb93d
Author: Ivan Suzdal <email address hidden>
Date: Fri Mar 17 10:18:57 2017

Dpdk 16.11

Sources from [0]

[0] http://packages.ubuntu.com/zesty/dpdk

Closes-Bug: #1663332

Change-Id: Ic1ec9896108708a9f138347e57a02a88dad39d85

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/30659
Submitter: Pkgs Jenkins <email address hidden>
Branch: master

Commit: 8f325718d1f27242f9ad7d58a9a9a36412b1a065
Author: Ivan Suzdal <email address hidden>
Date: Fri Mar 17 10:42:18 2017

Rebuilt against dpdk 16.11

Related-Bug: #1663332

Change-Id: I61dec05dcbf5dc93d3d95c17a353c1dc6b6f9935

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/32322
Submitter: Pkgs Jenkins <email address hidden>
Branch: 10.0/newton

Commit: 97edd55f892b1e56973314cf4dee0aba178ce05d
Author: Ivan Suzdal <email address hidden>
Date: Fri Mar 17 10:42:34 2017

Rebuilt against dpdk 16.11

Related-Bug: #1663332

Change-Id: I61dec05dcbf5dc93d3d95c17a353c1dc6b6f9935