Trusty kernel network performance regression

[Jay Vosburgh]

SRU Justification:

Impact:

Reduced TCP/IP receive performance for network devices that do not split packet headers into skb linear area (e.g., mlx4). The trusty kernel has incorporated

commit eff44f9cc9a02aad53d568d3ae5020b6792ae4f6
Author: Jerry Chu <email address hidden>
Date: Wed Dec 11 20:53:45 2013 -0800

    net-gro: Prepare GRO stack for the upcoming tunneling support

which modifies the GRO frag0 optimization, but unfortunately for some cases results in calls to __skb_pull_tail for every packet being received via the GRO path. This causes a reduction in TCP receive performance (or, more accurately, an increase in CPU load for TCP receive processing, which will cause throughput reduction for CPU limited workloads).

Fix:

This has already been fixed in mainline in

commit a50e233c50dbc881abaa0e4070789064e8d12d70
Author: Eric Dumazet <email address hidden>
Date: Sat Mar 29 21:28:21 2014 -0700

    net-gro: restore frag0 optimization

The fix has been backported to and verified on the trusty kernel using mlx4 devices and iperf; an increase from 7.5 to 8.5 Gb/sec was observed when adding the patch, and the relevant portion of perf captures show changes in the call paths from:

     7.17% iperf [kernel.kallsyms] [k] __pskb_pull_tail
                      |
                      --- __pskb_pull_tail
                         |
                         |--48.03%-- tcp_gro_receive
                         | tcp4_gro_receive
                         | inet_gro_receive
                         | dev_gro_receive
                         | napi_gro_frags
                         | mlx4_en_process_rx_cq
                         | mlx4_en_poll_rx_cq
                         | net_rx_action
                         | __do_softirq
[...]
                         |--28.53%-- napi_gro_frags
                         | mlx4_en_process_rx_cq
                         | mlx4_en_poll_rx_cq
                         | net_rx_action
                         | __do_softirq
[...]
                         |--13.11%-- inet_gro_receive
                         | dev_gro_receive
                         | napi_gro_frags
                         | mlx4_en_process_rx_cq
                         | mlx4_en_poll_rx_cq
                         | net_rx_action
                         | __do_softirq

to:

     4.87% iperf [kernel.kallsyms] [k] skb_gro_receive
                    |
                    --- skb_gro_receive
                       |
                       |--98.13%-- tcp_gro_receive
                       | tcp4_gro_receive
                       | inet_gro_receive
                       | dev_gro_receive
                       | napi_gro_frags
                       | mlx4_en_process_rx_cq
                       | mlx4_en_poll_rx_cq
                       | net_rx_action
                       | __do_softirq

Testcase:

The fix was tested using mlx4 10Gb/sec network devices between two arm64 systems using "iperf -s" on one end and "iperf -c" on the other. The unmodified kernel reported approximately 7.5 Gb/sec throughput, the fixed kernel approximately 8.5 Gb/sec.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1344323

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[dann frazier (dannf)]

I'm seeing the expected performance with the SRU kernel (technically a different build with a couple of SATA patches required to boot)

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.13.0-35.62

---------------
linux (3.13.0-35.62) trusty; urgency=low

  [ Joseph Salisbury ]

  * Release Tracking Bug
    - LP: #1357148

  [ Brad Figg ]

  * Start new release

  [ dann frazier ]

  * SAUCE: (no-up) Fix build failure on arm64
    - LP: #1353657
  * [debian] Allow for package revisions condusive for branching

  [ David Henningsson ]

  * SAUCE: Call broadwell specific functions from the hda driver
    - LP: #1317865

  [ Edward Lin ]

  * SAUCE: (no-up) Add use native backlight quirk for Dell Inspiron
    5547/5447
    - LP: #1332437

  [ Imre Deak ]

  * SAUCE: drm/i915: move power domain init earlier during system resume
    - LP: #1353405

  [ Jani Nikula ]

  * SAUCE: drm/i915: use lane count and link rate from VBT as minimums for
    eDP
    - LP: #1338582
  * SAUCE: drm/i915/dp: force eDP lane count to max available lanes on BDW
    - LP: #1338582
  * SAUCE: drm/i915: provide interface for audio driver to query cdclk
    - LP: #1188091
  * SAUCE: drm/i915: demote opregion excessive timeout WARN_ONCE to
    DRM_INFO_ONCE
    - LP: #1351014

  [ Joseph Salisbury ]

  * [Config] updateconfigs after Linux 3.13.11.6 updates

  [ Luis Henriques ]

  * Revert "[Packaging] linux-udeb-flavour -- standardise on linux prefix"

  [ Ming Lei ]

  * Revert "SAUCE: (no-up) ata: Fix the dma state machine lockup for the
    IDENTIFY DEVICE PIO mode command."
    - LP: #1335645

  [ Paulo Zanoni ]

  * SAUCE: drm/i915: consider the source max DP lane count too
    - LP: #1338582

  [ Tim Gardner ]

  * [Config] CONFIG_GPIO_SYSFS=y
    - LP: #1342153
  * [Config] CONFIG_KEYS_DEBUG_PROC_KEYS=y
    - LP: #1344405
  * [Config] updateconfigs
  * [Config] CONFIG_SCSI_IPR_TRACE=y, CONFIG_SCSI_IPR_DUMP=y
    - LP: #1343109
  * [Config] CONFIG_CONTEXT_TRACKING_FORCE=n
    - LP: #1349028

  [ Timo Aaltonen ]

  * SAUCE: Fix a typo in hda i915_bdw support.
    - LP: #1343140

  [ Upstream Kernel Changes ]

  * Revert "net/mlx4_en: Fix bad use of dev_id"
    - LP: #1347012
  * Revert "ACPI / AC: Remove AC's proc directory."
    - LP: #1356913
  * Revert "mac80211: move "bufferable MMPDU" check to fix AP mode scan"
    - LP: #1356913
  * mm, pcp: allow restoring percpu_pagelist_fraction default
    - LP: #1347088
  * net: Fix permission check in netlink_connect()
    - LP: #1312989
  * netlink: Rename netlink_capable netlink_allowed
    - LP: #1312989
  * net: Move the permission check in sock_diag_put_filterinfo to
    packet_diag_dump
    - LP: #1312989
  * net: Add variants of capable for use on on sockets
    - LP: #1312989
  * net: Add variants of capable for use on netlink messages
    - LP: #1312989
  * net: Use netlink_ns_capable to verify the permisions of netlink
    messages
    - LP: #1312989
  * netlink: Only check file credentials for implicit destinations
    - LP: #1312989
  * igb: fix stats for i210 rx_fifo_errors
    - LP: #1338893
  * HID: use multi input quirk for 22b9:2968
    - LP: #1339567
  * crypto/nx: disable NX on little endian builds
    - LP: #1338666
  * ACPI / video: Add Dell Inspiron 5737 to the blacklist
    - LP: #1250401
  * Input: elantech - deal with clickpads reportin...