Tunnel offload indications not stripped from encapsulated packets, causing performance overhead

Bug #1602755 reported by James Page on 2016-07-13
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)

Bug Description

Commit: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit?id=a09a4c8dd1ec7f830e1fb9e59eb72bddc965d168


tunnels: Remove encapsulation offloads on decap.

If a packet is either locally encapsulated or processed through GRO
it is marked with the offloads that it requires. However, when it is
decapsulated these tunnel offload indications are not removed. This
means that if we receive an encapsulated TCP packet, aggregate it with
GRO, decapsulate, and retransmit the resulting frame on a NIC that does
not support encapsulation, we won't be able to take advantage of hardware
offloads even though it is just a simple TCP packet at this point.

This fixes the problem by stripping off encapsulation offload indications
when packets are decapsulated.

The performance impacts of this bug are significant. In a test where a
Geneve encapsulated TCP stream is sent to a hypervisor, GRO'ed, decapsulated,
and bridged to a VM performance is improved by 60% (5Gbps->8Gbps) as a
result of avoiding unnecessary segmentation at the VM tap interface.

Reported-by: Ramu Ramamurthy <email address hidden>
Fixes: 68c33163 ("v4 GRE: Add TCP segmentation offload for GRE")
Signed-off-by: Jesse Gross <email address hidden>
Signed-off-by: David S. Miller <email address hidden>

Looks like we have picked this up in Yakkety for 16.10 already. Will mark that task Fix Committed.

git describe --contains a09a4c8dd1ec7f830e1fb9e59eb72bddc965d168

It's been requested we consider this for SRU for Xenial.

Changed in linux (Ubuntu):
status: New → Fix Committed
Stefan Bader (smb) on 2016-07-14
Changed in linux (Ubuntu Xenial):
assignee: nobody → Stefan Bader (smb)
importance: Undecided → Medium
status: New → In Progress
Stefan Bader (smb) wrote :

Would it be possible to verify whether the proposed backport satisfies performance expectations. In particular exercising ipv6/sit since I had to adapt the patch there to make up for iptunnel_pull_header only having 3 arguments in 4.4.

Test kernel can be found at: http://people.canonical.com/~smb/lp1602755/

tags: added: patch
Jesse Gross (jessegross) wrote :

I tested this (both IPv4 over Geneve and IPv6 over sit) as well as looked at the proposed patch and everything looks good to me.

With a very simple setup (this does not go up to VMs so the full performance difference is not there), I see correct packet aggregation and improved performance:


root@server89:~# netperf -H
MIGRATED TCP STREAM TEST from ( port 0 AF_INET to () port 0 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

 87380 16384 16384 10.01 5538.36


root@server89:~# netperf -H
MIGRATED TCP STREAM TEST from ( port 0 AF_INET to () port 0 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec

 87380 16384 16384 10.00 7291.98

Changed in linux (Ubuntu Xenial):
status: In Progress → Fix Committed
Stefan Bader (smb) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-xenial
Jesse Gross (jessegross) wrote :

I tested the proposed kernel and it solves the problem. I'm not sure if it is appropriate for me to change the tag though or if that should wait for a larger QA effort.

Stefan Bader (smb) wrote :

It would have been appropriate (who would be better qualified than the reporter ;)). But I can do that for you now. Thanks for testing and reporting back.

tags: added: verification-done-xenial
removed: verification-needed-xenial
Changed in linux (Ubuntu Xenial):
assignee: Stefan Bader (smb) → nobody
Launchpad Janitor (janitor) wrote :
Download full text (13.4 KiB)

This bug was fixed in the package linux - 4.4.0-36.55

linux (4.4.0-36.55) xenial; urgency=low

  [ Stefan Bader ]

  * Release Tracking Bug
    - LP: #1612305

  * I2C touchpad does not work on AMD platform (LP: #1612006)
    - SAUCE: pinctrl/amd: Remove the default de-bounce time

  * CVE-2016-5696
    - tcp: make challenge acks less predictable

linux (4.4.0-35.54) xenial; urgency=low

  [ Stefan Bader ]

  * Release Tracking Bug
    - LP: #1611215

  * [i915_bpo] Sync with v4.7 (LP: #1609742)
    - SAUCE: i915_bpo: Sync with v4.7

  * s390/cio: fix reset of channel measurement block (LP: #1609415)
    - s390/cio: allow to reset channel measurement block

  * in Ubuntu16.10: Hit on Call traces and system goes down when transactional
    memory tests are running in 32TB Brazos system (LP: #1606786)
    - powerpc/tm: Avoid SLB faults in treclaim/trecheckpoint when RI=0
    - powerpc/tm: Fix stack pointer corruption in __tm_recheckpoint()

  * Power Menu does not display after press the Power Button (LP: #1609204)
    - intel-vbtn: new driver for Intel Virtual Button
    - [config] enable CONFIG_INTEL_VBTN=m

  * OptiPlex 7450 AIO hangs when rebooting (LP: #1608762)
    - x86/reboot: Add Dell Optiplex 7450 AIO reboot quirk

  * virtualbox+usb 3.0 breaks boot, -28 kernel works (LP: #1604058)
    - SAUCE: xhci: Fix soft lockup in xhci_pci_probe path when XHCI_STATE_HALTED

  * linux-kernel: Freeing IRQ from IRQ context (LP: #1597908)
    - block: defer timeouts to a workqueue

  * Tunnel offload indications not stripped from encapsulated packets, causing
    performance overhead (LP: #1602755)
    - tunnels: Remove encapsulation offloads on decap.

  * lm-sensors is throwing "ERROR: Can't get value of subfeature temp1_input:
    I/O error" for be2net driver (LP: #1607387)
    - be2net: perform temperature query in adapter regardless of its interface

  * Dell dock MAC Address pass through doesn't work in Ubuntu (LP: #1579984)
    - r8152: Add support for setting pass through MAC address on RTL8153-AD

  * vmxnet3 LRO IPv6 performance issues (stalling TCP) (LP: #1605494)
    - Driver: Vmxnet3: set CHECKSUM_UNNECESSARY for IPv6 packets

  * ISST-LTE:pVM:monklp5:Ubuntu16.04.1:system crashed at
    lpfc_sli4_scmd_to_wqidx_distr (LP: #1597974)
    - SAUCE: lpfc: fix oops in lpfc_sli4_scmd_to_wqidx_distr() from

  * Backport cxlflash shutdown patch to Xenial SRU (LP: #1605405)
    - SAUCE: cxlflash: Verify problem state area is mapped before notifying

  * Xenial update to v4.4.16 stable release (LP: #1607404)
    - mac80211: fix fast_tx header alignment
    - mac80211: mesh: flush mesh paths unconditionally
    - mac80211_hwsim: Add missing check for HWSIM_ATTR_SIGNAL
    - mac80211: Fix mesh estab_plinks counting in STA removal case
    - EDAC, sb_edac: Fix rank lookup on Broadwell
    - IB/cm: Fix a recently introduced locking bug
    - IB/mlx4: Properly initialize GRH TClass and FlowLabel in AHs
    - powerpc/pseries: Fix IBM_ARCH_VEC_NRCORES_OFFSET since POWER8NVL was added
    - powerpc/tm: Always reclaim in start_thread() for exec() class syscalls
    - usb: dwc2: fix reg...

Changed in linux (Ubuntu Xenial):
status: Fix Committed → Fix Released
Gideon Naim (gideonn) wrote :

Please backport this fix also to trusty 14.04 LTS

tags: added: trusty
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers