[Hyper-V] netvsc: fix incorrect receive checksum offloading

Bug #1636656 reported by Joshua R. Poulson
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Medium
Joseph Salisbury
Xenial
Fix Released
Medium
Joseph Salisbury
Yakkety
Fix Released
Medium
Joseph Salisbury

Bug Description

The Hyper-V netvsc driver was looking at the incorrect status bits
in the checksum info. It was setting the receive checksum unnecessary
flag based on the IP header checksum being correct. The checksum
flag is skb is about TCP and UDP checksum status. Because of this
bug, any packet received with bad TCP checksum would be passed
up the stack and to the application causing data corruption.
The problem is reproducible via netcat and netem.

This had a side effect of not doing receive checksum offload
on IPv6. The driver was also also always doing checksum offload
independent of the checksum setting done via ethtool.

Signed-off-by: Stephen Hemminger <email address hidden>

https://patchwork.ozlabs.org/patch/685660/

When this patch is committed I will include the commit ID in this bug.

Revision history for this message
Joshua R. Poulson (jrp) wrote :
Revision history for this message
Joshua R. Poulson (jrp) wrote :

This affects 16.10, 16.04, 14.04, and 12.04. Please update the ongoing HWE kernels.

tags: added: patch
Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1636656

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Chris Valean (cvalean)
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Changed in linux (Ubuntu):
importance: Undecided → Medium
status: Confirmed → Triaged
Changed in linux (Ubuntu Yakkety):
status: New → Triaged
no longer affects: linux (Ubuntu Zesty)
Changed in linux (Ubuntu Xenial):
status: New → Triaged
Changed in linux (Ubuntu Vivid):
status: New → Triaged
Changed in linux (Ubuntu Trusty):
status: New → Triaged
Changed in linux (Ubuntu Precise):
status: New → Triaged
importance: Undecided → Medium
Changed in linux (Ubuntu Trusty):
importance: Undecided → Medium
Changed in linux (Ubuntu Vivid):
importance: Undecided → Medium
Changed in linux (Ubuntu Xenial):
importance: Undecided → Medium
Changed in linux (Ubuntu Yakkety):
importance: Undecided → Medium
tags: added: kernel-da-key kernel-hyper-v
tags: added: bot-stop-nagging precise trusty vivid xenial yakkety
Revision history for this message
Joshua R. Poulson (jrp) wrote :

Upstream commit ID e52fed7177f742c27de2cc5314790aebb6

Changed in linux (Ubuntu):
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu Precise):
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu Trusty):
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu Vivid):
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu Xenial):
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu Yakkety):
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu):
status: Triaged → In Progress
Changed in linux (Ubuntu Precise):
status: Triaged → In Progress
Changed in linux (Ubuntu Xenial):
status: Triaged → In Progress
Changed in linux (Ubuntu Trusty):
status: Triaged → In Progress
Changed in linux (Ubuntu Yakkety):
status: Triaged → In Progress
Changed in linux (Ubuntu Vivid):
status: Triaged → In Progress
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

I built Yakkety, Xenial and Vivid test kernels with the request commit. The can be downloaded from:

Yakkety: http://kernel.ubuntu.com/~jsalisbury/lp1636656/yakkety
Xenial: http://kernel.ubuntu.com/~jsalisbury/lp1636656/xenial
Vivid: http://kernel.ubuntu.com/~jsalisbury/lp1636656/vivid

Can you test this kernels and see if they resolve this bug?

Trusty and Precise require some backporting, but I'll post a link to them when they are built.

Revision history for this message
Chris Valean (cvalean) wrote :

Hello,
We've completed the testing of the kernels provided.
The issue is resolved in the test kernels and we've also ran a sanity check for the netvsc driver, with no issues.
Thank you!

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Thanks for the update, Chris. I'll submit SRU request for Yakkety, Xenial and Vivid. I'm still getting some build failures with Trusty and Precise, but should have test kernels for them shortly.

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

A Trusty test kernel is now available. It can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1636656/trusty

I'm still working on identifying the prereqs for Precise.

Brad Figg (brad-figg)
no longer affects: linux (Ubuntu Precise)
Revision history for this message
Chris Valean (cvalean) wrote :

Hi Joe,

We verified the test kernel for Trusty from comment #8, however it does not seem to be resolving the issue, as we ran into a different issue.

With the test kernel 3.13.0-105.152~lp1636656_amd64 the VM is loosing network connectivity if we corrupt the packages and transfer a file.
This behavior is not seen in all the other kernels already tested, so it seems to be specific to be backport to 3.13 - dependency patches maybe?

Using the latest kernel for Trusty - 3.13.0-105-generic - we do *not* observe the netvsc messages if we corrupt the packages.
I will have to check this internally if the given bug here might have been introduced in a later kernel, will reply once we clarify the situation for Trusty.

Last question would be that in order to include the netvsc patch in 3.13, did you have to backport other patches? If so, can you please provide us with the list of them, that might help to understand the behavior.

Thank you!

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

For Trusty, I also applied one prereq patch due to build failures. The prereq patch I added was:

commit e3d605ed441cf4d113f9a1cf9e1b3f7cabe0d781
Author: KY Srinivasan <email address hidden>
Date: Sat Mar 8 19:23:16 2014 -0800

    Drivers: net: hyperv: Enable receive side IP checksum offload

This prereq patch was not a clean cherry pick, so I had to backport it. I'll review my backport, and see if it was done properly.

Luis Henriques (henrix)
Changed in linux (Ubuntu Yakkety):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Xenial):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Vivid):
status: In Progress → Fix Committed
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

I built another Trust test kernel. I backported commit e3d605ed44 again. Can you test this kernel an see if you still do *not* observe the netvsc messages if you corrupt the packages?

The V2 test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1636656/trusty

Revision history for this message
Luis Henriques (henrix) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-vivid' to 'verification-done-vivid'. If the problem still exists, change the tag 'verification-needed-vivid' to 'verification-failed-vivid'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-vivid
Revision history for this message
Luis Henriques (henrix) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-xenial
Chris Valean (cvalean)
tags: added: verification-done-vivid verification-done-xenial
removed: verification-needed-vivid verification-needed-xenial
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 3.19.0-79.87

---------------
linux (3.19.0-79.87) vivid; urgency=low

  [ Thadeu Lima de Souza Cascardo ]

  * Release Tracking Bug
    - LP: #1651534

  [ Luis Henriques ]

  * CVE-2016-9756
    - KVM: x86: drop error recovery in em_jmp_far and em_ret_far

  * CVE-2016-9806
    - netlink: Fix dump skb leak/double free

  * CVE-2016-9794
    - ALSA: pcm : Call kill_fasync() in stream lock

  * [Hyper-V] netvsc: fix incorrect receive checksum offloading (LP: #1636656)
    - netvsc: fix incorrect receive checksum offloading

  * CVE-2016-9793
    - net: avoid signed overflows for SO_{SND|RCV}BUFFORCE

  * Ubuntu16.10 - EEH on BELL3 adapter fails to recover (serial/tty)
    (LP: #1646857)
    - serial: 8250_pci: Detach low-level driver during PCI error recovery

 -- Thadeu Lima de Souza Cascardo <email address hidden> Tue, 20 Dec 2016 15:35:23 -0200

Changed in linux (Ubuntu Vivid):
status: Fix Committed → Fix Released
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (5.9 KiB)

This bug was fixed in the package linux - 4.4.0-59.80

---------------
linux (4.4.0-59.80) xenial; urgency=low

  [ John Donnelly ]

  * Release Tracking Bug
    - LP: #1654282

  * [2.1.1] MAAS has nvme0n1 set as boot disk, curtin fails (LP: #1651602)
    - (fix) nvme: only require 1 interrupt vector, not 2+

linux (4.4.0-58.79) xenial; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1651402

  * Support ACPI probe for IIO sensor drivers from ST Micro (LP: #1650123)
    - SAUCE: iio: st_sensors: match sensors using ACPI handle
    - SAUCE: iio: st_accel: Support sensor i2c probe using acpi
    - SAUCE: iio: st_pressure: Support i2c probe using acpi
    - [Config] CONFIG_HTS221=m, CONFIG_HTS221_I2C=m, CONFIG_HTS221_SPI=m

  * Fix channel data parsing in ST Micro sensor IIO drivers (LP: #1650189)
    - SAUCE: iio: common: st_sensors: fix channel data parsing

  * ST Micro lng2dm 3-axis "femto" accelerometer support (LP: #1650112)
    - SAUCE: iio: st-accel: add support for lis2dh12
    - SAUCE: iio: st_sensors: support active-low interrupts
    - SAUCE: iio: accel: Add support for the h3lis331dl accelerometer
    - SAUCE: iio: st_sensors: verify interrupt event to status
    - SAUCE: iio: st_sensors: support open drain mode
    - SAUCE: iio:st_sensors: fix power regulator usage
    - SAUCE: iio: st_sensors: switch to a threaded interrupt
    - SAUCE: iio: accel: st_accel: Add lis3l02dq support
    - SAUCE: iio: st_sensors: fix scale configuration for h3lis331dl
    - SAUCE: iio: accel: st_accel: add support to lng2dm
    - SAUCE: iio: accel: st_accel: inline per-sensor data
    - SAUCE: Documentation: dt: iio: accel: add lng2dm sensor device binding

  * ST Micro hts221 relative humidity sensor support (LP: #1650116)
    - SAUCE: iio: humidity: add support to hts221 rh/temp combo device
    - SAUCE: Documentation: dt: iio: humidity: add hts221 sensor device binding
    - SAUCE: iio: humidity: remove
    - SAUCE: iio: humidity: Support acpi probe for hts211

  * crypto : tolerate new crypto hardware for z Systems (LP: #1644557)
    - s390/zcrypt: Introduce CEX6 toleration

  * Acer, Inc ID 5986:055a is useless after 14.04.2 installed. (LP: #1433906)
    - uvcvideo: uvc_scan_fallback() for webcams with broken chain

  * vmxnet3 driver could causes kernel panic with v4.4 if LRO enabled.
    (LP: #1650635)
    - vmxnet3: segCnt can be 1 for LRO packets

  * system freeze when swapping to encrypted swap partition (LP: #1647400)
    - mm, oom: rework oom detection
    - mm: throttle on IO only when there are too many dirty and writeback pages

  * Kernel Fixes to get TCMU File Backed Optical to work (LP: #1646204)
    - target/user: Use sense_reason_t in tcmu_queue_cmd_ring
    - target/user: Return an error if cmd data size is too large
    - target/user: Fix comments to not refer to data ring
    - SAUCE: (no-up) target/user: Fix use-after-free of tcmu_cmds if they are
      expired

  * CVE-2016-9756
    - KVM: x86: drop error recovery in em_jmp_far and em_ret_far

  * Dell Precision 5520 & 3520 freezes at login screent (LP: #1650054)
    - ACPI / blacklist: add _REV quirks for Dell Precision 5520 and 3520

  * CVE-2016-979...

Read more...

Changed in linux (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Chris Valean (cvalean) wrote :

Hi Joe,

For the v2 test kernel for Trusty from comment #11:

The kernels 3.13 and 3.19 at least from our testing might not be affected by this bug as those don't have any receive checksum offloading, and therefore the patch here is not applicable.

Even with the cherry-pick, the behavior with only the patch for "fix incorrect receive checksum offloading" and the related dependencies, a file copy over scp is not completing.

So please do not backport this patch to Trusty / kernel 3.13 and/or 3.19.

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Thanks for the update, Chris.

no longer affects: linux (Ubuntu Trusty)
no longer affects: linux (Ubuntu Vivid)
Brad Figg (brad-figg)
Changed in linux (Ubuntu Yakkety):
status: Fix Committed → Fix Released
Chris Valean (cvalean)
Changed in linux (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.