Latest fixes from storvsc & scsi drivers for Hyper-V/Azure

[Andy Whitcroft]

A number of significant fixes have landed upstream for Hyper-V/Azure clouds. The upstream devs for this recommend we apply the below list of patches:

commit 26b9fd8b3452dcf0a8862e307ee23f442f63fb51
Author: K. Y. Srinivasan <email address hidden>
Date: Fri Jul 18 17:11:27 2014 +0200

    sd: fix a bug in deriving the FLUSH_TIMEOUT from the basic I/O timeout

commit 4cd83ecdac20d30725b4f96e5d7814a1e290bc7e
Author: K. Y. Srinivasan <email address hidden>
Date: Sat Jul 12 09:48:26 2014 -0700

    Drivers: scsi: storvsc: Change the limits to reflect the values on the host

commit 52f9614dd8294e95d2c0929c2d4f64b077ae486f
Author: K. Y. Srinivasan <email address hidden>
Date: Sat Jul 12 09:48:27 2014 -0700

    Drivers: scsi: storvsc: Set cmd_per_lun to reflect value supported by the Host

commit 8caf92d80526f3d7cc96831ec18b384ebcaccdf0
Author: K. Y. Srinivasan <email address hidden>
Date: Sat Jul 12 09:48:28 2014 -0700

    Drivers: scsi: storvsc: Filter commands based on the storage protocol version

commit adb6f9e1a8c6af1037232b59edb11277471537ea
Author: K. Y. Srinivasan <email address hidden>
Date: Sat Jul 12 09:48:29 2014 -0700

    Drivers: scsi: storvsc: Fix a bug in handling VMBUS protocol version

commit 56b26e69c8283121febedd12b3cc193384af46b9
Author: K. Y. Srinivasan <email address hidden>
Date: Sat Jul 12 09:48:30 2014 -0700

    Drivers: scsi: storvsc: Implement a eh_timed_out handler

commit f885fb73f64154690c2158e813de56363389ffec
Author: K. Y. Srinivasan <email address hidden>
Date: Sat Jul 12 09:48:31 2014 -0700

    drivers: scsi: storvsc: Set srb_flags in all cases

commit 3533f8603d28b77c62d75ec899449a99bc6b77a1
Author: K. Y. Srinivasan <email address hidden>
Date: Sat Jul 12 09:48:32 2014 -0700

    drivers: scsi: storvsc: Correctly handle TEST_UNIT_READY failure

commit f3cfabce7a2e92564d380de3aad4b43901fb7ae6
Author: K. Y. Srinivasan <email address hidden>
Date: Mon Jul 21 16:06:01 2014 -0700

    Drivers: add blist flags

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1354397

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Andy Whitcroft (apw)]

List of just the sha1s:

f3cfabce7a2e92564d380de3aad4b43901fb7ae6
3533f8603d28b77c62d75ec899449a99bc6b77a1
f885fb73f64154690c2158e813de56363389ffec
56b26e69c8283121febedd12b3cc193384af46b9
adb6f9e1a8c6af1037232b59edb11277471537ea
8caf92d80526f3d7cc96831ec18b384ebcaccdf0
52f9614dd8294e95d2c0929c2d4f64b077ae486f
4cd83ecdac20d30725b4f96e5d7814a1e290bc7e
26b9fd8b3452dcf0a8862e307ee23f442f63fb51

These are all merged for v3.17-rc1.

[Andy Whitcroft (apw)]

There are two additional foundational patches needed, one for each of these sets (the last patch is logically separate above):

  commit c1d40a527e885a40bb9ea6c46a1b1145d42b66a0
  Author: Martin K. Petersen <email address hidden>
  Date: Tue Jul 15 12:49:17 2014 -0400

    scsi: add a blacklist flag which enables VPD page inquiries

and

  commit a118c6c1d907e52286df25ee1e8b217f25d6f73d
  Author: Christoph Hellwig <email address hidden>
  Date: Sat Jun 28 12:08:05 2014 +0200

    sd: don't use scsi_setup_blk_pc_cmnd for flush requests

[Andy Whitcroft (apw)]

I have applied the above fixes for testing to both Trusty and Utopic kernels, could you test those and report any such results back here. Kernels can be found at the following links:

    http://people.canonical.com/~apw/lp1354397-trusty/
    http://people.canonical.com/~apw/lp1354397-utopic/

Thanks.

[srikanth myakam (v-srm)]

Hi Andy,
I see kernel panic on Ubuntu1404 using packages given in http://people.canonical.com/~apw/lp1354397-trusty/ and works fine with the packages available in Ubuntu repository. Hardware is windows Hyper-V. Serial logs are attached.

We didn't see any such issues on Ubuntu 1410 but unable to install "linux-cloud-tools-3.16.0-8-generic_3.16.0-8.13~lp1354397v201408121801_amd64.deb" as its dependency "linux-cloud-tools-3.16.0-8.deb" is missing and we cant find it over Google. We used earlier version binaries here.

[Andy Whitcroft (apw)]

Ok, I think I have found the issue with the trusty bits and have respun those. Please test the kernels at the URL below and see if that works better:

    http://people.canonical.com/~apw/lp1354397-trusty/

Please report any testing back here.

[srikanth myakam (v-srm)]

Thanks Andy,
The new kernel images for Ubuntu 1404 are booting without any issues.
We will report back with any issues once complete testing is done.

[srikanth myakam (v-srm)]

Please find the attached serial logs of Ubuntu1410 while running diskstress tests.

[srikanth myakam (v-srm)]

Please find the attached serial logs of Ubuntu1410 while running diskstress tests.

[Stephen A. Zarkos (stevez)]

Thanks Srikanth. These logs look typical of some of our stress tests. Can you confirm that the other tests passed, including the new fix upstream to enable unmap/trim?

Thanks,
Steve

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.16.0-18.25

---------------
linux (3.16.0-18.25) utopic; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1373682
 -- Tim Gardner <email address hidden> Wed, 24 Sep 2014 19:23:23 -0600

[Stephen A. Zarkos (stevez)]

Andy, thanks for taking these. All these changes passed our tests except for TRIM.

It looks like we'll need to take a closer look at how to fix TRIM upstream. We probably will not get that upstream before kernel freeze. Do you think it be possible to re-base your original patch to re-enable SPC-3 in Hyper-V so that we don't regress this feature (http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-trusty.git;a=commit;h=ff2c5fa3fa9adf0b919b9425e71a8ba044c31a7d)?

[Andy Whitcroft (apw)]

That is likely to ordering issue we identified in the original patches (which upstream rejected). For the moment I have reapplied the change to use SPC-3.

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Stephen A. Zarkos (stevez)]

Hi,

The Azure test team has completed testing the proposed kernel package for Trusty and verified that all fixes work as expected. Please mark this as verification complete.

Thanks!
Steve

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.13.0-39.66

---------------
linux (3.13.0-39.66) trusty; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1386629

  [ Upstream Kernel Changes ]

  * KVM: x86: Check non-canonical addresses upon WRMSR
    - LP: #1384539
    - CVE-2014-3610
  * KVM: x86: Prevent host from panicking on shared MSR writes.
    - LP: #1384539
    - CVE-2014-3610
  * KVM: x86: Improve thread safety in pit
    - LP: #1384540
    - CVE-2014-3611
  * KVM: x86: Fix wrong masking on relative jump/call
    - LP: #1384545
    - CVE-2014-3647
  * KVM: x86: Warn if guest virtual address space is not 48-bits
    - LP: #1384545
    - CVE-2014-3647
  * KVM: x86: Emulator fixes for eip canonical checks on near branches
    - LP: #1384545
    - CVE-2014-3647
  * KVM: x86: emulating descriptor load misses long-mode case
    - LP: #1384545
    - CVE-2014-3647
  * KVM: x86: Handle errors when RIP is set during far jumps
    - LP: #1384545
    - CVE-2014-3647
  * kvm: vmx: handle invvpid vm exit gracefully
    - LP: #1384544
    - CVE-2014-3646
  * Input: synaptics - gate forcepad support by DMI check
    - LP: #1381815

linux (3.13.0-38.65) trusty; urgency=low

  [ Luis Henriques ]

  * Release Tracking Bug
    - LP: #1379244

  [ Andy Whitcroft ]

  * Revert "SAUCE: scsi: hyper-v storsvc switch up to SPC-3"
    - LP: #1354397
  * [Config] linux-image-extra is additive to linux-image
    - LP: #1375310
  * [Config] linux-image-extra postrm is not needed on purge
    - LP: #1375310

  [ Upstream Kernel Changes ]

  * Revert "KVM: x86: Increase the number of fixed MTRR regs to 10"
    - LP: #1377564
  * Revert "USB: option,zte_ev: move most ZTE CDMA devices to zte_ev"
    - LP: #1377564
  * aufs: bugfix, stop calling security_mmap_file() again
    - LP: #1371316
  * ipvs: fix ipv6 hook registration for local replies
    - LP: #1349768
  * Drivers: add blist flags
    - LP: #1354397
  * sd: fix a bug in deriving the FLUSH_TIMEOUT from the basic I/O timeout
    - LP: #1354397
  * drm/i915/bdw: Add 42ms delay for IPS disable
    - LP: #1374389
  * drm/i915: add null render states for gen6, gen7 and gen8
    - LP: #1374389
  * drm/i915/bdw: 3D_CHICKEN3 has write mask bits
    - LP: #1374389
  * drm/i915/bdw: Disable idle DOP clock gating
    - LP: #1374389
  * drm/i915: call lpt_init_clock_gating on BDW too
    - LP: #1374389
  * drm/i915: shuffle panel code
    - LP: #1374389
  * drm/i915: extract backlight minimum brightness from VBT
    - LP: #1374389
  * drm/i915: respect the VBT minimum backlight brightness
    - LP: #1374389
  * drm/i915/bdw: Apply workarounds in render ring init function
    - LP: #1374389
  * drm/i915/bdw: Cleanup pre prod workarounds
    - LP: #1374389
  * drm/i915: Replace hardcoded cacheline size with macro
    - LP: #1374389
  * drm/i915: Refactor Broadwell PIPE_CONTROL emission into a helper.
    - LP: #1374389
  * drm/i915: Add the WaCsStallBeforeStateCacheInvalidate:bdw workaround.
    - LP: #1374389
  * drm/i915/bdw: Remove BDW preproduction W/As until C stepping.
    - LP: #1374389
  * mptfusion: enable no_write_same for vmware scsi disks
    - LP: #1371591
  * iommu/amd: Fix cleanup_domai...