[Hyper-V] Additional patches for Lv2 storage performance

Bug #1805304 reported by Joshua R. Poulson on 2018-11-26
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux-azure (Ubuntu)
Undecided
Marcelo Cerri
Bionic
Undecided
Marcelo Cerri
Cosmic
Undecided
Unassigned

Bug Description

After analysis of the first 4.15 kernel for Lv2 performance, and while we are delayed getting to 4.18, we have identified and backported the following patches for the 4.15 linux-azure kernel:

commit 1268ed0c474a5c8f165ef386f3310521b5e00e27
Author: K. Y. Srinivasan <email address hidden>
Date: Tue Jul 3 16:01:55 2018 -0700
    x86/hyper-v: Fix the circular dependency in IPI enlightenment
linux-next: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20181126&id=1268ed0c474a5c8f165ef386f3310521b5e00e27

commit 366f03b0cf90ef55f063d4a54cf62b0ac9b6da9d
Author: K. Y. Srinivasan <email address hidden>
Date: Wed May 16 14:53:32 2018 -0700
    X86/Hyper-V: Enhanced IPI enlightenment
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20181126&id=366f03b0cf90ef55f063d4a54cf62b0ac9b6da9d

commit 68bb7bfb7985df2bd15c2dc975cb68b7a901488a
Author: K. Y. Srinivasan <email address hidden>
Date: Wed May 16 14:53:31 2018 -0700
    X86/Hyper-V: Enable IPI enlightenments
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20181126&id=68bb7bfb7985df2bd15c2dc975cb68b7a901488a

commit 6b48cb5f8347bc0153ff1d7b075db92e6723ffdb
Author: K. Y. Srinivasan <email address hidden>
Date: Wed May 16 14:53:30 2018 -0700
    X86/Hyper-V: Enlighten APIC access
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20181126&id=6b48cb5f8347bc0153ff1d7b075db92e6723ffdb

commit 68d1eb72ee99e26576913aa6824f7a703ca06b90
Author: Vitaly Kuznetsov <email address hidden>
Date: Tue Mar 20 15:02:09 2018 +0100
    x86/hyper-v: define struct hv_enlightened_vmcs and clean field bits
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20181126&id=68d1eb72ee99e26576913aa6824f7a703ca06b90

commit a46d15cc1ae5af905afac2af4cc0c188c2eb59b0
Author: Vitaly Kuznetsov <email address hidden>
Date: Tue Mar 20 15:02:08 2018 +0100
    x86/hyper-v: allocate and use Virtual Processor Assist Pages
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20181126&id=a46d15cc1ae5af905afac2af4cc0c188c2eb59b0

commit 415bd1cd3a42897f61a92cda0a9f9d7b04c28fb7
Author: Vitaly Kuznetsov <email address hidden>
Date: Tue Mar 20 15:02:06 2018 +0100
    x86/hyper-v: move definitions from TLFS to hyperv-tlfs.h
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20181126&id=415bd1cd3a42897f61a92cda0a9f9d7b04c28fb7

commit 5a485803221777013944cbd1a7cd5c62efba3ffa
Author: Vitaly Kuznetsov <email address hidden>
Date: Tue Mar 20 15:02:05 2018 +0100
    x86/hyper-v: move hyperv.h out of uapi
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20181126&id=5a485803221777013944cbd1a7cd5c62efba3ffa

commit e7c4e36c447daca2b7df49024f6bf230871cb155
Author: Vitaly Kuznetsov <email address hidden>
Date: Wed Jan 24 14:23:34 2018 +0100
    x86/hyperv: Redirect reenlightment notifications on CPU offlining
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20181126&id=e7c4e36c447daca2b7df49024f6bf230871cb155

commit 93286261de1b46339aa27cd4c639b21778f6cade
Author: Vitaly Kuznetsov <email address hidden>
Date: Wed Jan 24 14:23:33 2018 +0100
    x86/hyperv: Reenlightenment notifications support
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20181126&id=93286261de1b46339aa27cd4c639b21778f6cade

commit e2768eaa1ca4fbb7b778da5615cce3dd310352e6
Author: Vitaly Kuznetsov <email address hidden>
Date: Wed Jan 24 14:23:32 2018 +0100
    x86/hyperv: Add a function to read both TSC and TSC page value simulateneously
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20181126&id=e2768eaa1ca4fbb7b778da5615cce3dd310352e6

commit 4a5f3cde4d51c7afce859aed9d74d197751896d5
Author: Michael Kelley <email address hidden>
Date: Fri Dec 22 11:19:02 2017 -0700
    Drivers: hv: vmbus: Remove x86-isms from arch independent drivers
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/drivers/hv?h=next-20181126&id=4a5f3cde4d51c7afce859aed9d74d197751896d5

From: Dexuan Cui <email address hidden>

We can concurrently try to open the same sub-channel from 2 paths:

path #1: vmbus_onoffer() -> vmbus_process_offer() -> handle_sc_creation().
path #2: storvsc_probe() -> storvsc_connect_to_vsp() ->
  -> storvsc_channel_init() -> handle_multichannel_storage() ->
  -> vmbus_are_subchannels_present() -> handle_sc_creation().

They conflict with each other, but it was not an issue before the recent
commit ae6935ed7d42 ("vmbus: split ring buffer allocation from open"),
because at the beginning of vmbus_open() we checked newchannel->state so
only one path could succeed, and the other would return with -EINVAL.

After ae6935ed7d42, the failing path frees the channel's ringbuffer by
vmbus_free_ring(), and this causes a panic later.

Commit ae6935ed7d42 itself is good, and it just reveals the longstanding
race. We can resolve the issue by removing path #2, i.e. removing the
second vmbus_are_subchannels_present() in handle_multichannel_storage().

BTW, the comment "Check to see if sub-channels have already been created"
in handle_multichannel_storage() is incorrect: when we unload the driver,
we first close the sub-channel(s) and then close the primary channel, next
the host sends rescind-offer message(s) so primary->sc_list will become
empty. This means the first vmbus_are_subchannels_present() in
handle_multichannel_storage() is never useful.

Fixes: ae6935ed7d42 ("vmbus: split ring buffer allocation from open")

CVE References

Joshua R. Poulson (jrp) wrote :

Sorry, the name for the last patch on the list will be "scsi: storvsc: Fix a race in sub-channel creation that can cause panic"

Dexuan Cui (decui) wrote :

The link to "[PATCH] scsi: storvsc: Fix a race in sub-channel creation that can cause panic" is

https://lkml.org/lkml/2018/11/26/159
or
https://lore.kernel.org/patchwork/patch/1016903/

Joshua R. Poulson (jrp) wrote :

Awesome, thanks Dexuan.

Joshua R. Poulson (jrp) wrote :

Issue found: on L80v2 the kernel is allocating IRQs outside of NUMA0, but all of the NVME disks on L80v2 are on NUMA0. Looking for a relevant patch for this.

tags: added: patch
Marcelo Cerri (mhcerri) on 2018-11-27
Changed in linux-azure (Ubuntu):
status: New → Confirmed
assignee: nobody → Marcelo Cerri (mhcerri)
Changed in linux-azure (Ubuntu Bionic):
status: New → In Progress
assignee: nobody → Marcelo Cerri (mhcerri)
Marcelo Cerri (mhcerri) on 2018-11-27
Changed in linux-azure (Ubuntu Bionic):
status: In Progress → Fix Committed
Joshua R. Poulson (jrp) wrote :

@mhcerri looking at the changelog I don't see "[PATCH] scsi: storvsc: Fix a race in sub-channel creation that can cause panic" in 4.15.0-1034... https://launchpad.net/ubuntu/+source/linux-azure/4.15.0-1034.35

Launchpad Janitor (janitor) wrote :
Download full text (14.2 KiB)

This bug was fixed in the package linux-azure - 4.15.0-1035.36~14.04.2

---------------
linux-azure (4.15.0-1035.36~14.04.2) trusty; urgency=medium

  * linux-azure: 4.15.0-1035.36~14.04.2 -proposed tracker (LP: #1806063)

  [ Ubuntu: 4.15.0-1035.36 ]

  * linux-azure: 4.15.0-1035.36 -proposed tracker (LP: #1806021)
  * [Hyper-V] Additional patches for Lv2 storage performance (LP: #1805304)
    - SAUCE: scsi: storvsc: Fix a race in sub-channel creation that can cause
      panic

linux-azure (4.15.0-1034.35~14.04.2) trusty; urgency=medium

  * linux-azure: 4.15.0-1034.35~14.04.2 -proposed tracker (LP: #1805474)

  [ Ubuntu: 4.15.0-1034.35 ]

  * linux-azure: 4.15.0-1034.35 -proposed tracker (LP: #1805412)
  * [Hyper-V] Additional patches for Lv2 storage performance (LP: #1805304)
    - SAUCE: Drivers: hv: vmbus: Remove x86-isms from arch independent drivers
    - SAUCE: x86/hyperv: Add a function to read both TSC and TSC page value
      simulateneously
    - SAUCE: x86/hyperv: Reenlightenment notifications support
    - SAUCE: x86/hyperv: Redirect reenlightment notifications on CPU offlining
    - SAUCE: x86/hyper-v: move hyperv.h out of uapi
    - SAUCE: x86/hyper-v: move definitions from TLFS to hyperv-tlfs.h
    - SAUCE: x86/hyper-v: allocate and use Virtual Processor Assist Pages
    - SAUCE: x86/hyper-v: define struct hv_enlightened_vmcs and clean field bits
    - SAUCE: X86/Hyper-V: Enlighten APIC access
    - SAUCE: X86/Hyper-V: Enable IPI enlightenments
    - SAUCE: X86/Hyper-V: Enhanced IPI enlightenment
    - SAUCE: x86/hyper-v: Fix the circular dependency in IPI enlightenment

linux-azure (4.15.0-1033.34~14.04.2) trusty; urgency=medium

  * linux-azure: 4.15.0-1033.34~14.04.2 -proposed tracker (LP: #1802564)

  [ Ubuntu: 4.15.0-1033.34 ]

  * linux-azure: 4.15.0-1033.34 -proposed tracker (LP: #1802559)
  * linux: 4.15.0-42.45 -proposed tracker (LP: #1803592)
  * [FEAT] Guest-dedicated Crypto Adapters (LP: #1787405)
    - KVM: s390: reset crypto attributes for all vcpus
    - KVM: s390: vsie: simulate VCPU SIE entry/exit
    - KVM: s390: introduce and use KVM_REQ_VSIE_RESTART
    - KVM: s390: refactor crypto initialization
    - s390: vfio-ap: base implementation of VFIO AP device driver
    - s390: vfio-ap: register matrix device with VFIO mdev framework
    - s390: vfio-ap: sysfs interfaces to configure adapters
    - s390: vfio-ap: sysfs interfaces to configure domains
    - s390: vfio-ap: sysfs interfaces to configure control domains
    - s390: vfio-ap: sysfs interface to view matrix mdev matrix
    - KVM: s390: interface to clear CRYCB masks
    - s390: vfio-ap: implement mediated device open callback
    - s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl
    - s390: vfio-ap: zeroize the AP queues
    - s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl
    - KVM: s390: Clear Crypto Control Block when using vSIE
    - KVM: s390: vsie: Do the CRYCB validation first
    - KVM: s390: vsie: Make use of CRYCB FORMAT2 clear
    - KVM: s390: vsie: Allow CRYCB FORMAT-2
    - KVM: s390: vsie: allow CRYCB FORMAT-1
    - KVM: s390: vsie: allow CRYCB FORMAT-0
    - KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-1
    - KVM: ...

Changed in linux-azure (Ubuntu):
status: Confirmed → Fix Released
Launchpad Janitor (janitor) wrote :
Download full text (13.9 KiB)

This bug was fixed in the package linux-azure - 4.15.0-1035.36

---------------
linux-azure (4.15.0-1035.36) bionic; urgency=medium

  * linux-azure: 4.15.0-1035.36 -proposed tracker (LP: #1806021)

  * [Hyper-V] Additional patches for Lv2 storage performance (LP: #1805304)
    - SAUCE: scsi: storvsc: Fix a race in sub-channel creation that can cause
      panic

linux-azure (4.15.0-1034.35) bionic; urgency=medium

  * linux-azure: 4.15.0-1034.35 -proposed tracker (LP: #1805412)

  * [Hyper-V] Additional patches for Lv2 storage performance (LP: #1805304)
    - SAUCE: Drivers: hv: vmbus: Remove x86-isms from arch independent drivers
    - SAUCE: x86/hyperv: Add a function to read both TSC and TSC page value
      simulateneously
    - SAUCE: x86/hyperv: Reenlightenment notifications support
    - SAUCE: x86/hyperv: Redirect reenlightment notifications on CPU offlining
    - SAUCE: x86/hyper-v: move hyperv.h out of uapi
    - SAUCE: x86/hyper-v: move definitions from TLFS to hyperv-tlfs.h
    - SAUCE: x86/hyper-v: allocate and use Virtual Processor Assist Pages
    - SAUCE: x86/hyper-v: define struct hv_enlightened_vmcs and clean field bits
    - SAUCE: X86/Hyper-V: Enlighten APIC access
    - SAUCE: X86/Hyper-V: Enable IPI enlightenments
    - SAUCE: X86/Hyper-V: Enhanced IPI enlightenment
    - SAUCE: x86/hyper-v: Fix the circular dependency in IPI enlightenment

linux-azure (4.15.0-1033.34) bionic; urgency=medium

  * linux-azure: 4.15.0-1033.34 -proposed tracker (LP: #1802559)

  [ Ubuntu: 4.15.0-42.45 ]

  * linux: 4.15.0-42.45 -proposed tracker (LP: #1803592)
  * [FEAT] Guest-dedicated Crypto Adapters (LP: #1787405)
    - KVM: s390: reset crypto attributes for all vcpus
    - KVM: s390: vsie: simulate VCPU SIE entry/exit
    - KVM: s390: introduce and use KVM_REQ_VSIE_RESTART
    - KVM: s390: refactor crypto initialization
    - s390: vfio-ap: base implementation of VFIO AP device driver
    - s390: vfio-ap: register matrix device with VFIO mdev framework
    - s390: vfio-ap: sysfs interfaces to configure adapters
    - s390: vfio-ap: sysfs interfaces to configure domains
    - s390: vfio-ap: sysfs interfaces to configure control domains
    - s390: vfio-ap: sysfs interface to view matrix mdev matrix
    - KVM: s390: interface to clear CRYCB masks
    - s390: vfio-ap: implement mediated device open callback
    - s390: vfio-ap: implement VFIO_DEVICE_GET_INFO ioctl
    - s390: vfio-ap: zeroize the AP queues
    - s390: vfio-ap: implement VFIO_DEVICE_RESET ioctl
    - KVM: s390: Clear Crypto Control Block when using vSIE
    - KVM: s390: vsie: Do the CRYCB validation first
    - KVM: s390: vsie: Make use of CRYCB FORMAT2 clear
    - KVM: s390: vsie: Allow CRYCB FORMAT-2
    - KVM: s390: vsie: allow CRYCB FORMAT-1
    - KVM: s390: vsie: allow CRYCB FORMAT-0
    - KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-1
    - KVM: s390: vsie: allow guest FORMAT-1 CRYCB on host FORMAT-2
    - KVM: s390: vsie: allow guest FORMAT-0 CRYCB on host FORMAT-2
    - KVM: s390: device attrs to enable/disable AP interpretation
    - KVM: s390: CPU model support for AP virtualization
    - s390: doc: detailed specifications for AP virtualization
    ...

Changed in linux-azure (Ubuntu Bionic):
status: Fix Committed → Fix Released
Marcelo Cerri (mhcerri) on 2019-04-09
Changed in linux-azure (Ubuntu Cosmic):
status: New → In Progress
Marcelo Cerri (mhcerri) on 2019-04-09
Changed in linux-azure (Ubuntu Cosmic):
status: In Progress → Invalid
To post a comment you must log in.