NMI watchdog: BUG: soft lockup on Guest upon boot (KVM)

Bug #1727331 reported by bugproxy on 2017-10-25
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
The Ubuntu-power-systems project
Critical
Canonical Kernel Team
linux (Ubuntu)
Critical
Joseph Salisbury
Xenial
Critical
Joseph Salisbury

Bug Description

== SRU Justification ==
Xenial does not include the patch:
"KVM: PPC: Book3S: Treat VTB as a per-subcore register, not per-thread",
which is commit 88b02cf97bb7 in the upstream Linux kernel repository. The
symptom of not having this patch is that guests running with threads > 1
(that is, in SMT2, SMT4 or SMT8 mode) can give spurious soft-lockup messages
when they are not in fact locked up, if the guest kernel uses the VTB
(virtual timebase) register in its softlockup detector code.

IBM provided this backport of commit 88b02cf97bb7.

== Fix ==
commit 88b02cf97bb7e742db3e31671d54177e3e19fd89
Author: Paul Mackerras <email address hidden>
Date: Thu Sep 15 13:42:52 2016 +1000

    KVM: PPC: Book3S: Treat VTB as a per-subcore register, not per-thread

== Regression Potential ==
This patch is specific to powerpc. It was provided by and tested by IBM.

Environment:
Host OS: Ubuntu 16.04 (4.4.0-97-generic)
Guest OS: Ubuntu 16.04

Host is running NovaLink and GPFS. Guest gets filesystem from gpfs cluster.

Issue:

While booting the guest OS went into recovery. dmesg shows CPU soft lockup. This issue is easily recreatable when multiple VMs (about 10) are being started simultaneously.

.......................................................................................................................

 - Paul Mackerras <email address hidden> - 2017-10-24 00:42:25 ==

Looking at kernel sources:

The Ubuntu host kernel (4.4.0-97) does not include the patch "KVM: PPC: Book3S: Treat VTB as a per-subcore register, not per-thread", which is commit 88b02cf97bb7 in the upstream Linux kernel repository. The symptom of not having this patch is that guests running with threads > 1 (that is, in SMT2, SMT4 or SMT8 mode) can give spurious soft-lockup messages when they are not in fact locked up, if the guest kernel uses the VTB (virtual timebase) register in its softlockup detector code.

This is a backport of commit 88b02cf97bb7 from the upstream Linux kernel repository to the Ubuntu 4.4 kernel. It is the fix for the problem of seeing spurious soft lockup messages in guests running in an SMT mode greater than 1.

These tests were run on a system with 20 cores. Each VM had 1 core and 4 threads (SMT=4).
It is working very well with this patch

CVE References

tags: added: architecture-ppc64le bugnameltc-160018 severity-critical targetmilestone-inin16043
Changed in ubuntu:
assignee: nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
affects: ubuntu → kernel-package (Ubuntu)
affects: kernel-package (Ubuntu) → linux (Ubuntu)
Changed in ubuntu-power-systems:
importance: Undecided → Critical
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
Joseph Salisbury (jsalisbury) wrote :

Do you need me to build a test kernel with this backport and submit and SRU request for Xenial? Or do you plan on doing that?

Changed in linux (Ubuntu):
importance: Undecided → Critical
status: New → Triaged
tags: added: kernel-da-key

------- Comment From <email address hidden> 2017-10-27 02:13 EDT-------
We need official build ..

Changed in ubuntu-power-systems:
status: New → Triaged
Changed in linux (Ubuntu Xenial):
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu):
assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) → Joseph Salisbury (jsalisbury)
status: Triaged → In Progress
Changed in ubuntu-power-systems:
status: Triaged → In Progress
Joseph Salisbury (jsalisbury) wrote :

I built a Xenial test kernel with the back port of commit 88b02cf97bb7 posted in comment #1. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1727331/

Can you test this kernel and see if it resolves this bug? Be sure to install both the linux-image and linux-image-extra .deb packages.

If this kernel resolves the bug, I'll submit an SRU request for inclusion in Xenial.

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-11-02 23:28 EDT-------
Test kernel from Canonical works. Please request a SRU request.

description: updated
Manoj Iyer (manjo) on 2017-11-06
tags: added: triage-g
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-11-09 08:51 EDT-------
Any update on the SRU request?

------- Comment From <email address hidden> 2017-11-14 01:25 EDT-------
Marking LUVELLA MCFADDEN update "External"

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-11-14 03:50 EDT-------
please help submitting an SRU request for inclusion in Xenial.

Stefan Bader (smb) on 2017-11-20
Changed in linux (Ubuntu Xenial):
status: In Progress → Fix Committed
Manoj Iyer (manjo) on 2017-11-20
Changed in ubuntu-power-systems:
status: In Progress → Fix Committed
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-11-20 15:08 EDT-------
So is the official fix avail yet? Any outlook?

------- Comment From <email address hidden> 2017-11-21 01:07 EDT-------
Any update on the official fix ?

Joseph Salisbury (jsalisbury) wrote :

The fix is in the -proposed repository. It will be released with the Ubuntu-4.4.0-102.125 kernel.

Changed in linux (Ubuntu):
status: In Progress → Fix Committed
Khaled El Mously (kmously) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-xenial
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-11-30 14:02 EDT-------
I have verified the fix on xenial.

Linux neo160.blr.stglabs.ibm.com 4.4.0-102-generic #125-Ubuntu SMP Tue Nov 21 15:13:58 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux

tags: added: verification-done-xenial
removed: verification-needed-xenial
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-12-04 23:56 EDT-------
tag changed to verification-done-xenial

Launchpad Janitor (janitor) wrote :
Download full text (9.5 KiB)

This bug was fixed in the package linux - 4.4.0-103.126

---------------
linux (4.4.0-103.126) xenial; urgency=low

  * linux: 4.4.0-103.126 -proposed tracker (LP: #1736181)

  * CVE-2017-1000405
    - mm, thp: Do not make page table dirty unconditionally in touch_p[mu]d()

  * CVE-2017-16939
    - netlink: add a start callback for starting a netlink dump
    - ipsec: Fix aborted xfrm policy dump crash

linux (4.4.0-102.125) xenial; urgency=low

  * linux: 4.4.0-102.125 -proposed tracker (LP: #1733541)

  * tar -x sometimes fails on overlayfs (LP: #1728489)
    - ovl: check if all layers are on the same fs
    - ovl: persistent inode number for directories

  * NVMe timeout is too short (LP: #1729119)
    - nvme: update timeout module parameter type

  * Set PANIC_TIMEOUT=10 on Power Systems (LP: #1730660)
    - [Config]: Set PANIC_TIMEOUT=10 on ppc64el

  * Cannot pair BLE remote devices when using combo BT SoC (LP: #1731467)
    - Bluetooth: increase timeout for le auto connections

  * CIFS errors on 4.4.0-98, but not on 4.4.0-97 with same config (LP: #1729337)
    - SMB3: Validate negotiate request must always be signed

  * Plantronics P610 does not support sample rate reading (LP: #1719853)
    - ALSA: usb-audio: Add sample rate quirk for Plantronics P610

  * Invalid btree pointer causes the kernel NULL pointer dereference
    (LP: #1729256)
    - xfs: reinit btree pointer on attr tree inactivation walk

  * Samba mount/umount in docker container triggers kernel Oops (LP: #1729637)
    - ipv6: only call ip6_route_dev_notify() once for NETDEV_UNREGISTER
    - ipv6: fix NULL dereference in ip6_route_dev_notify()

  * [kernel] tty/hvc: Use opal irqchip interface if available (LP: #1728098)
    - tty/hvc: Use opal irqchip interface if available

  * Device hotplugging with MPT SAS cannot work for VMWare ESXi (LP: #1730852)
    - scsi: mptsas: Fixup device hotplug for VMWare ESXi

  * NMI watchdog: BUG: soft lockup on Guest upon boot (KVM) (LP: #1727331)
    - KVM: PPC: Book3S: Treat VTB as a per-subcore register, not per-thread

  * Attempt to map rbd image from ceph jewel/luminous hangs (LP: #1728739)
    - crush: ensure bucket id is valid before indexing buckets array
    - crush: ensure take bucket value is valid
    - crush: add chooseleaf_stable tunable
    - crush: decode and initialize chooseleaf_stable
    - libceph: advertise support for TUNABLES5
    - libceph: MOSDOpReply v7 encoding

  * Xenial update to 4.4.98 stable release (LP: #1732698)
    - adv7604: Initialize drive strength to default when using DT
    - video: fbdev: pmag-ba-fb: Remove bad `__init' annotation
    - PCI: mvebu: Handle changes to the bridge windows while enabled
    - xen/netback: set default upper limit of tx/rx queues to 8
    - drm: drm_minor_register(): Clean up debugfs on failure
    - KVM: PPC: Book 3S: XICS: correct the real mode ICP rejecting counter
    - iommu/arm-smmu-v3: Clear prior settings when updating STEs
    - powerpc/corenet: explicitly disable the SDHC controller on kmcoge4
    - ARM: omap2plus_defconfig: Fix probe errors on UARTs 5 and 6
    - crypto: vmx - disable preemption to enable vsx in aes_ctr.c
    - iio: trigger: free trigger...

Read more...

Changed in linux (Ubuntu Xenial):
status: Fix Committed → Fix Released
Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
Changed in ubuntu-power-systems:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers