Ubuntu 16.04 - Full EEH Recovery Support for NVMe devices

Bug #1602724 reported by bugproxy on 2016-07-13
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
High
Canonical Kernel Team
Xenial
Undecided
Tim Gardner

Bug Description

== Comment: #0 - Heitor Ricardo Alves de Siqueira <email address hidden> - 2016-07-12 12:54:27 ==
Current nvme driver in Ubuntu 16.04 kernel does not handle error recovery; we are missing some patches from the upstream nvme driver.

We would like to ask Canonical to cherry pick the following patches for the 16.04 kernel, if possible:
    * 9396dec916c0 ("nvme: use a work item to submit async event requests")
    * 79f2b358c9ba ("nvme: don't poll the CQ from the kthread")
    * 2d55cd5f511d ("nvme: replace the kthread with a per-device watchdog timer")
    * 9bf2b972afea ("NVMe: Fix reset/remove race")
    * c875a7093f04 ("nvme: Avoid reset work on watchdog timer function during error recovery")
    * a5229050b69c ("NVMe: Always use MSI/MSI-x interrupts")

bugproxy (bugproxy) on 2016-07-13
tags: added: architecture-ppc64le bugnameltc-143694 severity-high targetmilestone-inin1604
Changed in ubuntu:
assignee: nobody → Taco Screen team (taco-screen-team)
affects: ubuntu → linux (Ubuntu)
Changed in linux (Ubuntu):
assignee: Taco Screen team (taco-screen-team) → Canonical Kernel Team (canonical-kernel-team)
importance: Undecided → High
status: New → Triaged

------- Comment From <email address hidden> 2016-07-18 10:24 EDT-------
*** Bug 143100 has been marked as a duplicate of this bug. ***

Tim Gardner (timg-tpi) wrote :
Changed in linux (Ubuntu):
status: Triaged → Fix Released
Changed in linux (Ubuntu Xenial):
assignee: nobody → Tim Gardner (timg-tpi)
status: New → In Progress
Changed in linux (Ubuntu Xenial):
status: In Progress → Fix Committed
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2016-08-08 15:05 EDT-------
(In reply to comment #9)
> I Have Tested this with , Test Kernel available at
> http://people.canonical.com/~rtg/eeh-lp1602724/ . on Ubuntu 16.04.1
>
> Test Kernel :
> root@everest-lp13-leaf:~# uname -a
> Linux everest-lp13-leaf 4.4.0-32-generic #51 SMP Tue Jul 19 21:41:04 UTC
> 2016 ppc64le ppc64le ppc64le GNU/Linux
>
>
>
> Nvme (Leaf) is getting recovered till 5 times on triggering the EEH, But
> "hitting a kernel crash" after on 6th time trigger of EEH.
>

This is most likely fixed by

http://lists.infradead.org/pipermail/linux-nvme/2016-August/005670.html
("[PATCH v2] nvme: Suspend all queues before deletion")

Which is not upstream yet. Once it gets accepted, it should be pushed to Ubuntu via another bugzilla. When that happens, we'll need a new test kernel for this one.

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2016-08-08 15:09 EDT-------
(In reply to comment #13)
> (In reply to comment #9)
> > I Have Tested this with , Test Kernel available at
> > http://people.canonical.com/~rtg/eeh-lp1602724/ . on Ubuntu 16.04.1
> >
> > Test Kernel :
> > root@everest-lp13-leaf:~# uname -a
> > Linux everest-lp13-leaf 4.4.0-32-generic #51 SMP Tue Jul 19 21:41:04 UTC
> > 2016 ppc64le ppc64le ppc64le GNU/Linux
> >
> >
> >
> > Nvme (Leaf) is getting recovered till 5 times on triggering the EEH, But
> > "hitting a kernel crash" after on 6th time trigger of EEH.
> >
>
> This is most likely fixed by
>
> http://lists.infradead.org/pipermail/linux-nvme/2016-August/005670.html
> ("[PATCH v2] nvme: Suspend all queues before deletion")
>
> Which is not upstream yet. Once it gets accepted, it should be pushed to
> Ubuntu via another bugzilla. When that happens, we'll need a new test
> kernel for this one.

Canonical,

For a little more context, we have identified an issue in the test kernel you provided. After a sequence of 6 EEHs, DD will attempt to remove the adapter, which ends up hitting a BUG_ON.

We think the above patch is a fix, but it's still not confirmed. Can you provide a kernel with that patch also applied for verification? It's still not upstream yet, but it has already been ack-ed by the driver maintainer, Keith Busch.

Stefan Bader (smb) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-xenial
Tim Gardner (timg-tpi) wrote :

Test kernel at http://people.canonical.com/~rtg/eeh-lp1602724/ with upstream commit c21377f8366c95440d533edbe47d070f662c62ef ('nvme: Suspend all queues before deletion') applied.

Changed in linux (Ubuntu Xenial):
status: Fix Committed → In Progress
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2016-08-22 13:11 EDT-------
(In reply to comment #16)
> Test kernel at http://people.canonical.com/~rtg/eeh-lp1602724/ with upstream
> commit c21377f8366c95440d533edbe47d070f662c62ef ('nvme: Suspend all queues
> before deletion') applied.

This test kernel is not ok, it stalls the wq:

[ 540.097661] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 540.103320] 1-...: (1 GPs behind) idle=d35/140000000000000/0 softirq=2385/2386 fqs=65
[ 540.103411] (detected by 11, t=5472 jiffies, g=1335, c=1334, q=793)
[ 540.103492] Task dump for CPU 1:
[ 540.103539] kworker/u32:1 D 0000000000000000 0 101 0 0x00000800
[ 540.103656] Call Trace:
[ 540.103692] [c00000017bc539c0] [c00000017bc53a00] 0xc00000017bc53a00 (unreliable)
[ 540.103805] [c00000017bc53a00] [d000000001614480] nvme_suspend_queue+0x30/0x150 [nvme]
[ 540.103914] [c00000017bc53a30] [d000000001616850] nvme_dev_disable+0x110/0x440 [nvme]
[ 540.104022] [c00000017bc53b10] [d000000001617e60] nvme_reset_work+0xe0/0x1120 [nvme]
[ 540.104132] [c00000017bc53c50] [c0000000000dd630] process_one_work+0x1e0/0x5a0
[ 540.104239] [c00000017bc53ce0] [c0000000000ddb84] worker_thread+0x194/0x680
[ 540.104331] [c00000017bc53d80] [c0000000000e6680] kthread+0x110/0x130
[ 540.104424] [c00000017bc53e30] [c000000000009538] ret_from_kernel_thread+0x5c/0xa4
[ 604.094501] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 604.094699] 1-...: (1 GPs behind) idle=d35/140000000000000/0 softirq=2385/2386 fqs=82
[ 604.094700] (detected by 5, t=21472 jiffies, g=1335, c=1334, q=1283)
[ 604.094705] Task dump for CPU 1:

Can you provide the backported patch for verification?

Tim Gardner (timg-tpi) wrote :

git://kernel.ubuntu.com/rtg/ubuntu-xenial.git eeh-lp1602724
commit fcb182b0c9d79b9dae70f2c9104997a0ec63f080 ('nvme: Suspend all queues before deletion')

The 6 patches mentioned in the bug description have already been merged but not yet packaged in a released version, though they _are_ included in this test kernel.

Changed in linux (Ubuntu Xenial):
status: In Progress → Fix Committed
Stefan Bader (smb) wrote :

Part of the fix was in the current update. Marking the verification done to proceed with the current cycle. This bug needs to be reset to fix-committed for tracking the remaining patch after it gets closed by automatic processes.

tags: added: verification-done-xenial
removed: verification-needed-xenial
Launchpad Janitor (janitor) wrote :
Download full text (13.4 KiB)

This bug was fixed in the package linux - 4.4.0-36.55

---------------
linux (4.4.0-36.55) xenial; urgency=low

  [ Stefan Bader ]

  * Release Tracking Bug
    - LP: #1612305

  * I2C touchpad does not work on AMD platform (LP: #1612006)
    - SAUCE: pinctrl/amd: Remove the default de-bounce time

  * CVE-2016-5696
    - tcp: make challenge acks less predictable

linux (4.4.0-35.54) xenial; urgency=low

  [ Stefan Bader ]

  * Release Tracking Bug
    - LP: #1611215

  * [i915_bpo] Sync with v4.7 (LP: #1609742)
    - SAUCE: i915_bpo: Sync with v4.7

  * s390/cio: fix reset of channel measurement block (LP: #1609415)
    - s390/cio: allow to reset channel measurement block

  * in Ubuntu16.10: Hit on Call traces and system goes down when transactional
    memory tests are running in 32TB Brazos system (LP: #1606786)
    - powerpc/tm: Avoid SLB faults in treclaim/trecheckpoint when RI=0
    - powerpc/tm: Fix stack pointer corruption in __tm_recheckpoint()

  * Power Menu does not display after press the Power Button (LP: #1609204)
    - intel-vbtn: new driver for Intel Virtual Button
    - [config] enable CONFIG_INTEL_VBTN=m

  * OptiPlex 7450 AIO hangs when rebooting (LP: #1608762)
    - x86/reboot: Add Dell Optiplex 7450 AIO reboot quirk

  * virtualbox+usb 3.0 breaks boot, -28 kernel works (LP: #1604058)
    - SAUCE: xhci: Fix soft lockup in xhci_pci_probe path when XHCI_STATE_HALTED

  * linux-kernel: Freeing IRQ from IRQ context (LP: #1597908)
    - block: defer timeouts to a workqueue

  * Tunnel offload indications not stripped from encapsulated packets, causing
    performance overhead (LP: #1602755)
    - tunnels: Remove encapsulation offloads on decap.

  * lm-sensors is throwing "ERROR: Can't get value of subfeature temp1_input:
    I/O error" for be2net driver (LP: #1607387)
    - be2net: perform temperature query in adapter regardless of its interface
      state

  * Dell dock MAC Address pass through doesn't work in Ubuntu (LP: #1579984)
    - r8152: Add support for setting pass through MAC address on RTL8153-AD

  * vmxnet3 LRO IPv6 performance issues (stalling TCP) (LP: #1605494)
    - Driver: Vmxnet3: set CHECKSUM_UNNECESSARY for IPv6 packets

  * ISST-LTE:pVM:monklp5:Ubuntu16.04.1:system crashed at
    lpfc_sli4_scmd_to_wqidx_distr (LP: #1597974)
    - SAUCE: lpfc: fix oops in lpfc_sli4_scmd_to_wqidx_distr() from
      lpfc_send_taskmgmt()

  * Backport cxlflash shutdown patch to Xenial SRU (LP: #1605405)
    - SAUCE: cxlflash: Verify problem state area is mapped before notifying
      shutdown

  * Xenial update to v4.4.16 stable release (LP: #1607404)
    - mac80211: fix fast_tx header alignment
    - mac80211: mesh: flush mesh paths unconditionally
    - mac80211_hwsim: Add missing check for HWSIM_ATTR_SIGNAL
    - mac80211: Fix mesh estab_plinks counting in STA removal case
    - EDAC, sb_edac: Fix rank lookup on Broadwell
    - IB/cm: Fix a recently introduced locking bug
    - IB/mlx4: Properly initialize GRH TClass and FlowLabel in AHs
    - powerpc/pseries: Fix IBM_ARCH_VEC_NRCORES_OFFSET since POWER8NVL was added
    - powerpc/tm: Always reclaim in start_thread() for exec() class syscalls
    - usb: dwc2: fix reg...

Changed in linux (Ubuntu Xenial):
status: Fix Committed → Fix Released
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2016-09-12 22:46 EDT-------
(In reply to comment #20)
> Part of the fix was in the current update. Marking the verification done to
> proceed with the current cycle. This bug needs to be reset to fix-committed
> for tracking the remaining patch after it gets closed by automatic processes.

Hi smb,

The final patch, which was not included in the current release depends on a fixup I pushed upstream. I retested this, and it allowed it to successfully disable the device after 6 EEHs;

The fixup is not in Linus tree yet, but it should be included in the next -rc. it's queued in Jens Axboe's linux-block tree:

82469c59d222 ("nvme: Don't suspend admin queue that wasn't created")

<https://git.kernel.org/cgit/linux/kernel/git/axboe/linux-block.git/commit/?h=for-linus&id=82469c59d222f839ded5cd282172258e026f9112>

This needs to go on top of

c21377f8366c ("nvme: Suspend all queues before deletion")

Which is already in the Ubuntu's kernel -next branch.

Thanks,

Tim Gardner (timg-tpi) wrote :
Changed in linux (Ubuntu Xenial):
status: Fix Released → In Progress
Launchpad Janitor (janitor) wrote :
Download full text (22.8 KiB)

This bug was fixed in the package linux - 4.4.0-38.57

---------------
linux (4.4.0-38.57) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1620658

  * CIFS client: access problems after updating to kernel 4.4.0-29-generic
    (LP: #1612135)
    - Revert "UBUNTU: SAUCE: (namespace) Bypass sget() capability check for nfs"
    - fs: Call d_automount with the filesystems creds

  * apt-key add fails in overlayfs (LP: #1618572)
    - SAUCE: overlayfs: fix regression in whiteout detection

linux (4.4.0-37.56) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1618040

  * [Feature] Instruction decoder support for new SKX instructions- AVX512
    (LP: #1591655)
    - x86/insn: perf tools: Fix vcvtph2ps instruction decoding
    - x86/insn: Add AVX-512 support to the instruction decoder
    - perf tools: Add AVX-512 support to the instruction decoder used by Intel PT
    - perf tools: Add AVX-512 instructions to the new instructions test

  * [Ubuntu 16.04] FCoE Lun not visible in OS with inbox driver - Issue with
    ioremap() call on 32bit kernel (LP: #1608652)
    - lpfc: Correct issue with ioremap() call on 32bit kernel

  * [Feature] turbostat support for Skylake-SP server (LP: #1591802)
    - tools/power turbostat: decode more CPUID fields
    - tools/power turbostat: CPUID(0x16) leaf shows base, max, and bus frequency
    - tools/power turbostat: decode HWP registers
    - tools/power turbostat: Decode MSR_MISC_PWR_MGMT
    - tools/power turbostat: allow sub-sec intervals
    - tools/power turbostat: Intel Xeon x200: fix erroneous bclk value
    - tools/power turbostat: Intel Xeon x200: fix turbo-ratio decoding
    - tools/power turbostat: re-name "%Busy" field to "Busy%"
    - tools/power turbostat: add --out option for saving output in a file
    - tools/power turbostat: fix compiler warnings
    - tools/power turbostat: make fewer systems calls
    - tools/power turbostat: show IRQs per CPU
    - tools/power turbostat: show GFXMHz
    - tools/power turbostat: show GFX%rc6
    - tools/power turbostat: detect and work around syscall jitter
    - tools/power turbostat: indicate SMX and SGX support
    - tools/power turbostat: call __cpuid() instead of __get_cpuid()
    - tools/power turbostat: correct output for MSR_NHM_SNB_PKG_CST_CFG_CTL dump
    - tools/power turbostat: bugfix: TDP MSRs print bits fixing
    - tools/power turbostat: SGX state should print only if --debug
    - tools/power turbostat: print IRTL MSRs
    - tools/power turbostat: initial BXT support
    - tools/power turbostat: decode BXT TSC frequency via CPUID
    - tools/power turbostat: initial SKX support

  * [BYT] display hotplug doesn't work on console (LP: #1616894)
    - drm/i915/vlv: Make intel_crt_reset() per-encoder
    - drm/i915/vlv: Reset the ADPA in vlv_display_power_well_init()
    - drm/i915/vlv: Disable HPD in valleyview_crt_detect_hotplug()
    - drm/i915: Enable polling when we don't have hpd

  * [Feature]intel_idle enabling on Broxton-P (LP: #1520446)
    - intel_idle: add BXT support

  * [Feature] EDAC: Update driver for SKX-SP (LP: #1591815)
    - [Config] CONFIG_EDAC_SKX=m
    - EDAC, skx_edac: Ad...

Changed in linux (Ubuntu Xenial):
status: In Progress → Fix Released
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2016-09-22 18:01 EDT-------
Thanks Canonical,

Ran 6 EEHs in a row and got recovered each time. The last one removed the device correctly.

Closing this.

------- Comment (attachment only) From <email address hidden> 2016-09-22 18:00 EDT-------

Erics (dublio) wrote :

hello,

who can give me the test script of 6 EEHs ?

Thanks a lot.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers