CPU Soft Lockups when stress-ng stack stressor runs with M.2 NVMe as root FS

Bug #1798127 reported by Jeff Lane on 2018-10-16
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
High
Joseph Salisbury
Xenial
Undecided
Unassigned
Bionic
High
Joseph Salisbury
Cosmic
High
Joseph Salisbury

Bug Description

This was reported by a hardware partner. The system set up is a server with 512GB RAM and an M.2 NVMe drive as the root filesystem/boot device.

Per the customer, when running the certification Memory Stress test (utilizing several stress-ng stressors run in sequence) the system freezes with CPU Soft Lockup errors appearing on console whe the "stack" stressor is run.

Tester has tried with 2.5” SATA (1TB), 2.5” NVMe (800GB), and M.2 NVMe (1.9TB).

So far, this only seems to affect the 4.15 kernel. The tester has tried using the 2.5" SATA SSD as the RootFS/Boot device and the tests pass on all attempts. It is ONLY when using the M.2 NVMe as the root / boot device that the tests cause a lockup. The tester is re-trying now with the 2.5" NVMe device to see if this only occurs with the M.2 NVMe.

The tester has tried this on the following while using the M.2 NVMe as the rootFS/Boot device:
Test run #1 – 16.04.5 at kernel 4.15; Result: Failed stress-ng memory on stack stressor
Test run #2 – 18.04.1 at kernel 4.15; Result: Failed stress-ng memory on stack stressor
Test run #3 – 16.04.5 at kernel 4.4; Result: Passed stress-ng memory test

The stress-ng command invoked at the time the soft lockups occur is this:

'stress-ng -k --aggressive --verify --timeout 300 --stack 0'

This can be reproduced by running the memory_stress_ng test script from the cert suite:

sudo /usr/lib/plainbox-provider-certification-server/bin/memory_stress_ng

It may be more easily reproducible running the stack stressor alone, or the whole memory stress script without dealing with Checkbox.

UPDATE: The tester also confirms that the 2.5" NVMe drives also fail with the 4.15 kernel and pass with the 4.4 kernel. The SSD works on all kernels.

Jeff Lane (bladernr) on 2018-10-16
tags: added: blocks-hwcert-server

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1798127

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Joseph Salisbury (jsalisbury) wrote :

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.19 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19-rc8

Changed in linux (Ubuntu):
importance: Undecided → Medium
status: Incomplete → Triaged
Changed in linux (Ubuntu Bionic):
status: New → Triaged
importance: Undecided → Medium
tags: added: kernel-da-key
Changed in linux (Ubuntu Bionic):
importance: Medium → High
Changed in linux (Ubuntu):
importance: Medium → High
tags: added: kernel-key
removed: kernel-da-key
Jeff Lane (bladernr) on 2018-10-17
description: updated
Alec Duroy (acduroy) on 2018-10-18
Changed in linux (Ubuntu):
status: Triaged → Confirmed
Alec Duroy (acduroy) wrote :

'kernel-fixed-upstream'

Alec Duroy (acduroy) wrote :

Successfully upgraded to 4.19. Stress-ng memory test passed without lockup issue. New rebuild kernel 4.19 has fixed the issue. Refer to text below:
-------
ubuntu@fluent-orca:~$ uname -r
4.19.0-041900rc8-generic

ubuntu@fluent-orca:~$ sudo stress-ng -k --aggressive --verify --timeout 300 --stack 0
stress-ng: info: [3516] dispatching hogs: 112 stack
stress-ng: info: [3516] successful run completed in 311.37s (5 mins, 11.37 secs) ubuntu@fluent-orca:~$
--------

Jeff Lane (bladernr) wrote :

I added a nomination for Xenial, but that may not be the right way (I'm not sure how to add a Xenial task like the Bionic task already added.

But this is a regression in as much as on Xenial this does not occur on the 4.4 kernel but shows up at least in the 4.15 HWE kernel. For Bionic it seems to exist in both 4.15 and 4.18 (in cosmic).

So any fix that is pulled into 4.18 will need to be pulled into 4.15 for both Bionic and Xenial.

Thanks!

Jeff Lane (bladernr) on 2018-11-07
tags: added: kernel-fixed-upstream
Changed in linux (Ubuntu Bionic):
status: Triaged → Confirmed
Joseph Salisbury (jsalisbury) wrote :

Would it be possible for you to test the latest upstream stable 4.18? It can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.18.19/

tags: added: kernel-da-key
removed: kernel-key
Changed in linux (Ubuntu Bionic):
assignee: nobody → Joseph Salisbury (jsalisbury)
status: Confirmed → Triaged
Changed in linux (Ubuntu Cosmic):
status: New → Triaged
importance: Undecided → High
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu Xenial):
status: New → Invalid
Changed in linux (Ubuntu):
status: Confirmed → Triaged
assignee: nobody → Joseph Salisbury (jsalisbury)
Jeff Lane (bladernr) wrote :

Why is Xenial marked invalid? AFAIK this affects 4.15, but not 4.4, which means Xenial HWE also has this regression.

Joseph Salisbury (jsalisbury) wrote :

The Xenial bug task is for the base Xenial kernel version 4.4. Any commits/fixes applied to Bionic flow down into Xenial HWE because Bionic is 4.15 based, which is the source for Xenial HWE.

Download full text (3.9 KiB)

Ahhh, ok. Thanks. That makes sense, then.
On Fri, Nov 16, 2018 at 12:01 AM Joseph Salisbury
<email address hidden> wrote:
>
> The Xenial bug task is for the base Xenial kernel version 4.4. Any
> commits/fixes applied to Bionic flow down into Xenial HWE because Bionic
> is 4.15 based, which is the source for Xenial HWE.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1798127
>
> Title:
> CPU Soft Lockups when stress-ng stack stressor runs with M.2 NVMe as
> root FS
>
> Status in linux package in Ubuntu:
> Triaged
> Status in linux source package in Xenial:
> Invalid
> Status in linux source package in Bionic:
> Triaged
> Status in linux source package in Cosmic:
> Triaged
>
> Bug description:
> This was reported by a hardware partner. The system set up is a
> server with 512GB RAM and an M.2 NVMe drive as the root
> filesystem/boot device.
>
> Per the customer, when running the certification Memory Stress test
> (utilizing several stress-ng stressors run in sequence) the system
> freezes with CPU Soft Lockup errors appearing on console whe the
> "stack" stressor is run.
>
> Tester has tried with 2.5” SATA (1TB), 2.5” NVMe (800GB), and M.2 NVMe
> (1.9TB).
>
> So far, this only seems to affect the 4.15 kernel. The tester has
> tried using the 2.5" SATA SSD as the RootFS/Boot device and the tests
> pass on all attempts. It is ONLY when using the M.2 NVMe as the root
> / boot device that the tests cause a lockup. The tester is re-trying
> now with the 2.5" NVMe device to see if this only occurs with the M.2
> NVMe.
>
> The tester has tried this on the following while using the M.2 NVMe as the rootFS/Boot device:
> Test run #1 – 16.04.5 at kernel 4.15; Result: Failed stress-ng memory on stack stressor
> Test run #2 – 18.04.1 at kernel 4.15; Result: Failed stress-ng memory on stack stressor
> Test run #3 – 16.04.5 at kernel 4.4; Result: Passed stress-ng memory test
>
> The stress-ng command invoked at the time the soft lockups occur is
> this:
>
> 'stress-ng -k --aggressive --verify --timeout 300 --stack 0'
>
> This can be reproduced by running the memory_stress_ng test script
> from the cert suite:
>
> sudo /usr/lib/plainbox-provider-certification-
> server/bin/memory_stress_ng
>
> It may be more easily reproducible running the stack stressor alone,
> or the whole memory stress script without dealing with Checkbox.
>
> UPDATE: The tester also confirms that the 2.5" NVMe drives also fail
> with the 4.15 kernel and pass with the 4.4 kernel. The SSD works on
> all kernels.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1798127/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: distribution=ubuntu; sourcepackage=linux; component=main; status=Triaged; importance=High; <email address hidden>;
> Launchpad-Bug: distribution=ubuntu; distroseries=xenial; sourcepackage=linux; component=main; status=Invalid; importance=Undecided; assignee=None;
> Launchpad-Bug: distribution=ubuntu; distroseries=bionic; s...

Read more...

Brad Figg (brad-figg) on 2019-07-24
tags: added: cscc
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers