softlockups/rcu sched

[Ryan Harper]

1.$ lsb_release -rd
Description: Ubuntu Artful Aardvark (development branch)
Release: 17.10

2. $ apt-cache policy linux-image-virtual
linux-image-virtual:
  Installed: 4.13.0.11.12
  Candidate: 4.13.0.11.12
  Version table:
 *** 4.13.0.11.12 500
        500 http://archive.ubuntu.com/ubuntu artful/main amd64 Packages
        100 /var/lib/dpkg/status

3. System shutsdown cleanly, qemu exits non-zero

4. System stays hung until testcase timesout.

Single VCPU kvm guest on Artful host.

Reached target Shutdown.
[ 60.036030] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1]
[ 88.036012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1]
[ 96.920015] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 96.920526] (detected by 0, t=15002 jiffies, g=3321, c=3320, q=377)
[ 96.921161] All QSes seen, last rcu_sched kthread activity 15002 (4294916526-4294901524), jiffies_till_next_fqs=1, root ->qsmask 0x0
[ 96.922409] rcu_sched kthread starved for 15002 jiffies! g3321 c3320 f0x2 RCU_GP_WAIT_FQS(3) ->state=0x0
[ 124.036015] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1]
[ 152.036020] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1]
[ 180.036016] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1]
[ 208.036013] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1]
[ 236.036015] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1]
[ 264.036012] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1]
[ 276.940010] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 276.940520] (detected by 0, t=60007 jiffies, g=3321, c=3320, q=377)
[ 276.941088] All QSes seen, last rcu_sched kthread activity 60007 (4294961531-4294901524), jiffies_till_next_fqs=1, root ->qsmask 0x0
[ 276.942165] rcu_sched kthread starved for 60007 jiffies! g3321 c3320 f0x2 RCU_GP_WAIT_FQS(3) ->state=0x0

ProblemType: Bug
DistroRelease: Ubuntu 17.10
Package: linux-image-4.13.0-11-generic 4.13.0-11.12
ProcVersionSignature: User Name 4.13.0-11.12-generic 4.13.1
Uname: Linux 4.13.0-11-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Oct 9 15:06 seq
 crw-rw---- 1 root audio 116, 33 Oct 9 15:06 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.7-0ubuntu1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
Date: Mon Oct 9 15:13:30 2017
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.13.0-11-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
RelatedPackageVersions:
 linux-restricted-modules-4.13.0-11-generic N/A
 linux-backports-modules-4.13.0-11-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.10.2-1ubuntu1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-artful
dmi.modalias: dmi:bvnSeaBIOS:bvr1.10.2-1ubuntu1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-artful:cvnQEMU:ct1:cvrpc-i440fx-artful:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-artful
dmi.sys.vendor: QEMU

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.14 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14-rc4

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[Ryan Harper (raharper)]

On Tue, Oct 10, 2017 at 2:30 PM, Joseph Salisbury <
<email address hidden>> wrote:

> Did this issue start happening after an update/upgrade? Was there a
> prior kernel version where you were not having this particular problem?
>

Around, September this year is when we started noticing the softlockups in
Artful guests
this continues now in Bionic guests as well. Host is Xenial, we're now on
Xenial-hwe
 (4.10.0.40-generic x86).

We still see the issue daily.

>
> Would it be possible for you to test the latest upstream kernel? Refer
> to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest
> v4.14 kernel[0].
>

System is used in CI production, so we can't easily test upstream.

>
> If this bug is fixed in the mainline kernel, please add the following
> tag 'kernel-fixed-upstream'.
>
> If the mainline kernel does not fix this bug, please add the tag:
> 'kernel-bug-exists-upstream'.
>
> Once testing of the upstream kernel is complete, please mark this bug as
> "Confirmed".
>
>
> Thanks in advance.
>
> [0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14-rc4
>
> ** Changed in: linux (Ubuntu)
> Importance: Undecided => Medium
>
> ** Changed in: linux (Ubuntu)
> Status: Confirmed => Incomplete
>
> ** Tags added: kernel-da-key
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1722311
>
> Title:
> softlockups/rcu sched
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/
> 1722311/+subscriptions
>