Kernel panic on reboot after sched_autogroup_enabled disabled

[Chris Adamson]

I disabled the autogroup scheduler on precise. I added a line to my rc.local to disable it on boot as follows:

echo 0 > /proc/sys/kernel/sched_autogroup_enabled

In linux-image-3.2.0-29, linux-image-3.2.0-30, and linux-image-3.2.0-31 this causes a kernel panic during the reboot or halt sequence.

I have been unable to store logs of the output but I always see a unable to dereference null pointer. The function reported in the stack trace tends to be update_cfs_shares, update_cfs_load and reweight_entity.

Here are some other details about my Ubuntu version:
lsb_release -rd
Description: Ubuntu 12.04.1 LTS
Release: 12.04

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-29-generic (not installed)
ProcVersionSignature: Ubuntu 3.2.0-27.43-generic 3.2.21
Uname: Linux 3.2.0-27-generic x86_64
NonfreeKernelModules: nvidia
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
ApportVersion: 2.0.1-0ubuntu13
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: addo 2869 F.... pulseaudio
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xfbbd8000 irq 58'
   Mixer name : 'Realtek ALC888'
   Components : 'HDA:10ec0888,15d9f380,00100202'
   Controls : 43
   Simple ctrls : 20
CurrentDmesg:
 Error: command ['sh', '-c', 'dmesg | comm -13 --nocheck-order /var/log/dmesg -'] failed with exit code 1: comm: /var/log/dmesg: Permission denied
 dmesg: write failed: Broken pipe
Date: Mon Sep 24 10:30:50 2012
InstallationMedia: Ubuntu 12.04 LTS "Precise Pangolin" - Release amd64 (20120425)
MachineType: Supermicro X8SAX
ProcEnviron:
 SHELL=/bin/bash
 TERM=xterm
 PATH=(custom, user)
 LANG=en_AU.UTF-8
 LANGUAGE=en_AU:en
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-27-generic root=UUID=fdd5524c-2bb2-4cce-8fbf-886c004c8640 ro
RfKill:

SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
WifiSyslog:

dmi.bios.date: 09/17/10
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 2.0
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: X8SAX
dmi.board.vendor: Supermicro
dmi.board.version: 2.0
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: Supermicro
dmi.chassis.version: 1234567890
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr2.0:bd09/17/10:svnSupermicro:pnX8SAX:pvr1234567890:rvnSupermicro:rnX8SAX:rvr2.0:cvnSupermicro:ct3:cvr1234567890:
dmi.product.name: X8SAX
dmi.product.version: 1234567890
dmi.sys.vendor: Supermicro
---
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
ApportVersion: 2.0.1-0ubuntu13
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: dfbiadmin 2730 F.... pulseaudio
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xf7fd8000 irq 58'
   Mixer name : 'Realtek ALC883'
   Components : 'HDA:10ec0883,15d9f380,00100002'
   Controls : 43
   Simple ctrls : 20
DistroRelease: Ubuntu 12.04
HibernationDevice: RESUME=UUID=f7078ca7-88d8-4e7b-bfa9-f30ffabfc04c
InstallationMedia: Ubuntu 12.04 LTS "Precise Pangolin" - Release amd64 (20120425)
IwConfig:
 lo no wireless extensions.

 eth1 no wireless extensions.

 eth0 no wireless extensions.
MachineType: Supermicro X8SAX
NonfreeKernelModules: nvidia
Package: linux (not installed)
ProcEnviron:
 TERM=xterm
 SHELL=/bin/bash
 PATH=(custom, no user)
 LANG=en_AU.UTF-8
 LANGUAGE=en_AU:en
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-29-generic root=UUID=7394fe9b-d364-4e4d-8eb1-3bad6fcf7f49 ro
ProcVersionSignature: Ubuntu 3.2.0-29.46-generic 3.2.24
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-29-generic N/A
 linux-backports-modules-3.2.0-29-generic N/A
 linux-firmware 1.79.1
RfKill:

Tags: precise
Uname: Linux 3.2.0-29-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

dmi.bios.date: 11/26/2008
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1.0a
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: X8SAX
dmi.board.vendor: Supermicro
dmi.board.version: 1234567890
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: Supermicro
dmi.chassis.version: 1234567890
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1.0a:bd11/26/2008:svnSupermicro:pnX8SAX:pvr1234567890:rvnSupermicro:rnX8SAX:rvr1234567890:cvnSupermicro:ct3:cvr1234567890:
dmi.product.name: X8SAX
dmi.product.version: 1234567890
dmi.sys.vendor: Supermicro

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1055222

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

apport information

[Chris Adamson (addo81)]

I couldnt find any way of storing the kernel panic stack trace as it happens after rsyslog and/or klogd is terminated.

[Chris Adamson (addo81)]

Was able to capture the stack trace in a Virtualbox window...

[Chris Adamson (addo81)]

Screenshot part 2

[Doug Smythies (dsmythies)]

I have the same issue, and have repeated it on two computers (with up to date Ubuntu server 12.04) now (and I have a third I can test also).
Procedure:
1.) start with a fresh boot.
2.) from a tty session Issue "sudo sysctl kernel.sched_autogroup_enabled=0" (or via the method Chris used above)
3.) open an ssh session to the computer (in my case I am using putty from a windows computer).
4.) from either the putty session or the tty session issue "sudo shutdown -r now".
5.) the computer will crash.

Note: if I disable autogroup as a kernel option in grub, then the crash will not occur. I.e. this line in grub:
GRUB_CMDLINE_LINUX_DEFAULT="noautogroup"

I found this bug report while doing my due dilegence searching before entering a new bug report.

[Chris Adamson (addo81)]

Thank you for the workaround.

[Carey Underwood (cwillu)]

I ran a bisect which traced it to this commit: "4f83989 sched: Fix race in task_group()"

This commit is included in 3.5.5 and 3.6-rc1 and later.

[Shaun Thomas (0-sthomas)]

This broke startup and shutdown on all our HP DL380 G7's and our Dell r710s. We didn't use the setting on other servers, so can't confirm.

We're wondering if restarting a service recently caused a kernel panic due to this bug as well, but can't reproduce reliably.

[Luis Henriques (henrix)]

The fix for this issue has finally hit the Precise kernel master-next branch and should be released in 3.2.0-36 (it won't make it into 3.2.0-36.57). I've uploaded a test kernel for anyone willing to give it a try (I've tested it myself and I'm not able to reproduce the issue anymore). Here's the URL for the test kernels (both 64 and 32 bits):

http://people.canonical.com/~henrix/lp1055222/v1/

[Doug Smythies (dsmythies)]

I tried the 64 bit test kernel. I went back and forth between the test kernel and the current kernel a few times, showing issue fixed with the test kernel and issue present with the current kernel.

[Luis Henriques (henrix)]

Great, thank you for your time. I'll post a comment again once this fix is released (again, it won't be present in the next Precise kernel to be released).

[Luis Henriques (henrix)]

This bug should be fixed now in the Precised kernel in -proposed pocket. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Luis Henriques (henrix)]

Tagging bug as fix released.