ulatencyd causes random kernel crashes during boot

[Mikko Rantalainen]

I've been running Ubuntu 12.04 LTS 64 bit with ulatencyd for months now and after rebooting yesterday, I hit a very hard to diagnose kernel crash. It seemed that random processes (usually Xorg, sometimes apache2, mysqld etc.) caused kernel BUG and PANIC triggers. After trying all the kernels I had installed it got clear that the problem was not caused by kernel version.

I finally was able to detect that the problem was caused by ulatencyd doing something early in the boot process that caused kernel to crash. I fixed the problem by adding "exit 1" at the start of the /etc/init.d/ulatencyd script to prevent normal startup of ulatencyd without uninstalling it.

The system had been running for two months before the restart so ulatencyd had probably collected lots of data during that time.

My guess is that ulatencyd causes some kind of race condition in the kernel early in the boot process that causes kernel crashes.

I'll attach a couple of photos of the kernel crashes.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: ulatencyd 0.5.0-4ubuntu1
ProcVersionSignature: Ubuntu 3.2.0-35.55-generic 3.2.34
Uname: Linux 3.2.0-35-generic x86_64
ApportVersion: 2.0.1-0ubuntu15.1
Architecture: amd64
Date: Wed Dec 19 10:10:37 2012
InstallationMedia: Ubuntu 12.04.1 LTS "Precise Pangolin" - Release amd64 (20120823.1)
MarkForUpload: True
SourcePackage: ulatencyd
UpgradeStatus: No upgrade log present (probably fresh install)
mtime.conffile..etc.init.d.ulatencyd: 2012-12-19T10:06:56.904449

[Mikko Rantalainen (mira)]

Additional info: I was able to boot into "recovery mode" and diagnose the system from there. However, trying to run normal boot resulted in kernel crash every time (I booted about 30 times while trying to fix the system). The crash occured usually before the purple "Ubuntu" boot screen was visible but one the system crashed in the lightdm greeter and I was able to move the mouse cursor and move focus to password input before the crash. I guess that the exact moment of the crash was depending on what ulatencyd was doing during startup and during that boot, ulatencyd was running slightly later than usual.

The system is installed on Intel 320 series SSD so the actual boot is pretty fast. My monitor is quite slow to display the image after resolution change so I'm not sure what the system was trying to display the moment the crashes occurred.

[Mikko Rantalainen (mira)]

Additional info: once the system is running, I can remove the "exit 1" from /etc/init.d/ulatencyd and then start ulatencyd process so this seems to be some kind of race condition in the kernel. Some crashes have kernel watchdog messages saying "Watchdog detected hard LOCKUP on cpu 1" (sometimes the LOCKUP was on cpu 0).

Is it possible that the ulatencyd gets started before the system is running on both cores (this system has dual core processor) and kernel crashes with some feature that ulatencyd uses if all CPUs are not yet up? Perhaps this really is a kernel bug that is triggered by ulatencyd?

[Mikko Rantalainen (mira)]

The kernels I was able to reproduce the problem with:

linux-image-3.2.0-29-generic:
  Installed: 3.2.0-29.46
  Candidate: 3.2.0-29.46
  Version table:
 *** 3.2.0-29.46 0
        500 http://fi.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ precise-security/main amd64 Packages
        100 /var/lib/dpkg/status
linux-image-3.2.0-30-generic:
  Installed: 3.2.0-30.48
  Candidate: 3.2.0-30.48
  Version table:
 *** 3.2.0-30.48 0
        500 http://fi.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        100 /var/lib/dpkg/status
linux-image-3.2.0-31-generic:
  Installed: 3.2.0-31.50
  Candidate: 3.2.0-31.50
  Version table:
 *** 3.2.0-31.50 0
        500 http://fi.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ precise-security/main amd64 Packages
        100 /var/lib/dpkg/status
linux-image-3.2.0-32-generic:
  Installed: 3.2.0-32.51
  Candidate: 3.2.0-32.51
  Version table:
 *** 3.2.0-32.51 0
        500 http://fi.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ precise-security/main amd64 Packages
        100 /var/lib/dpkg/status
linux-image-3.2.0-33-generic:
  Installed: 3.2.0-33.52
  Candidate: 3.2.0-33.52
  Version table:
 *** 3.2.0-33.52 0
        500 http://fi.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ precise-security/main amd64 Packages
        100 /var/lib/dpkg/status
linux-image-3.2.0-34-generic:
  Installed: 3.2.0-34.53
  Candidate: 3.2.0-34.53
  Version table:
 *** 3.2.0-34.53 0
        500 http://fi.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ precise-security/main amd64 Packages
        100 /var/lib/dpkg/status
linux-image-3.2.0-35-generic:
  Installed: 3.2.0-35.55
  Candidate: 3.2.0-35.55
  Version table:
 *** 3.2.0-35.55 0
        500 http://fi.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ precise-security/main amd64 Packages
        100 /var/lib/dpkg/status

Additional details for the last one:
$ uname -a
Linux semyol-329-b 3.2.0-35-generic #55-Ubuntu SMP Wed Dec 5 17:42:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
$ cat /proc/version_signature
Ubuntu 3.2.0-35.55-generic 3.2.34

[Mikko Rantalainen (mira)]

Does ulatencyd collect any data from the system that is used for deciding grouping and scheduling changes? I see only static choices in /etc/ulatencyd/simple.d and /etc/ulatencyd/rules but perhaps I'm missing something...

[Mikko Rantalainen (mira)]

Possibly duplicate of Bug #1034099 ?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.