Kernel Oops - Unable to handle kernel paging request at ffffc900117da000

[Marco Scannadinari]

Ocasionally, when I shut down, I am sent to the plymouth splash screen as usual, however the ubuntu logo and the progress dots are static - they are not moving. This has been happening more recently over the past week.
The hard drive LED on my case is not lit up at all when this happens.

EDIT:
More info has been uncovered, after quietsplash was turned off at shutdown and boot, these logs were in my view:

"""
BUG: Unable to handle kernel paging request at ffffc900117da000
Oops: 000 [#1] SMP

PID comm: chromium-browser
RIP [<ffffffff81066cff>]
warn_slowpath_common
(A LOT of text)

At /build/buildd/linux-3.2.0/kernel/watchdog.c:241 watchdog_overflow_callback+0x9a/0xc0 ()
Watchdog detected hard lockup on cpu 0
modules linked in: nls_utf 8 ... ... etc etc
-----[end snippet]-----
"""
(This is not exact)

Bug didnt occur while the latest Catalyst driver was installed from the AMD website using the --buildpkg -precise option. Uninstalled and reinstalled the ubuntu packaged one via:
sudo apt-get install fglrx fglrx-amdcccle fglrx -dev
 shutdown and bug reoccured.
I can now conclude that the bug exists with the Ubuntu-packaged fglrx driver. Luckily i took a photo. (though linux-crashdump didnt pick it up in /var/crash/)

http://i47.tinypic.com/2s6mq7r.jpg
http://i48.tinypic.com/5ey9sk.jpg

These bugs appear to by synonymous with my problem http://lkml.indiana.edu/hypermail/linux/kernel/1107.1/02053.html
and https://bugzilla.redhat.com/show_bug.cgi?id=822287.

This is the code that is supposed to trigger it http://lxr.free-electrons.com/source/kernel/watchdog.c#L214.

Attatched:
Xorg.log
Syslog.log

Motherboard ASUS M5A78L -M LX V2
GPU Sapphire Radeon HD 6570 WITH UBUNTU-PACKAGED FGLRX INSTALLED DRIVER
CPU AMD Athlon II X2 255 3.2GHz
HD WD Caviar 500GB
Monitor Benq G2255

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: plymouth 0.8.2-2ubuntu30
ProcVersionSignature: Ubuntu 3.2.0-29.46-generic 3.2.24
Uname: Linux 3.2.0-29-generic x86_64
NonfreeKernelModules: fglrx
ApportVersion: 2.0.1-0ubuntu12
Architecture: amd64
Date: Thu Aug 30 11:42:47 2012
DefaultPlymouth: /lib/plymouth/themes/ubuntu-logo/ubuntu-logo.plymouth
InstallationMedia: Ubuntu 12.04 LTS "Precise Pangolin" - Release amd64 (20120425)
MachineType: System manufacturer System Product Name
ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-29-generic root=UUID=e97b3482-c14c-4320-b621-19dbc962822b ro
ProcEnviron:
 LANGUAGE=en_GB:en
 PATH=(custom, no user)
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-29-generic root=UUID=e97b3482-c14c-4320-b621-19dbc962822b ro
SourcePackage: plymouth
TextPlymouth: /lib/plymouth/themes/ubuntu-text/ubuntu-text.plymouth
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 03/26/2012
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 0902
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: M5A78L-M LX V2
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev X.0x
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr0902:bd03/26/2012:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnM5A78L-MLXV2:rvrRevX.0x:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer

[Marco Scannadinari (zheoffec)]

In the meantime, can anyone suggest a safe way to shutdown bypassing the plymouth splash, as ths may be damaging my drive, MB, etc

[Steve Langasek (vorlon)]

On Thu, Aug 30, 2012 at 10:58:28AM -0000, Marco M Scannadinari wrote:
> In the meantime, can anyone suggest a safe way to shutdown bypassing the
> plymouth splash, as ths may be damaging my drive, MB, etc

It is very unlikely that plymouth itself is causing the issue. You can try
booting without the 'splash' option to see if the hang still happens without
plymouth's graphical splash.

[Marco Scannadinari (zheoffec)]

How may I do this? I have previously removed the ... = "quietsplash" perameter and this had started the bootup process in verbose, however this does not affect shutdown.
What, at this point is in the list of possible culprits? Graphics Card? Driver?

[Steve Langasek (vorlon)]

Ah - indeed, I see that your /proc/cmdline shows you are booting without 'splash' as a boot option. Which means this should not be showing a splash screen either on startup or on shutdown. If it's still showing a splash screen on shutdown, that would seem to be a bug in plymouth. However, I can't reproduce this behavior at all. Can you please do the following:

- switch to console with Ctrl-Alt-F1
- log in
- run 'sudo -s' to get a root shell
- confirm that 'grep splash /proc/cmdline' doesn't return anything (i.e., you are currently booted without splash)
- run 'service lightdm stop' (and confirm that this command succeeds... if it doesn't, then something else is wrong)
- run 'plymouthd --mode=shutdown'
- run 'plymouth show-splash'
- confirm that you do NOT get a graphical splash screen, but instead get a switch to a different text console
- hit Ctrl-Alt-F1 again to switch back to your root login
- run 'plymouth quit' to stop plymouth
- run 'service lightdm start' to restart X

Please report the results from the above test.

> What, at this point is in the list of possible culprits? Graphics Card? Driver?

At this point, I have no idea at all what's causing the hang on shutdown. It *may* be the graphics driver - but what shows up on the screen may also be entirely unrelated to the hang. The point of asking you to run without 'splash' was so we could see the output of whatever is happening at shutdown.

[Marco Scannadinari (zheoffec)]

Thanks for your help, although I did, after removing the quietsplash perameter from /etc/default/grub notice that it was not affecting shutown, only the the bootup process.
And can I not run thes commands from the gnome terminal?

[Marco Scannadinari (zheoffec)]

EDIT

Thanks for your help, although I did, after removing the quietsplash perameter from /etc/default/grub notice that it was not affecting shutown, only the the bootup process, so I re added the perameter because the boot logs were not important to me.
And can I not run thes commands from the gnome terminal?

[Steve Langasek (vorlon)]

On Thu, Aug 30, 2012 at 09:58:46PM -0000, Marco M Scannadinari wrote:

> And can I not run thes commands from the gnome terminal?

No, you cannot! The X server needs to be shut down to run these tests. If
you try to run them from inside a gnome-terminal, you will crash your GUI.
:)

[Marco Scannadinari (zheoffec)]

Ok done!

Results:
Ctrl Alt F1 : OK
Log in : OK
sudo -s : OK
grep splash /proc/cmdline : OK (Returned nothing and gave me a new prompt)
service lightdm stop : OK (Something like "lightdm stopping/waiting", then new prompt)
plymouthd --mode=shutdown : OK
plymouth show-splash : OK (Gave me what I would have seen when pressing Esc on a normal splash screen, but stopped on "Stopping system V runlevel compatibility [OK]. I then pressed Ctrl Alt F1)
Ctrl Alt F1 : OK
plymouth quit : OK
service lightdm start : OK (Got me into the Unity greeter like on a usual boot.)

[Marco Scannadinari (zheoffec)]

So now I should just keep using my box as normal until I get a halt like this, then I will look at what the log says and report back to this page.
But, just for the sake of another opinion, do you think holding down the power button damages my setup (If your response is that it damages the Hard Disk, what if the Hard Drive LED is not active?)

[Steve Langasek (vorlon)]

Ok, it sounds like plymouth is working as expected when called as 'plymouthd --mode=shutdown'. So I think the fixed splash screen that you see on shutdown is *not* due to plymouth. It may be a bug in the video driver, or it may be something that lightdm is doing on shutdown, I'm not sure. Reassigning to lightdm for further triaging.

> But, just for the sake of another opinion, do you think holding down the power
> button damages my setup (If your response is that it damages the Hard Disk,
> what if the Hard Drive LED is not active?)

An unclean shutdown of the system, whether due to a hang or a power-off, can cause corruption and data loss on your filesystem. It will not damage the hard disk itself, but you can lose data. Usually the risk of data loss is very small.

To further reduce your risk, when this hang happens you can try to use the SysRq key (Alt+PrtScn): SysRq+S to sync the disk, SysRq+U to remount read-only, and SysRq+B to reboot. If this sequence *doesn't* let you reboot, then that further tells us that the kernel has crashed, pointing the finger strongly at the video driver.

[Marco Scannadinari (zheoffec)]

Ok! Thanks alot! I have previously read of the "Magic" sysrq key in a magazine called Linux Format.

[Marco Scannadinari (zheoffec)]

Darn! It happened to me again, this time with the logs, but I missed the crucial point where it said "BUG: Overflow ... ? ... Kernel ..."
Here is what I have written down:

WARNING
BUG: (Did not completely see this message, something about kernel overflow)

At /build/buildd/linux-3.2.0/kernel/watchdog.c:241 watchdog_overflow_callback+0x9a/0xc0 ()
Watchdog detected hard lockup on cpu 0

Do you know where these logs are saved?
At the moment it looks like some sort of kernel bug and some lockup on the CPU.

I had looked up the /build directory but it was nowhere to be found.
This seems to be the same problem as me, though slightly different http://lkml.indiana.edu/hypermail/linux/kernel/1107.1/02053.html, also this https://bugzilla.redhat.com/show_bug.cgi?id=822287.
This, I beleive is the code in the kernel which triggers this log output which may help you to understand the cause of the error http://lxr.free-electrons.com/source/kernel/watchdog.c#L214.

[Marco Scannadinari (zheoffec)]

Also, if of any intrest, I had enabled the Secure Virtualisation Mode (AMD-V) in the BIOS for utilising multiple cores in VirtualBox. And, come to mention it I did see something like 'vbox' in those logs...

[Marco Scannadinari (zheoffec)]

This bug appears to be identical although it is triggered upod waking from suspend.

[Steve Langasek (vorlon)]

> Do you know where these logs are saved?

Unfortunately by the time the kernel is in such a state, it is probably no longer writing logs.

> I had looked up the /build directory but it was nowhere to be found.

This was a filesystem path on the machine where the kernel was built; you won't find it.

I would suggest trying to reproduce the bug after disabling the fglrx driver.

[Marco Scannadinari (zheoffec)]

After a fresh install without the chromium browser, bug still persists.
Will remove the fglrx driver and inspect untill the bug reoccurs.

[Marco Scannadinari (zheoffec)]

Have removed the fglrx driver and I am now with the somewhat slow Gallium driver. None of the bug so far...
At the moment, my two main suspects are the Linux kernel or the fglrx driver.

And BTW, Alt + SysRq + REISUB has no effect on the freeze. So this is a kernel crash?

[Marco Scannadinari (zheoffec)]

I am going to reinstall the fglrx driver and try and debug these messages when they come up according to this wiki page https://wiki.ubuntu.com/DebuggingKernelOops.

[Marco Scannadinari (zheoffec)]

Bug didnt occur while the latest Catalyst driver was installed from the AMD website using the --buildpkg -precise option. Uninstalled and reinstalled the ubuntu packaged one via:
sudo apt-get install fglrx fglrx-amdcccle fglrx -dev
 shutdown and bug reoccured.
I can now conclude that the bug exists with the Ubuntu-packaged fglrx driver. Luckily i took a photo. (though linux-crashdump didnt pick it up in /var/crash/)

http://i47.tinypic.com/2s6mq7r.jpg
http://i48.tinypic.com/5ey9sk.jpg

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.