Since May, some of my servers crashed from time to time. This one crashed today. No information is available in syslog and other log files. The CPU load was not very high, but a lot of swap was used, even though the disks were not very active.
Here's the OCRed screenshot of the hanging system:
[5288936.542226] [<ffffffff81011e9b>] cpu_idle+0xeb/0x110
[5288936.542229] [<ffffffff81552b33>] start_secondary+0xa8/Oxaa
[5288936.542231] Code: 06 89 85 c0 fe ff ff c7 85 c4 fe ff ff 01 00 00 00 e9 9'
fb ff ff 90 48 8b 95 e0 fe ff ff 48 8b 45 a8 8b 72 08 48 e1 e0 0a 31 d2 <48) f’
f6 48 8b 75 b0 48 89 45 a0 31 c0 48 85 f6 74 0c 48 8b 45
[5288936.542248] RIP [<ffffffff810562a4>] find_busiest_group+0x634/0x8f0
[5288936.542251] RSP <ffff88101cce3c58>
[5288936.542253] ---[ end trace 17d6ca884388b3c0 ]--—
[5288936.542255] Kernel panic — not syncing: attempted to kill the idle task!
[5288936.54225?] Pid: 0, comm: swapper Tainted: G D 2.6.32-24-server #
-Ubuntu
[5288936.542259] Call Trace:
[5288936.542261] [<ffffffff8155?dda>] panic+0x78/0x137
[5288936.542266] [<ffffffff8106b53a>] do_exit+0x35a/0x380
[5288936.542269] [<ffffffff8155bc40>] oops_end+0xb0/0xf0
[5288936.5422?1] [<ffffffff8101?11b>] die+0x5b/0x90
[5288936.5422?3] [<ffffffff8155b514>] do_trap+0xc4/0x1?0
[5288936.5422?6] [<ffffffff810150af>] do_divide_error+0x8f/0xb0
[5288936.5422?9] [<ffffffff810562a4>] ? find_busiest_group+0x634/0x8f0
[5288936.542282] [<ffffffff812ba38c>] ? put_dec+0x10c/0x110
[5288936.542284] [<ffffffff812ba6?e>] ? number+0x2ee/0x320
[5288936.542286] [<ffffffff812b8ece>] ? __rb_erase_co1or+0x1be/0x1d0
[5288936.542289] [<ffffffff81013f1b>] divide_error+0x1b/0x20
[5288936.542292] [<ffffffff810562a4>] ? find_busiest_group+0x634/0x8f0
[5288936.542295] [<ffffffff81055ea4>] ? find_busiest_group+0x234/0x8f0
[5288936.542299] [<ffffffff8105c8a8>] load_balance_newidle+0xa8/0x310
[5288936.542302] [<ffffffff81558a8a>] thread_return+0x352/0x418
[5288936.542305] [<ffffffff81011e9b>] cpu_idle+0xeb/0x110
[5288936.54230?] [<ffffffff81552b33>] start_secondary+0xa8/Oxaa
---
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access /dev/snd/: No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
CurrentDmesg:
DistroRelease: Ubuntu 10.04
Frequency: Once every few days.
InstallationMedia: Ubuntu-Server 10.04.1 LTS "Lucid Lynx" - Release amd64 (20100816.2)
MachineType: DEPO Computers X8DT3
Package: linux (not installed)
PciMultimedia:
ProcCmdLine: BOOT_IMAGE=/vmlinuz-2.6.32-41-server root=/dev/mapper/S--WEB--01-root ro quiet
ProcEnviron:
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.32-41.88-server 2.6.32.59+drm33.24
Regression: No
Reproducible: No
Tags: lucid needs-upstream-testing
Uname: Linux 2.6.32-41-server x86_64
UserGroups:
dmi.bios.date: 09/14/2010
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 2.0a
dmi.board.asset.tag: 1234567890
dmi.board.name: X8DT3
dmi.board.vendor: Supermicro
dmi.board.version: 2.0
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 17
dmi.chassis.vendor: Supermicro
dmi.chassis.version: 1234567890
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr2.0a:bd09/14/2010:svnDEPOComputers:pnX8DT3:pvr1234567890:rvnSupermicro:rnX8DT3:rvr2.0:cvnSupermicro:ct17:cvr1234567890:
dmi.product.name: X8DT3
dmi.product.version: 1234567890
dmi.sys.vendor: DEPO Computers
On July, 13, when the system crashed, it ran the kernel 2.6.32-24-server, as seen from the screenshot above. Apport was launched under new kernel 2.6.32-41-server, which was installed on April, 23, but the server was not rebooted.