kernel BUG at /build/buildd/linux-3.2.0/lib/radix-tree.c:356

[David Andruczyk]

Ubuntu 3.2.0-64.97-virtual 3.2.59

One of our Production systems crashed with the following kernel.log messages (we use a custom syslog setup):

Jun 13 01:06:04 www-node01 kernel: [593611.895631] ------------[ cut here ]------------
Jun 13 01:06:04 www-node01 kernel: [593611.895822] kernel BUG at /build/buildd/linux-3.2.0/lib/radix-tree.c:35
6!
Jun 13 01:06:04 www-node01 kernel: [593611.895906] invalid opcode: 0000 [#1] SMP
Jun 13 01:06:04 www-node01 kernel: [593611.896003] CPU 1
Jun 13 01:06:04 www-node01 kernel: [593611.896024] Modules linked in: pcnet32 vmblock(O) vmsync(O) vmhgfs(O) n
fsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc psmouse vmwgfx ttm serio_raw ppdev shpchp drm vmw_balloon ext
2 i2c_piix4 lp parport_pc mac_hid vmci(O) vmxnet(O) parport floppy e1000 mptspi mptscsih mptbase
Jun 13 01:06:04 www-node01 kernel: [593611.896311]
Jun 13 01:06:04 www-node01 kernel: [593611.896328] Pid: 16194, comm: bzip2 Tainted: G O 3.2.0-64-vir
tual #97-Ubuntu VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
Jun 13 01:06:04 www-node01 kernel: [593611.896466] RIP: 0010:[<ffffffff81313511>] [<ffffffff81313511>] radix_
tree_insert+0x171/0x180
Jun 13 01:06:04 www-node01 kernel: [593611.896557] RSP: 0018:ffff88010c1c1a08 EFLAGS: 00010086
Jun 13 01:06:04 www-node01 kernel: [593611.896609] RAX: ffff880055519630 RBX: 00000000000000bd RCX: 00000000ff
fffffa
Jun 13 01:06:04 www-node01 kernel: [593611.896678] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff880091
240528
Jun 13 01:06:04 www-node01 kernel: [593611.896747] RBP: ffff88010c1c1a48 R08: 0000000000000000 R09: ffffea0004
1cbae0
Jun 13 01:06:04 www-node01 kernel: [593611.896815] R10: ffffea00047cc040 R11: ffff88013fffbe70 R12: ffff880091
240528
Jun 13 01:06:04 www-node01 kernel: [593611.897037] R13: ffffea00047cc040 R14: 0000000000000000 R15: 00000000ff
ffffff
Jun 13 01:06:04 www-node01 kernel: [593611.897306] FS: 00007f3da2203740(0000) GS:ffff88013fc80000(0000) knlGS
:0000000000000000
Jun 13 01:06:04 www-node01 kernel: [593611.897505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 13 01:06:04 www-node01 kernel: [593611.897621] CR2: 0000000001c6a3e0 CR3: 0000000115df7000 CR4: 0000000000
0006e0
Jun 13 01:06:04 www-node01 kernel: [593611.897841] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000
000000
Jun 13 01:06:04 www-node01 kernel: [593611.898058] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000
000400
Jun 13 01:06:04 www-node01 kernel: [593611.898254] Process bzip2 (pid: 16194, threadinfo ffff88010c1c0000, tas
k ffff8800b29e5bc0)
Jun 13 01:06:04 www-node01 kernel: [593611.898453] Stack:
Jun 13 01:06:04 www-node01 kernel: [593611.898533] ffff880125c9b5d0 ffffea00047cc040 ffff880091240520 ffffea0
0047cc040
Jun 13 01:06:04 www-node01 kernel: [593611.898730] ffff880091240538 00000000000000bd ffff880091240520 0000000
0000000d0
Jun 13 01:06:04 www-node01 kernel: [593611.898927] ffff88010c1c1a78 ffffffff81118ad8 ffffea00047cc040 ffff880
091240520
Jun 13 01:06:04 www-node01 kernel: [593611.899123] Call Trace:
Jun 13 01:06:04 www-node01 kernel: [593611.899219] [<ffffffff81118ad8>] add_to_page_cache_locked.part.27+0x58
/0xd0
Jun 13 01:06:04 www-node01 kernel: [593611.899347] [<ffffffff81118bd5>] add_to_page_cache_locked+0x85/0xa0
Jun 13 01:06:04 www-node01 kernel: [593611.899469] [<ffffffff81118c11>] add_to_page_cache_lru+0x21/0x50
Jun 13 01:06:04 www-node01 kernel: [593611.899590] [<ffffffff81118cb8>] grab_cache_page_write_begin+0x78/0xe0
Jun 13 01:06:04 www-node01 kernel: [593611.899715] [<ffffffff811f9a50>] ext3_write_begin+0x80/0x270
Jun 13 01:06:04 www-node01 kernel: [593611.899832] [<ffffffff810514f4>] ? check_preempt_curr+0x84/0xa0
Jun 13 01:06:04 www-node01 kernel: [593611.899951] [<ffffffff8111818a>] generic_perform_write+0xca/0x210
Jun 13 01:06:04 www-node01 kernel: [593611.900071] [<ffffffff810538c6>] ? ttwu_do_activate.constprop.178+0x66
/0x70
Jun 13 01:06:04 www-node01 kernel: [593611.900202] [<ffffffff8111832d>] generic_file_buffered_write+0x5d/0x90
Jun 13 01:06:04 www-node01 kernel: [593611.900325] [<ffffffff81119c99>] __generic_file_aio_write+0x229/0x440
Jun 13 01:06:04 www-node01 kernel: [593611.900449] [<ffffffff8104d3d8>] ? __wake_up_common+0x58/0x90
Jun 13 01:06:04 www-node01 kernel: [593611.900567] [<ffffffff811928eb>] ? touch_atime+0x6b/0x160
Jun 13 01:06:04 www-node01 kernel: [593611.900680] [<ffffffff81119f22>] generic_file_aio_write+0x72/0xe0
Jun 13 01:06:04 www-node01 kernel: [593611.900801] [<ffffffff8117898a>] do_sync_write+0xda/0x120
Jun 13 01:06:04 www-node01 kernel: [593611.900915] [<ffffffff812d9118>] ? apparmor_file_permission+0x18/0x20
Jun 13 01:06:04 www-node01 kernel: [593611.901040] [<ffffffff8129e7fc>] ? security_file_permission+0x2c/0xb0
Jun 13 01:06:04 www-node01 kernel: [593611.901163] [<ffffffff81178f31>] ? rw_verify_area+0x61/0xf0
Jun 13 01:06:04 www-node01 kernel: [593611.901278] [<ffffffff81179293>] vfs_write+0xb3/0x180
Jun 13 01:06:04 www-node01 kernel: [593611.901388] [<ffffffff811795ba>] sys_write+0x4a/0x90
Jun 13 01:06:04 www-node01 kernel: [593611.901510] [<ffffffff81665842>] system_call_fastpath+0x16/0x1b
Jun 13 01:06:04 www-node01 kernel: [593611.901628] Code: c3 41 8b 44 24 04 4d 89 6c 24 08 a9 00 00 00 01 75 17
 a9 00 00 00 02 74 99 0f 0b 0f 1f 80 00 00 00 00 b8 f4 ff ff ff eb 8b 0f 0b <0f> 0b 0f 0b 0f 0b 66 0f 1f 84 00
 00 00 00 00 55 48 89 e5 41 57
Jun 13 01:06:04 www-node01 kernel: [593611.902078] RIP [<ffffffff81313511>] radix_tree_insert+0x171/0x180
Jun 13 01:06:04 www-node01 kernel: [593611.902200] RSP <ffff88010c1c1a08>
Jun 13 01:06:04 www-node01 kernel: [593611.902854] ---[ end trace e75964c5e86e02b0 ]---

 lspci -vnvn output:

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image (not installed)
ProcVersionSignature: Ubuntu 3.2.0-64.97-virtual 3.2.59
Uname: Linux 3.2.0-64-virtual x86_64
AlsaDevices:
 total 0
 crw-rw---T 1 root audio 116, 1 Jun 13 08:51 seq
 crw-rw---T 1 root audio 116, 33 Jun 13 08:51 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.0.1-0ubuntu17.6
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
CurrentDmesg:
 [ 30.588514] eth1: no IPv6 routers present
 [ 40.705692] auditd (2315): /proc/2315/oom_adj is deprecated, please use /proc/2315/oom_score_adj instead.
 [ 40.897566] pcnet32: pcnet32.c:v1.35 21.Apr.2008 <email address hidden>
 [ 42.601458] init: plymouth-upstart-bridge main process (1099) killed by TERM signal
Date: Fri Jun 13 09:07:56 2014
HibernationDevice: RESUME=UUID=f5af135f-4114-4437-a670-9aac8b575df7
IwConfig:
 lo no wireless extensions.

 eth1 no wireless extensions.

 eth0 no wireless extensions.
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MachineType: VMware, Inc. VMware Virtual Platform
MarkForUpload: True
PciMultimedia:

ProcEnviron:
 LANGUAGE=en_US:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.2.0-64-virtual root=/dev/mapper/vg00-root ro elevator=noop
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-64-virtual N/A
 linux-backports-modules-3.2.0-64-virtual N/A
 linux-firmware 1.79.14
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
WifiSyslog:

dmi.bios.date: 01/15/2013
dmi.bios.vendor: Phoenix Technologies LTD
dmi.bios.version: 6.00
dmi.board.name: 440BX Desktop Reference Platform
dmi.board.vendor: Intel Corporation
dmi.board.version: None
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 1
dmi.chassis.vendor: No Enclosure
dmi.chassis.version: N/A
dmi.modalias: dmi:bvnPhoenixTechnologiesLTD:bvr6.00:bd01/15/2013:svnVMware,Inc.:pnVMwareVirtualPlatform:pvrNone:rvnIntelCorporation:rn440BXDesktopReferencePlatform:rvrNone:cvnNoEnclosure:ct1:cvrN/A:
dmi.product.name: VMware Virtual Platform
dmi.product.version: None
dmi.sys.vendor: VMware, Inc.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Do you have a way to reproduce this bug or was it a one time event?

[David Andruczyk (david-andruczyk)]

This was a one time event, and I don't know of a way to replicate it at this time. Monitors of the box showed it was largely idle at the time the bug exhibited itself. VMware reported repeated high/low CPU consumption from the time of the bug until it was forced off and restarted. The host WAS responding to pings, but was unable to be logged into and monitors of the host stopped at the time of the syslog message of the bug assertion. cacti graphs stopped capturing data at the time of the hang

Prior to the hang, cacti graphs showed a fair bit of Disk IO (for this VM), about 543 sectors/second read, and 1300 sectors/second write, where daily averages are 15 read and 44 write sectors/second respectively.

cpu graph from VMware attached