kernel BUG: soft lockup - CPU#0 stuck for 11s! [jfsCommit:2338]

[Mark Foster]

This keeps happening on our backup server which is running Ubuntu 8.04.1 LTS
Oct 15 08:15:06 sea-backup-001 kernel: [514527.213472] BUG: soft lockup - CPU#0 stuck for 11s! [jfsCommit:2338]
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214360] CPU 0:
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214361] Modules linked in: iptable_filter ip_tables x_tables 8021q parport_pc lp parport loop ipv6 evdev usbhid hid ipmi_si ipmi_msghandler container ata_piix pata_acpi button psmouse pcspkr serio_raw shpchp pci_hotplug iTCO_wdt iTCO_vendor_support jfs ata_generic ehci_hcd uhci_hcd libata usbcore tg3 cciss scsi_mod dm_mirror dm_snapshot dm_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214404] Pid: 2338, comm: jfsCommit Not tainted 2.6.24-19-server #1
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214406] RIP: 0010:[dm_mirror:_spin_unlock_irqrestore+0x8/0x20] [dm_mirror:_spin_unlock_irqrestore+0x8/0x20] _spin_unlock_irqrestore+0x8/0x10
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214414] RSP: 0018:ffff81003abc3da8 EFLAGS: 00000202
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214416] RAX: 00000000007fcaf0 RBX: 0000000000000100 RCX: 0000000001ae7c10
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214418] RDX: 00000000012eaff8 RSI: 0000000000000202 RDI: ffff81002682f510
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214420] RBP: ffffffff00000000 R08: ffff81000bce3b80 R09: ffff810027e92c00
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214422] R10: 0000000000000000 R11: 0000000000000000 R12: ffff81003f927ff0
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214424] R13: 0000000000021513 R14: 0000000000000000 R15: 0000000000000100
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214426] FS: 0000000000000000(0000) GS:ffffffff805c4000(0000) knlGS:0000000000000000
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214428] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214430] CR2: 00007f7f8b692000 CR3: 0000000038f73000 CR4: 00000000000006e0
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214432] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214434] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214435]
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214436] Call Trace:
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214451] [jfs:dbUpdatePMap+0x2a3/0x480] :jfs:dbUpdatePMap+0x2a3/0x480
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214460] [jfs:txAllocPMap+0xdd/0x120] :jfs:txAllocPMap+0xdd/0x120
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214467] [jfs:txUpdateMap+0x1ae/0x270] :jfs:txUpdateMap+0x1ae/0x270
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214472] [usbhid:__wake_up+0x43/0x1d50] __wake_up+0x43/0x70
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214479] [jfs:jfs_lazycommit+0x120/0x270] :jfs:jfs_lazycommit+0x120/0x270
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214482] [<ffffffff802362c0>] default_wake_function+0x0/0x10
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214489] [jfs:jfs_lazycommit+0x0/0x270] :jfs:jfs_lazycommit+0x0/0x270
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214493] [kthread+0x4b/0x80] kthread+0x4b/0x80
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214496] [child_rip+0xa/0x12] child_rip+0xa/0x12
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214500] [kthread+0x0/0x80] kthread+0x0/0x80
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214503] [child_rip+0x0/0x12] child_rip+0x0/0x12
Oct 15 08:15:06 sea-backup-001 kernel: [514527.214505]

root@sea-backup-001:/etc/cron.d# mount
/dev/cciss/c0d0p2 on / type jfs (rw,noatime,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
/sys on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,noexec,nosuid,nodev,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777)
udev on /dev type tmpfs (rw,mode=0755)
devshm on /dev/shm type tmpfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/mapper/system-backup on /bitpusher type jfs (rw,noatime)

root@sea-backup-001:/etc/cron.d# uname -a
Linux sea-backup-001 2.6.24-19-server #1 SMP Fri Jul 11 21:50:43 UTC 2008 x86_64 GNU/Linux

Kernel options (from /boot/grub/menu.lst)
kernel /boot/vmlinuz-2.6.24-19-server root=UUID=49cb3e65-df4a-4810-a7d2-ebc32f91611d ro console=tty0 console=ttyS1,115200 noapic

[Mark Foster (fostermarkd)]

wtf - almost two months and nobody has even triaged this?

[Dave Kleikamp (shaggy-linux)]

I'm not sure this will fix it, but it might.

It seems possible that if the jfsCommit thread is kept busy, and the metadata it is operating on is all in cache, it could hold onto the cpu longer than it should.

[Leann Ogasawara (leannogasawara)]

[This is an automated message. Apologies if it has reached you inappropriately.]

This bug was reported against the linux-meta package when it likely should have been reported against the linux package instead. We are automatically transitioning this to the linux kernel package so that the appropriate teams are notified and made aware of this issue. Thanks.

[Leann Ogasawara (leannogasawara)]

Hi Mark,

Can you confirm this issue remains with the latest Jaunty release? http://www.ubuntu.com/getubuntu/download . Thanks.

[kernel-janitor (kernel-janitor)]

[This is an automated message. Apologies if it has reached you inappropriately.]

This bug was flagged as having a patch attached. The Ubuntu Kernel Team's preferred policy is for all patches to be submitted and accepted into the upstream kernel before agreeing to merge them into the Ubuntu kernel. The goal for the Ubuntu kernel is to have little to no divergence from the upstream linux kernel source.

https://wiki.ubuntu.com/KernelTeam/KernelPatches has been written to document the suggested policy and procedures for helping get a patch merged upstream and subsequently into the Ubuntu kernel. Please take the time to review that wiki if this patch should be considered for inclusion into the upstream and Ubuntu kernel. Let us know if you have any questions or need any help via the Ubuntu Kernel Team mailing list. Thanks in advance.

[Mark Foster (fostermarkd)]

After some time this problem has gone away, It was only after we replaced 5 of the 12 disks and rebuild the system numerous times.

The disks were Seagate drives reporting ST3_INVALID_PFM on inquiry.

You can close this issue.

[Leann Ogasawara (leannogasawara)]

Closing per Mark's last comment. Thanks.