system hangs and errors at /build/buildd/linux-3.2.0/arch/x86/kernel/apic/ipi.c:113 default_send_IPI_mask_logical+0xdc/0xf0()

[Jean-Baptiste Lallement]

Precise i386 20111129

Fresh installation of Precise with kernel 3.2.0-2-generic-pae on a Dell PE/R210
After installation and reboot,during normal operations (bzr, ps, file copy, ...) , system suddenly hangs

The following error is displayed in syslog (full syslog attached)

[ 260.527275] ------------[ cut here ]------------
[ 260.527282] WARNING: at /build/buildd/linux-3.2.0/arch/x86/kernel/apic/ipi.c:113 default_send_IPI_mask_logical+0xdc/0xf0()
[ 260.527285] Hardware name: PowerEdge R210 II
[ 260.527286] Modules linked in: vesafb dcdbas joydev lp parport usbhid hid mpt2sas scsi_transport_sas bnx2 raid_class
[ 260.527296] Pid: 274, comm: udevd Not tainted 3.2.0-2-generic-pae #5-Ubuntu
[ 260.527298] Call Trace:
[ 260.527304] [<c158e783>] ? printk+0x2d/0x2f
[ 260.527309] [<c1059fb2>] warn_slowpath_common+0x72/0xa0
[ 260.527312] [<c102cb4c>] ? default_send_IPI_mask_logical+0xdc/0xf0
[ 260.527315] [<c102cb4c>] ? default_send_IPI_mask_logical+0xdc/0xf0
[ 260.527318] [<c105a002>] warn_slowpath_null+0x22/0x30
[ 260.527320] [<c102cb4c>] default_send_IPI_mask_logical+0xdc/0xf0
[ 260.527325] [<c111e3d4>] ? anon_vma_chain_free+0x14/0x20
[ 260.527327] [<c111fa68>] ? __put_anon_vma+0x48/0x80
[ 260.527329] [<c111fa68>] ? __put_anon_vma+0x48/0x80
[ 260.527333] [<c103b6ba>] flush_tlb_others_ipi+0xba/0xd0
[ 260.527336] [<c103b83d>] native_flush_tlb_others+0xd/0x10
[ 260.527338] [<c103b8eb>] flush_tlb_mm+0x4b/0x90
[ 260.527341] [<c1113ecd>] tlb_flush_mmu+0x3d/0x80
[ 260.527344] [<c1113f21>] tlb_finish_mmu+0x11/0x40
[ 260.527346] [<c111a168>] unmap_region+0xc8/0xe0
[ 260.527350] [<c1119edd>] ? detach_vmas_to_be_unmapped+0x7d/0xc0
[ 260.527352] [<c111af9e>] do_munmap+0x1ae/0x200
[ 260.527355] [<c111c6a0>] sys_munmap+0x40/0x60
[ 260.527359] [<c15ab1df>] sysenter_do_call+0x12/0x28
[ 260.527361] ---[ end trace ce3cad64ec8db642 ]---
[ 260.628565] CPU 2 is now offline
[ 320.494503] INFO: rcu_sched detected stall on CPU 0 (t=15000 jiffies)
[ 320.500301] Pid: 274, comm: udevd Tainted: G W 3.2.0-2-generic-pae #5-Ubuntu
[ 320.500302] Call Trace:
[ 320.500307] [<c158e783>] ? printk+0x2d/0x2f
[ 320.500311] [<c10c4c33>] check_cpu_stall.isra.36+0x93/0xe0
[ 320.500313] [<c10c4caa>] __rcu_pending+0x2a/0x140
[ 320.500315] [<c10c505c>] rcu_check_callbacks+0x6c/0x1d0
[ 320.500318] [<c10699bb>] update_process_times+0x3b/0x70
[ 320.500321] [<c108ac2e>] tick_sched_timer+0x5e/0xc0
[ 320.500323] [<c107d46f>] __run_hrtimer+0x6f/0x1b0
[ 320.500326] [<c108abd0>] ? tick_nohz_handler+0x100/0x100
[ 320.500328] [<c107dde5>] hrtimer_interrupt+0xe5/0x260
[ 320.500331] [<c15ab9e4>] smp_apic_timer_interrupt+0x54/0x88
[ 320.500334] [<c15a4739>] apic_timer_interrupt+0x31/0x38
[ 320.500337] [<c111007b>] ? bdi_destroy+0xfb/0x130
[ 320.500339] [<c11100d8>] ? bdi_setup_and_register+0x28/0xc0
[ 320.500342] [<c103b6c2>] ? flush_tlb_others_ipi+0xc2/0xd0
[ 320.500344] [<c103b83d>] native_flush_tlb_others+0xd/0x10
[ 320.500346] [<c103b8eb>] flush_tlb_mm+0x4b/0x90
[ 320.500349] [<c1113ecd>] tlb_flush_mmu+0x3d/0x80
[ 320.500351] [<c1113f21>] tlb_finish_mmu+0x11/0x40
[ 320.500353] [<c111a168>] unmap_region+0xc8/0xe0
[ 320.500355] [<c1119edd>] ? detach_vmas_to_be_unmapped+0x7d/0xc0
[ 320.500358] [<c111af9e>] do_munmap+0x1ae/0x200
[ 320.500360] [<c111c6a0>] sys_munmap+0x40/0x60
[ 320.500362] [<c15ab1df>] sysenter_do_call+0x12/0x28

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-2-generic-pae 3.2.0-2.5
ProcVersionSignature: Ubuntu 3.2.0-2.5-generic-pae 3.2.0-rc3
Uname: Linux 3.2.0-2-generic-pae i686
AlsaDevices:
 total 0
 crw-rw---T 1 root audio 116, 1 Nov 30 06:59 seq
 crw-rw---T 1 root audio 116, 33 Nov 30 06:59 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 1.90-0ubuntu1
Architecture: i386
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
Date: Wed Nov 30 07:03:39 2011
HibernationDevice: RESUME=UUID=a6c1e5d7-e92f-462f-9258-736b3c584368
IwConfig: Error: [Errno 2] No such file or directory
MachineType: Dell Inc. PowerEdge R210 II
PciMultimedia:

ProcEnviron:
 LANGUAGE=en_US:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-2-generic-pae root=UUID=f45c8d50-694a-40c6-93c3-0c82d62e8069 ro quiet
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-2-generic-pae N/A
 linux-backports-modules-3.2.0-2-generic-pae N/A
 linux-firmware 1.62
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 05/06/2011
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 1.1.1
dmi.board.name: 09T7VV
dmi.board.vendor: Dell Inc.
dmi.board.version: A01
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr1.1.1:bd05/06/2011:svnDellInc.:pnPowerEdgeR210II:pvr:rvnDellInc.:rn09T7VV:rvrA01:cvnDellInc.:ct23:cvr:
dmi.product.name: PowerEdge R210 II
dmi.sys.vendor: Dell Inc.

[James Page (james-page)]

Running the following command to disable CPU control in powernap works around this issue:

chmod -x /etc/pm/power.d/01cpu_online

[Joseph Salisbury (jsalisbury)]

Jean-Baptiste,

Do you know if this only happens with the generic-pae kernel? Does it also happen with the server flavor?

Also, are you seeing this on any other hardware, or just the Dell PE/R210? Do you have other Dell servers you could test?

[Jean-Baptiste Lallement (jibel)]

I haven't tried another kernel but can redo the test with the server flavour. I only have PE R210 available for testing.

[Jean-Baptiste Lallement (jibel)]

Joseph, actually linux-image-server installs linux-image-generic-pae. Is there any other server kernel flavour ?

[Joseph Salisbury (jsalisbury)]

Jean-Baptiste,

Ahh, right, your are testing 32 bit, which does not have a server flavour. The full list of flavours is available at:

https://wiki.ubuntu.com/Kernel/Dev/Flavours

For 32 bit, the only other flavours are generic and virtual. Do you have the option to test generic? Also, do you plan on testing 64 bit flavours to see if the issue happens there as well?

[Brad Figg (brad-figg)]

Thank you for taking the time to file a bug report on this issue.

However, given the number of bugs that the Kernel Team receives during any development cycle it is impossible for us to review them all. Therefore, we occasionally resort to using automated bots to request further testing. This is such a request.

We have noted that there is a newer version of the development kernel than the one you last tested when this issue was found. Please test again with the newer kernel and indicate in the bug if this issue still exists or not.

If the bug still exists, change the bug status from Incomplete to Confirmed. If the bug no longer exists, change the bug status from Incomplete to Fix Released.

If you want this bot to quit automatically requesting kernel tests, add a tag named: bot-stop-nagging.

 Thank you for your help, we really do appreciate it.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/898127

[Brad Figg (brad-figg)]

@Jean-Baptiste,

Does this issue still exist with the current kernel?

[Jean-Baptiste Lallement (jibel)]

not reproduced with latest kernel. closing.