BUG: unable to handle kernel NULL pointer dereference at (null)

[Eugene Nelen]

We are using HP DL180G6 servers.
Time to time servers hangs with next kernel dump.

[ 5208.502006] BUG: unable to handle kernel NULL pointer dereference at (null)
[ 5208.510756] IP: [< (null)>] (null)
[ 5208.516391] PGD 0
[ 5208.518640] Oops: 0010 [#1] SMP
[ 5208.522253] CPU 0
[ 5208.524297] Modules linked in: xt_multiport xt_tcpudp xt_conntrack ip6table_filter ip6_tables ebtable_nat ebtables iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrac
k nf_defrag_ipv4 ipmi_devintf iptable_filter ip_tables x_tables nbd ipmi_si ipmi_msghandler kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr isc
si_tcp libiscsi_tcp libiscsi scsi_transport_iscsi xfs 8021q serio_raw i7core_edac dm_multipath garp stp edac_core mac_hid lp parport raid10 raid456 async_pq async_xor
 xor async_memcpy async_raid6_recov igb hpsa dca raid6_pq async_tx raid1 raid0 multipath linear
[ 5208.582735]
[ 5208.584394] Pid: 0, comm: swapper/0 Not tainted 3.2.0-24-generic #37-Ubuntu HP ProLiant DL180 G6
[ 5208.594400] RIP: 0010:[<0000000000000000>] [< (null)>] (null)
[ 5208.602747] RSP: 0018:ffff88013bc03d90 EFLAGS: 00010046
[ 5208.608664] RAX: 0000000000000000 RBX: ffff8802376d3800 RCX: 0000000000000000
[ 5208.616614] RDX: 0000000000000000 RSI: ffff880036c20160 RDI: ffff8802376d3800
[ 5208.624566] RBP: ffff88013bc03e68 R08: ffff880036c14000 R09: ffff88013b678268
[ 5208.632517] R10: 0000000000001a19 R11: 0000000000000001 R12: ffff880036c20150
[ 5208.640468] R13: ffff880036d01180 R14: ffff8801340f0000 R15: ffff880036d01180
[ 5208.648417] FS: 0000000000000000(0000) GS:ffff88013bc00000(0000) knlGS:0000000000000000
[ 5208.657432] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 5208.663832] CR2: 0000000000000000 CR3: 0000000001c05000 CR4: 00000000000006f0
[ 5208.671783] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 5208.679735] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 5208.687684] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c0d020)
[ 5208.696893] Stack:
[ 5208.699132] ffffffffa0059689 0000000000000075 ffff88013bc03e20 ffffffffa007021e
[ 5208.707416] ffff88013bc03dc0 0000000000000075 ffff880138a33750 000000003401a900
[ 5208.715700] ffff88013401a940 ffff88013401a900 0000003c00000001 0001ffff0000003f
[ 5208.723981] Call Trace:
[ 5208.726703] <IRQ>
[ 5208.729054] [<ffffffffa0059689>] ? complete_scsi_command+0x139/0x580 [hpsa]
[ 5208.736905] [<ffffffffa007021e>] ? igb_clean_rx_irq+0x46e/0x4e0 [igb]
[ 5208.744182] [<ffffffffa006fc7b>] ? igb_ring_irq_enable+0x8b/0xb0 [igb]
[ 5208.751555] [<ffffffffa005a9cc>] do_hpsa_intr_msi+0xcc/0x2d0 [hpsa]
[ 5208.758639] [<ffffffff810d8e15>] handle_irq_event_percpu+0x55/0x220
[ 5208.765718] [<ffffffff810d9031>] handle_irq_event+0x51/0x80
[ 5208.772026] [<ffffffff810dc187>] handle_edge_irq+0x87/0x140
[ 5208.778333] [<ffffffff81015282>] handle_irq+0x22/0x40
[ 5208.784058] [<ffffffff8166759a>] do_IRQ+0x5a/0xe0
[ 5208.789398] [<ffffffff8165c92e>] common_interrupt+0x6e/0x6e
[ 5208.795701] <EOI>
[ 5208.798048] [<ffffffff8136b42d>] ? intel_idle+0xed/0x150
[ 5208.804064] [<ffffffff8136b40f>] ? intel_idle+0xcf/0x150
[ 5208.810079] [<ffffffff81504a01>] cpuidle_idle_call+0xc1/0x280
[ 5208.816578] [<ffffffff8101222a>] cpu_idle+0xca/0x120
[ 5208.822208] [<ffffffff816235ae>] rest_init+0x72/0x74
[ 5208.827838] [<ffffffff81cfbc0d>] start_kernel+0x3ba/0x3c7
[ 5208.833943] [<ffffffff81cfb388>] x86_64_start_reservations+0x132/0x136
[ 5208.841314] [<ffffffff81cfb140>] ? early_idt_handlers+0x140/0x140
[ 5208.848200] [<ffffffff81cfb459>] x86_64_start_kernel+0xcd/0xdc
[ 5208.854794] Code: Bad RIP value.
[ 5208.858508] RIP [< (null)>] (null)
[ 5208.864241] RSP <ffff88013bc03d90>
[ 5208.868124] CR2: 0000000000000000
[ 5208.872162] ---[ end trace c03593270af25d03 ]---
[ 5208.877304] Kernel panic - not syncing: Fatal exception in interrupt
[ 5208.884384] Pid: 0, comm: swapper/0 Tainted: G D 3.2.0-24-generic #37-Ubuntu
[ 5208.893205] Call Trace:
[ 5208.895926] <IRQ> [<ffffffff81644023>] panic+0x91/0x1a4
[ 5208.901955] [<ffffffff8165d65a>] oops_end+0xea/0xf0
[ 5208.907486] [<ffffffff81642ebd>] no_context+0x150/0x15d
[ 5208.913405] [<ffffffff81643093>] __bad_area_nosemaphore+0x1c9/0x1e8
[ 5208.920487] [<ffffffff81531a82>] ? kfree_skbmem+0x32/0x90
[ 5208.926599] [<ffffffff816430c5>] bad_area_nosemaphore+0x13/0x15
[ 5208.933292] [<ffffffff81660276>] do_page_fault+0x426/0x520
[ 5208.939504] [<ffffffff8105a471>] ? find_busiest_group+0x171/0xbb0
[ 5208.946392] [<ffffffff8116578a>] ? __kmalloc_node_track_caller+0x3a/0x1e0
[ 5208.954053] [<ffffffff81531dab>] ? __alloc_skb+0x4b/0x240
[ 5208.960168] [<ffffffff81532424>] ? __netdev_alloc_skb+0x24/0x50
[ 5208.966852] [<ffffffff81161abc>] ? ksize+0x1c/0xc0
[ 5208.972286] [<ffffffff81531dec>] ? __alloc_skb+0x8c/0x240
[ 5208.978399] [<ffffffff8165cbf5>] page_fault+0x25/0x30
[ 5208.984127] [<ffffffffa0059689>] ? complete_scsi_command+0x139/0x580 [hpsa]
[ 5208.991985] [<ffffffffa007021e>] ? igb_clean_rx_irq+0x46e/0x4e0 [igb]
[ 5208.999261] [<ffffffffa006fc7b>] ? igb_ring_irq_enable+0x8b/0xb0 [igb]
[ 5209.006635] [<ffffffffa005a9cc>] do_hpsa_intr_msi+0xcc/0x2d0 [hpsa]
[ 5209.013716] [<ffffffff810d8e15>] handle_irq_event_percpu+0x55/0x220
[ 5209.020796] [<ffffffff810d9031>] handle_irq_event+0x51/0x80
[ 5209.027101] [<ffffffff810dc187>] handle_edge_irq+0x87/0x140
[ 5209.033406] [<ffffffff81015282>] handle_irq+0x22/0x40
[ 5209.039130] [<ffffffff8166759a>] do_IRQ+0x5a/0xe0
[ 5209.044467] [<ffffffff8165c92e>] common_interrupt+0x6e/0x6e
[ 5209.050771] <EOI> [<ffffffff8136b42d>] ? intel_idle+0xed/0x150
[ 5209.057476] [<ffffffff8136b40f>] ? intel_idle+0xcf/0x150
[ 5209.063491] [<ffffffff81504a01>] cpuidle_idle_call+0xc1/0x280
[ 5209.069990] [<ffffffff8101222a>] cpu_idle+0xca/0x120
[ 5209.075618] [<ffffffff816235ae>] rest_init+0x72/0x74
[ 5209.081246] [<ffffffff81cfbc0d>] start_kernel+0x3ba/0x3c7
[ 5209.087358] [<ffffffff81cfb388>] x86_64_start_reservations+0x132/0x136
[ 5209.094730] [<ffffffff81cfb140>] ? early_idt_handlers+0x140/0x140
[ 5209.101618] [<ffffffff81cfb459>] x86_64_start_kernel+0xcd/0xdc

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-24-generic 3.2.0-24.37
ProcVersionSignature: Ubuntu 3.2.0-24.37-generic 3.2.14
Uname: Linux 3.2.0-24-generic x86_64
AlsaDevices:
 total 0
 crw-rw---T 1 root audio 116, 1 Sep 21 15:03 seq
 crw-rw---T 1 root audio 116, 33 Sep 21 15:03 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.0.1-0ubuntu7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
Date: Fri Sep 21 15:06:48 2012
HibernationDevice: RESUME=UUID=95dc726b-6d57-41cb-ab95-5f561b0a2f86
IwConfig: Error: [Errno 2] No such file or directory
MachineType: HP ProLiant DL180 G6
PciMultimedia:

ProcEnviron:
 LANGUAGE=en_US:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.2.0-24-generic root=/dev/mapper/vg-root ro quiet console=ttyS0,115200n8
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-24-generic N/A
 linux-backports-modules-3.2.0-24-generic N/A
 linux-firmware 1.79
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 02/01/2012
dmi.bios.vendor: HP
dmi.bios.version: O20
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrO20:bd02/01/2012:svnHP:pnProLiantDL180G6:pvr:cvnHP:ct23:cvr:
dmi.product.name: ProLiant DL180 G6
dmi.sys.vendor: HP

[Eugene Nelen (enelen)]

Also in kernel log we see next messages just before hang.

Sep 21 14:51:55 hc0606 kernel: [ 5153.280432] hpsa 0000:06:00.0: resetting device 6:0:0:6
Sep 21 14:51:55 hc0606 kernel: [ 5154.282842] hpsa 0000:06:00.0: device is ready.
Sep 21 14:52:36 hc0606 kernel: [ 5195.240606] hpsa 0000:06:00.0: resetting device 6:0:0:6
Sep 21 14:52:37 hc0606 kernel: [ 5196.243124] hpsa 0000:06:00.0: device is ready.

So the bug can be in hpsa module.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.6 kernel[0] (Not a kernel in the daily directory) and install both the linux-image and linux-image-extra .deb packages.

Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. Please only remove that one tag and leave the other tags. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.6-rc6-quantal/

[Eugene Nelen (enelen)]

We have tested this bug with v3.6-rc6-quantal kernel.
This is kernel dump
[264194.437473] BUG: unable to handle kernel NULL pointer dereference at (null)
[264194.446311] IP: [< (null)>] (null)
[264194.452047] PGD 0
[264194.454392] Oops: 0010 [#1] SMP
[264194.458102] Modules linked in: xt_multiport xt_tcpudp xt_conntrack ip6table_filter ip6_tables ebtable_nat ebtables ipmi_devintf ipmi_si ipmi_msghandler iptable_na
t nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 nbd iptable_filter ip_tables x_tables ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libisc
si_tcp libiscsi scsi_transport_iscsi coretemp kvm_intel dm_multipath scsi_dh kvm 8021q i7core_edac garp edac_core serio_raw ghash_clmulni_intel aesni_intel ablk_helpe
r cryptd gpio_ich aes_x86_64 stp lpc_ich microcode llc mac_hid xfs lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov hpsa igb dca raid6_
pq async_tx raid1 raid0 multipath linear
[264194.527204] CPU 0
[264194.529348] Pid: 0, comm: swapper/0 Not tainted 3.6.0-030600rc6-generic #201209161835 HP ProLiant DL180 G6
[264194.540601] RIP: 0010:[<0000000000000000>] [< (null)>] (null)
[264194.549047] RSP: 0018:ffff88013bc03e00 EFLAGS: 00010046
[264194.555061] RAX: 0000000000000000 RBX: ffff88001e735400 RCX: 0000000000000000
[264194.563107] RDX: 0000000000000000 RSI: ffff880036c20070 RDI: ffff88001e735400
[264194.571154] RBP: ffff88013bc03e58 R08: ffff88023857b600 R09: ffff88023aa81000
[264194.579199] R10: 00000000000010b9 R11: 0000000000000000 R12: ffff880036c20060
[264194.587246] R13: ffff880036d00500 R14: ffff8802381d0000 R15: 0000000000000051
[264194.595293] FS: 0000000000000000(0000) GS:ffff88013bc00000(0000) knlGS:0000000000000000
[264194.604404] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[264194.610901] CR2: 0000000000000000 CR3: 0000000001c0c000 CR4: 00000000000007f0
[264194.618947] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[264194.626995] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[264194.635042] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c14440)
[264194.644348] Stack:
[264194.646683] ffffffffa0080549 ffff88013bc03e28 ffff88013bc03e28 0000000000000086
[264194.655065] ffff8802381d0000 ffff88013bc03e38 ffff8802381d0000 0000000000004280
[264194.663442] 0000000000000000 ffff880036d00500 0000000000000051 ffff88013bc03ea8
[264194.671822] Call Trace:
[264194.674639] <IRQ>
[264194.676880] [<ffffffffa0080549>] ? complete_scsi_command+0x139/0x4c0 [hpsa]
[264194.685055] [<ffffffffa0082aae>] do_hpsa_intr_msi+0x12e/0x270 [hpsa]
[264194.692334] [<ffffffff81069785>] ? get_next_timer_interrupt+0xc5/0x140
[264194.699804] [<ffffffff810e7195>] handle_irq_event_percpu+0x55/0x210
[264194.706982] [<ffffffff810e739e>] handle_irq_event+0x4e/0x80
[264194.713385] [<ffffffff810e9ce4>] handle_edge_irq+0x84/0x130
[264194.719790] [<ffffffff81016632>] handle_irq+0x22/0x40
[264194.725612] [<ffffffff816a8e4a>] do_IRQ+0x5a/0xe0
[264194.731049] [<ffffffff8169ef2a>] common_interrupt+0x6a/0x6a
[264194.737450] <EOI>
[264194.739687] [<ffffffff81044c39>] ? default_s...

[Joseph Salisbury (jsalisbury)]

This issue appears to be an upstream bug, since you tested the latest upstream kernel. Would it be possible for you to open an upstream bug report[0]? That will allow the upstream Developers to examine the issue, and may provide a quicker resolution to the bug.

If you are comfortable with opening a bug upstream, It would be great if you can report back the upstream bug number in this bug report. That will allow us to link this bug to the upstream report.

[0] https://wiki.ubuntu.com/Bugs/Upstream/kernel

[Eugene Nelen (enelen)]

[1.] One line summary of the problem:
BUG: unable to handle kernel NULL pointer dereference at (null)
[2.] Full description of the problem/report:

We are using HP DL180G6 servers.
Time to time servers hangs with next kernel dump.
[3.] Keywords (i.e., modules, networking, kernel):
hpsa,kernel

[4.] Kernel version (from /proc/version):

Linux version 3.6.0-030600rc6-generic (apw@gomeisa) (gcc version 4.6.3
(Ubuntu/Linaro 4.6.3-1ubuntu5) ) #201209161835 SMP Sun Sep 16 22:36:28 UTC
2012
[5.] Output of Oops.. message (if applicable) with symbolic information
resolved (see Documentation/oops-tracing.txt)

[ 5208.502006] BUG: unable to handle kernel NULL pointer dereference at
(null)
[ 5208.510756] IP: [< (null)>] (null)
[ 5208.516391] PGD 0
[ 5208.518640] Oops: 0010 [#1] SMP
[ 5208.522253] CPU 0
[ 5208.524297] Modules linked in: xt_multiport xt_tcpudp xt_conntrack
ip6table_filter ip6_tables ebtable_nat ebtables iptable_nat nf_nat
nf_conntrack_ipv4 nf_conntrac
k nf_defrag_ipv4 ipmi_devintf iptable_filter ip_tables x_tables nbd ipmi_si
ipmi_msghandler kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad
ib_core ib_addr isc
si_tcp libiscsi_tcp libiscsi scsi_transport_iscsi xfs 8021q serio_raw
i7core_edac dm_multipath garp stp edac_core mac_hid lp parport raid10
raid456 async_pq async_xor
 xor async_memcpy async_raid6_recov igb hpsa dca raid6_pq async_tx raid1
raid0 multipath linear
[ 5208.582735]
[ 5208.584394] Pid: 0, comm: swapper/0 Not tainted 3.2.0-24-generic
#37-Ubuntu HP ProLiant DL180 G6
[ 5208.594400] RIP: 0010:[<0000000000000000>] [< (null)>] (null)
[ 5208.602747] RSP: 0018:ffff88013bc03d90 EFLAGS: 00010046
[ 5208.608664] RAX: 0000000000000000 RBX: ffff8802376d3800 RCX:
0000000000000000
[ 5208.616614] RDX: 0000000000000000 RSI: ffff880036c20160 RDI:
ffff8802376d3800
[ 5208.624566] RBP: ffff88013bc03e68 R08: ffff880036c14000 R09:
ffff88013b678268
[ 5208.632517] R10: 0000000000001a19 R11: 0000000000000001 R12:
ffff880036c20150
[ 5208.640468] R13: ffff880036d01180 R14: ffff8801340f0000 R15:
ffff880036d01180
[ 5208.648417] FS: 0000000000000000(0000) GS:ffff88013bc00000(0000)
knlGS:0000000000000000
[ 5208.657432] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 5208.663832] CR2: 0000000000000000 CR3: 0000000001c05000 CR4:
00000000000006f0
[ 5208.671783] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 5208.679735] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[ 5208.687684] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task
ffffffff81c0d020)
[ 5208.696893] Stack:
[ 5208.699132] ffffffffa0059689 0000000000000075 ffff88013bc03e20
ffffffffa007021e
[ 5208.707416] ffff88013bc03dc0 0000000000000075 ffff880138a33750
000000003401a900
[ 5208.715700] ffff88013401a940 ffff88013401a900 0000003c00000001
0001ffff0000003f
[ 5208.723981] Call Trace:
[ 5208.726703] <IRQ>
[ 5208.729054] [<ffffffffa0059689>] ? complete_scsi_command+0x139/0x580
[hpsa]
[ 5208.736905] [<ffffffffa007021e>] ? igb_clean_rx_irq+0x46e/0x4e0 [igb]
[ 5208.744182] [<ffffffffa006fc7b>] ? igb_ring_irq_enable+0x8b/0xb0 [igb]
[ 5208.751555] [<ffffffffa005a9cc>] do_hpsa_intr_msi+0xcc/0x2d0 [hpsa]
[ 5208.758639] [<ffffffff810d8e15>] handle_irq_ev...

[Gaspar Fernández (gasparfm)]

I had the same problem, but after upgrading to kernel 3.8.3 everything runs fine.

[penalvch (penalvch)]

Eugene Nelen, as per http://h20565.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdHome/?sp4ts.oid=3884340&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3DswEnvOID%253D4049%257CswLang%253D%257Caction%253DlistDriver&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken#BIOS%20-%20System%20ROM an update is available for your BIOS (2013.07.01). If you update to this during your maintenance window, does it change anything?

If not, could you please both specify what happened, and provide the output of the following terminal command:
sudo dmidecode -s bios-version && sudo dmidecode -s bios-release-date

Thank you for your understanding.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]