Comment 9 for bug 999755

Revision history for this message
Gavin Heavyside (mydrive) wrote : Re: Kernel crash on EC2 m1.large instances

Triggered this again by running ohai in a continuous loop, took about 24 hours to occur:

[18438803.627371] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
[18438803.627388] IP: [<ffffffff8130d7f1>] rb_next+0x1/0x50
[18438803.627402] PGD 1d0efa067 PUD 1d232d067 PMD 0
[18438803.627411] Oops: 0000 [#1] SMP
[18438803.627419] CPU 1
[18438803.627422] Modules linked in: ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables isofs acpiphp
[18438803.627447]
[18438803.627452] Pid: 29083, comm: ohai Not tainted 3.2.0-23-virtual #36-Ubuntu
[18438803.627460] RIP: e030:[<ffffffff8130d7f1>] [<ffffffff8130d7f1>] rb_next+0x1/0x50
[18438803.627469] RSP: e02b:ffff8801d225d808 EFLAGS: 00010046
[18438803.627473] RAX: 0000000000000000 RBX: ffff8801d2232400 RCX: 0000000000000000
[18438803.627479] RDX: fffffffffffffff0 RSI: ffff8801dffa2760 RDI: 0000000000000010
[18438803.627485] RBP: ffff8801d225d838 R08: 0000000000000000 R09: 0000000000000000
[18438803.627490] R10: ffff8801dff866c0 R11: 0000000000000000 R12: 0000000000000000
[18438803.627497] R13: 0000000000000000 R14: 0000000000000280 R15: ffff8801d0992300
[18438803.627508] FS: 00007f34206c2700(0000) GS:ffff8801dff8f000(0000) knlGS:0000000000000000
[18438803.627515] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[18438803.627521] CR2: 0000000000000010 CR3: 00000001d0e9e000 CR4: 0000000000002660
[18438803.627527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[18438803.627534] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[18438803.627541] Process ohai (pid: 29083, threadinfo ffff8801d225c000, task ffff8801d260adc0)
[18438803.627547] Stack:
[18438803.627551] ffff8801d225d838 ffffffff8104ece9 ffff8801d2232400 ffff8801dffa26c0
[18438803.627562] ffff8801d0f8fc00 0000000000000280 ffff8801d225d868 ffffffff810544b8
[18438803.627573] ffff8801d225d868 ffff8801dffa26c0 0000000000000001 ffff8801d260b168
[18438803.627584] Call Trace:
[18438803.627596] [<ffffffff8104ece9>] ? pick_next_entity+0xb9/0xe0
[18438803.627604] [<ffffffff810544b8>] pick_next_task_fair+0x38/0x70
[18438803.627861] [<ffffffff81652ddc>] __schedule+0x14c/0x6f0
[18438803.627874] [<ffffffff8111d335>] ? prep_new_page+0x145/0x1e0
[18438803.627881] [<ffffffff8165344f>] schedule+0x3f/0x60
[18438803.627889] [<ffffffff8165454c>] schedule_hrtimeout_range_clock+0x12c/0x170
[18438803.627901] [<ffffffff8108c890>] ? update_rmtp+0x70/0x70
[18438803.627908] [<ffffffff8108d684>] ? hrtimer_start_range_ns+0x14/0x20
[18438803.627916] [<ffffffff816545a3>] schedule_hrtimeout_range+0x13/0x20
[18438803.627927] [<ffffffff811877a9>] poll_schedule_timeout+0x49/0x70
[18438803.627934] [<ffffffff81188326>] do_select+0x4d6/0x600
[18438803.627942] [<ffffffff811878b0>] ? poll_freewait+0xe0/0xe0
[18438803.627949] [<ffffffff811879a0>] ? __pollwait+0xf0/0xf0
[18438803.627956] [<ffffffff811879a0>] ? __pollwait+0xf0/0xf0
[18438803.627966] [<ffffffff8100a25d>] ? xen_force_evtchn_callback+0xd/0x10
[18438803.627974] [<ffffffff8100aa32>] ? check_events+0x12/0x20
[18438803.627981] [<ffffffff8100a25d>] ? xen_force_evtchn_callback+0xd/0x10
[18438803.627988] [<ffffffff8100aa32>] ? check_events+0x12/0x20
[18438803.627995] [<ffffffff8100aa1f>] ? xen_restore_fl_direct_reloc+0x4/0x4
[18438803.628003] [<ffffffff81006d1d>] ? xen_flush_tlb_single+0xbd/0x210
[18438803.628013] [<ffffffff81306dbd>] ? cpumask_any_but+0x2d/0x40
[18438803.628022] [<ffffffff81044b98>] ? flush_tlb_page+0x48/0xb0
[18438803.628030] [<ffffffff810438ac>] ? ptep_set_access_flags+0x6c/0x70
[18438803.628038] [<ffffffff81138c52>] ? do_wp_page+0x382/0x740
[18438803.628045] [<ffffffff81006739>] ? pte_mfn_to_pfn+0x89/0xf0
[18438803.628053] [<ffffffff81005209>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
[18438803.628061] [<ffffffff81188611>] core_sys_select+0x1c1/0x330
[18438803.628069] [<ffffffff8113af98>] ? handle_mm_fault+0x1f8/0x350
[18438803.628076] [<ffffffff8103cc65>] ? pvclock_clocksource_read+0x55/0xf0
[18438803.628085] [<ffffffff8100a540>] ? xen_clocksource_read+0x20/0x30
[18438803.628092] [<ffffffff8100a629>] ? xen_clocksource_get_cycles+0x9/0x10
[18438803.628101] [<ffffffff810933ed>] ? ktime_get_ts+0xad/0xe0
[18438803.628108] [<ffffffff811889bb>] sys_select+0xbb/0x100
[18438803.628117] [<ffffffff8165d8c2>] system_call_fastpath+0x16/0x1b
[18438803.628123] Code: 89 06 48 8b 47 08 48 89 46 08 48 8b 47 10 48 89 46 10 c3 0f 1f 80 00 00 00 00 48 89 32 eb b2 0f 1f 00 48 89 70 10 eb a9 66 90 55 <48> 8b 17 48 89 e5 48 89 d0 48 83 e0 fc 48 39 c7 74 34 48 8b 47
[18438803.628207] RIP [<ffffffff8130d7f1>] rb_next+0x1/0x50
[18438803.628215] RSP <ffff8801d225d808>
[18438803.628219] CR2: 0000000000000010
[18438803.628229] ---[ end trace 6e3e751b67665edf ]---