Triggered this again by running ohai in a continuous loop, took about 24 hours to occur:
[18438803.627371] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 [18438803.627388] IP: [<ffffffff8130d7f1>] rb_next+0x1/0x50 [18438803.627402] PGD 1d0efa067 PUD 1d232d067 PMD 0 [18438803.627411] Oops: 0000 [#1] SMP [18438803.627419] CPU 1 [18438803.627422] Modules linked in: ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables isofs acpiphp [18438803.627447] [18438803.627452] Pid: 29083, comm: ohai Not tainted 3.2.0-23-virtual #36-Ubuntu [18438803.627460] RIP: e030:[<ffffffff8130d7f1>] [<ffffffff8130d7f1>] rb_next+0x1/0x50 [18438803.627469] RSP: e02b:ffff8801d225d808 EFLAGS: 00010046 [18438803.627473] RAX: 0000000000000000 RBX: ffff8801d2232400 RCX: 0000000000000000 [18438803.627479] RDX: fffffffffffffff0 RSI: ffff8801dffa2760 RDI: 0000000000000010 [18438803.627485] RBP: ffff8801d225d838 R08: 0000000000000000 R09: 0000000000000000 [18438803.627490] R10: ffff8801dff866c0 R11: 0000000000000000 R12: 0000000000000000 [18438803.627497] R13: 0000000000000000 R14: 0000000000000280 R15: ffff8801d0992300 [18438803.627508] FS: 00007f34206c2700(0000) GS:ffff8801dff8f000(0000) knlGS:0000000000000000 [18438803.627515] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [18438803.627521] CR2: 0000000000000010 CR3: 00000001d0e9e000 CR4: 0000000000002660 [18438803.627527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18438803.627534] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [18438803.627541] Process ohai (pid: 29083, threadinfo ffff8801d225c000, task ffff8801d260adc0) [18438803.627547] Stack: [18438803.627551] ffff8801d225d838 ffffffff8104ece9 ffff8801d2232400 ffff8801dffa26c0 [18438803.627562] ffff8801d0f8fc00 0000000000000280 ffff8801d225d868 ffffffff810544b8 [18438803.627573] ffff8801d225d868 ffff8801dffa26c0 0000000000000001 ffff8801d260b168 [18438803.627584] Call Trace: [18438803.627596] [<ffffffff8104ece9>] ? pick_next_entity+0xb9/0xe0 [18438803.627604] [<ffffffff810544b8>] pick_next_task_fair+0x38/0x70 [18438803.627861] [<ffffffff81652ddc>] __schedule+0x14c/0x6f0 [18438803.627874] [<ffffffff8111d335>] ? prep_new_page+0x145/0x1e0 [18438803.627881] [<ffffffff8165344f>] schedule+0x3f/0x60 [18438803.627889] [<ffffffff8165454c>] schedule_hrtimeout_range_clock+0x12c/0x170 [18438803.627901] [<ffffffff8108c890>] ? update_rmtp+0x70/0x70 [18438803.627908] [<ffffffff8108d684>] ? hrtimer_start_range_ns+0x14/0x20 [18438803.627916] [<ffffffff816545a3>] schedule_hrtimeout_range+0x13/0x20 [18438803.627927] [<ffffffff811877a9>] poll_schedule_timeout+0x49/0x70 [18438803.627934] [<ffffffff81188326>] do_select+0x4d6/0x600 [18438803.627942] [<ffffffff811878b0>] ? poll_freewait+0xe0/0xe0 [18438803.627949] [<ffffffff811879a0>] ? __pollwait+0xf0/0xf0 [18438803.627956] [<ffffffff811879a0>] ? __pollwait+0xf0/0xf0 [18438803.627966] [<ffffffff8100a25d>] ? xen_force_evtchn_callback+0xd/0x10 [18438803.627974] [<ffffffff8100aa32>] ? check_events+0x12/0x20 [18438803.627981] [<ffffffff8100a25d>] ? xen_force_evtchn_callback+0xd/0x10 [18438803.627988] [<ffffffff8100aa32>] ? check_events+0x12/0x20 [18438803.627995] [<ffffffff8100aa1f>] ? xen_restore_fl_direct_reloc+0x4/0x4 [18438803.628003] [<ffffffff81006d1d>] ? xen_flush_tlb_single+0xbd/0x210 [18438803.628013] [<ffffffff81306dbd>] ? cpumask_any_but+0x2d/0x40 [18438803.628022] [<ffffffff81044b98>] ? flush_tlb_page+0x48/0xb0 [18438803.628030] [<ffffffff810438ac>] ? ptep_set_access_flags+0x6c/0x70 [18438803.628038] [<ffffffff81138c52>] ? do_wp_page+0x382/0x740 [18438803.628045] [<ffffffff81006739>] ? pte_mfn_to_pfn+0x89/0xf0 [18438803.628053] [<ffffffff81005209>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e [18438803.628061] [<ffffffff81188611>] core_sys_select+0x1c1/0x330 [18438803.628069] [<ffffffff8113af98>] ? handle_mm_fault+0x1f8/0x350 [18438803.628076] [<ffffffff8103cc65>] ? pvclock_clocksource_read+0x55/0xf0 [18438803.628085] [<ffffffff8100a540>] ? xen_clocksource_read+0x20/0x30 [18438803.628092] [<ffffffff8100a629>] ? xen_clocksource_get_cycles+0x9/0x10 [18438803.628101] [<ffffffff810933ed>] ? ktime_get_ts+0xad/0xe0 [18438803.628108] [<ffffffff811889bb>] sys_select+0xbb/0x100 [18438803.628117] [<ffffffff8165d8c2>] system_call_fastpath+0x16/0x1b [18438803.628123] Code: 89 06 48 8b 47 08 48 89 46 08 48 8b 47 10 48 89 46 10 c3 0f 1f 80 00 00 00 00 48 89 32 eb b2 0f 1f 00 48 89 70 10 eb a9 66 90 55 <48> 8b 17 48 89 e5 48 89 d0 48 83 e0 fc 48 39 c7 74 34 48 8b 47 [18438803.628207] RIP [<ffffffff8130d7f1>] rb_next+0x1/0x50 [18438803.628215] RSP <ffff8801d225d808> [18438803.628219] CR2: 0000000000000010 [18438803.628229] ---[ end trace 6e3e751b67665edf ]---
Triggered this again by running ohai in a continuous loop, took about 24 hours to occur:
[18438803.627371] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 7f1>] rb_next+0x1/0x50 ffffffff8130d7f 1>] [<ffffffff8130d 7f1>] rb_next+0x1/0x50 25d808 EFLAGS: 00010046 0(0000) GS:ffff8801dff8 f000(0000) knlGS:000000000 0000000 ce9>] ? pick_next_ entity+ 0xb9/0xe0 4b8>] pick_next_ task_fair+ 0x38/0x70 ddc>] __schedule+ 0x14c/0x6f0 335>] ? prep_new_ page+0x145/ 0x1e0 44f>] schedule+0x3f/0x60 54c>] schedule_ hrtimeout_ range_clock+ 0x12c/0x170 890>] ? update_ rmtp+0x70/ 0x70 684>] ? hrtimer_ start_range_ ns+0x14/ 0x20 5a3>] schedule_ hrtimeout_ range+0x13/ 0x20 7a9>] poll_schedule_ timeout+ 0x49/0x70 326>] do_select+ 0x4d6/0x600 8b0>] ? poll_freewait+ 0xe0/0xe0 9a0>] ? __pollwait+ 0xf0/0xf0 9a0>] ? __pollwait+ 0xf0/0xf0 25d>] ? xen_force_ evtchn_ callback+ 0xd/0x10 a32>] ? check_events+ 0x12/0x20 25d>] ? xen_force_ evtchn_ callback+ 0xd/0x10 a32>] ? check_events+ 0x12/0x20 a1f>] ? xen_restore_ fl_direct_ reloc+0x4/ 0x4 d1d>] ? xen_flush_ tlb_single+ 0xbd/0x210 dbd>] ? cpumask_ any_but+ 0x2d/0x40 b98>] ? flush_tlb_ page+0x48/ 0xb0 8ac>] ? ptep_set_ access_ flags+0x6c/ 0x70 c52>] ? do_wp_page+ 0x382/0x740 739>] ? pte_mfn_ to_pfn+ 0x89/0xf0 209>] ? __raw_callee_ save_xen_ pmd_val+ 0x11/0x1e 611>] core_sys_ select+ 0x1c1/0x330 f98>] ? handle_ mm_fault+ 0x1f8/0x350 c65>] ? pvclock_ clocksource_ read+0x55/ 0xf0 540>] ? xen_clocksource _read+0x20/ 0x30 629>] ? xen_clocksource _get_cycles+ 0x9/0x10 3ed>] ? ktime_get_ ts+0xad/ 0xe0 9bb>] sys_select+ 0xbb/0x100 8c2>] system_ call_fastpath+ 0x16/0x1b 7f1>] rb_next+0x1/0x50
[18438803.627388] IP: [<ffffffff8130d
[18438803.627402] PGD 1d0efa067 PUD 1d232d067 PMD 0
[18438803.627411] Oops: 0000 [#1] SMP
[18438803.627419] CPU 1
[18438803.627422] Modules linked in: ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables isofs acpiphp
[18438803.627447]
[18438803.627452] Pid: 29083, comm: ohai Not tainted 3.2.0-23-virtual #36-Ubuntu
[18438803.627460] RIP: e030:[<
[18438803.627469] RSP: e02b:ffff8801d2
[18438803.627473] RAX: 0000000000000000 RBX: ffff8801d2232400 RCX: 0000000000000000
[18438803.627479] RDX: fffffffffffffff0 RSI: ffff8801dffa2760 RDI: 0000000000000010
[18438803.627485] RBP: ffff8801d225d838 R08: 0000000000000000 R09: 0000000000000000
[18438803.627490] R10: ffff8801dff866c0 R11: 0000000000000000 R12: 0000000000000000
[18438803.627497] R13: 0000000000000000 R14: 0000000000000280 R15: ffff8801d0992300
[18438803.627508] FS: 00007f34206c270
[18438803.627515] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[18438803.627521] CR2: 0000000000000010 CR3: 00000001d0e9e000 CR4: 0000000000002660
[18438803.627527] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[18438803.627534] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[18438803.627541] Process ohai (pid: 29083, threadinfo ffff8801d225c000, task ffff8801d260adc0)
[18438803.627547] Stack:
[18438803.627551] ffff8801d225d838 ffffffff8104ece9 ffff8801d2232400 ffff8801dffa26c0
[18438803.627562] ffff8801d0f8fc00 0000000000000280 ffff8801d225d868 ffffffff810544b8
[18438803.627573] ffff8801d225d868 ffff8801dffa26c0 0000000000000001 ffff8801d260b168
[18438803.627584] Call Trace:
[18438803.627596] [<ffffffff8104e
[18438803.627604] [<ffffffff81054
[18438803.627861] [<ffffffff81652
[18438803.627874] [<ffffffff8111d
[18438803.627881] [<ffffffff81653
[18438803.627889] [<ffffffff81654
[18438803.627901] [<ffffffff8108c
[18438803.627908] [<ffffffff8108d
[18438803.627916] [<ffffffff81654
[18438803.627927] [<ffffffff81187
[18438803.627934] [<ffffffff81188
[18438803.627942] [<ffffffff81187
[18438803.627949] [<ffffffff81187
[18438803.627956] [<ffffffff81187
[18438803.627966] [<ffffffff8100a
[18438803.627974] [<ffffffff8100a
[18438803.627981] [<ffffffff8100a
[18438803.627988] [<ffffffff8100a
[18438803.627995] [<ffffffff8100a
[18438803.628003] [<ffffffff81006
[18438803.628013] [<ffffffff81306
[18438803.628022] [<ffffffff81044
[18438803.628030] [<ffffffff81043
[18438803.628038] [<ffffffff81138
[18438803.628045] [<ffffffff81006
[18438803.628053] [<ffffffff81005
[18438803.628061] [<ffffffff81188
[18438803.628069] [<ffffffff8113a
[18438803.628076] [<ffffffff8103c
[18438803.628085] [<ffffffff8100a
[18438803.628092] [<ffffffff8100a
[18438803.628101] [<ffffffff81093
[18438803.628108] [<ffffffff81188
[18438803.628117] [<ffffffff8165d
[18438803.628123] Code: 89 06 48 8b 47 08 48 89 46 08 48 8b 47 10 48 89 46 10 c3 0f 1f 80 00 00 00 00 48 89 32 eb b2 0f 1f 00 48 89 70 10 eb a9 66 90 55 <48> 8b 17 48 89 e5 48 89 d0 48 83 e0 fc 48 39 c7 74 34 48 8b 47
[18438803.628207] RIP [<ffffffff8130d
[18438803.628215] RSP <ffff8801d225d808>
[18438803.628219] CR2: 0000000000000010
[18438803.628229] ---[ end trace 6e3e751b67665edf ]---