And another one from the debug kernel on EC2, with a slightly different call stack:
[ 4389.480352] ------------[ cut here ]------------ [ 4389.480884] kernel BUG at /home/smb/precise-amd64/ubuntu-2.6/kernel/sched_fair.c:1239! [ 4389.480894] invalid opcode: 0000 [#1] SMP [ 4389.480902] CPU 0 [ 4389.480905] Modules linked in: ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables bnep rfcomm bluetooth parport_pc ppdev lp parport isofs acpiphp [ 4389.480945] [ 4389.480949] Pid: 24612, comm: ohai Not tainted 3.2.0-24-virtual #37+lp999755dbg1 [ 4389.480958] RIP: e030:[<ffffffff81050cd5>] [<ffffffff81050cd5>] pick_next_entity+0x105/0x110 [ 4389.480972] RSP: e02b:ffff8801d0cdd818 EFLAGS: 00010092 [ 4389.480977] RAX: 000000000000002c RBX: ffff8801d1106e00 RCX: 000000000003ffff [ 4389.480983] RDX: 0000000000000000 RSI: ffffffff81e000a0 RDI: 0000000000000004 [ 4389.480988] RBP: ffff8801d0cdd838 R08: 000000000000000a R09: 0000000000000000 [ 4389.480994] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801dff866c0 [ 4389.481000] R13: ffff8801d1107a00 R14: 0000000000000008 R15: ffff8801d1e97800 [ 4389.481009] FS: 00007f09fcc8a700(0000) GS:ffff8801dff73000(0000) knlGS:0000000000000000 [ 4389.481015] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [ 4389.481021] CR2: 00000000012dbf30 CR3: 00000001d0702000 CR4: 0000000000002660 [ 4389.481028] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4389.481035] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 4389.481041] Process ohai (pid: 24612, threadinfo ffff8801d0cdc000, task ffff8801d2905b80) [ 4389.481047] Stack: [ 4389.481051] ffff8801d1106e00 ffff8801dff866c0 ffff8801d1107a00 0000000000000008 [ 4389.481063] ffff8801d0cdd868 ffffffff810544e8 ffff8801d0cdd868 ffff8801dff866c0 [ 4389.481075] 0000000000000000 ffff8801d2905f28 ffff8801d0cdd8e8 ffffffff81652f3c [ 4389.481086] Call Trace: [ 4389.481093] [<ffffffff810544e8>] pick_next_task_fair+0x38/0x70 [ 4389.481101] [<ffffffff81652f3c>] __schedule+0x14c/0x6f0 [ 4389.481108] [<ffffffff816535af>] schedule+0x3f/0x60 [ 4389.481114] [<ffffffff816546cd>] schedule_hrtimeout_range_clock+0x14d/0x170 [ 4389.481124] [<ffffffff8100aa1f>] ? xen_restore_fl_direct_reloc+0x4/0x4 [ 4389.481131] [<ffffffff8165564e>] ? _raw_spin_unlock_irqrestore+0x1e/0x30 [ 4389.481139] [<ffffffff8108935d>] ? add_wait_queue+0x4d/0x60 [ 4389.481146] [<ffffffff81654703>] schedule_hrtimeout_range+0x13/0x20 [ 4389.481154] [<ffffffff811877d9>] poll_schedule_timeout+0x49/0x70 [ 4389.481160] [<ffffffff81188356>] do_select+0x4d6/0x600 [ 4389.481167] [<ffffffff811878e0>] ? poll_freewait+0xe0/0xe0 [ 4389.481173] [<ffffffff811879d0>] ? __pollwait+0xf0/0xf0 [ 4389.481179] [<ffffffff81005191>] ? __raw_callee_save_xen_pte_val+0x11/0x1e [ 4389.481186] [<ffffffff81054c3e>] ? update_curr+0x21e/0x230 [ 4389.481192] [<ffffffff8103cc65>] ? pvclock_clocksource_read+0x55/0xf0 [ 4389.481199] [<ffffffff810553db>] ? check_preempt_wakeup+0x15b/0x230 [ 4389.481207] [<ffffffff8104ed74>] ? check_preempt_curr+0x84/0xa0 [ 4389.481214] [<ffffffff8104edcd>] ? ttwu_do_wakeup+0x3d/0x120 [ 4389.481222] [<ffffffff8130fd49>] ? put_dec+0x59/0x60 [ 4389.481228] [<ffffffff81310c1f>] ? number.isra.2+0x31f/0x350 [ 4389.481236] [<ffffffff81323436>] ? nla_parse+0x86/0xe0 [ 4389.481242] [<ffffffff8165564e>] ? _raw_spin_unlock_irqrestore+0x1e/0x30 [ 4389.481250] [<ffffffff8105e610>] ? try_to_wake_up+0x190/0x200 [ 4389.481257] [<ffffffff81188641>] core_sys_select+0x1c1/0x330 [ 4389.481263] [<ffffffff8100aa32>] ? check_events+0x12/0x20 [ 4389.481269] [<ffffffff8100a25d>] ? xen_force_evtchn_callback+0xd/0x10 [ 4389.481276] [<ffffffff8100aa32>] ? check_events+0x12/0x20 [ 4389.481282] [<ffffffff8100aa1f>] ? xen_restore_fl_direct_reloc+0x4/0x4 [ 4389.481289] [<ffffffff81004c62>] ? xen_mc_flush+0xb2/0x1c0 [ 4389.481295] [<ffffffff8100aa1f>] ? xen_restore_fl_direct_reloc+0x4/0x4 [ 4389.481302] [<ffffffff811889eb>] sys_select+0xbb/0x100 [ 4389.481308] [<ffffffff8105edf7>] ? schedule_tail+0x27/0xb0 [ 4389.481314] [<ffffffff8165da42>] system_call_fastpath+0x16/0x1b [ 4389.481319] Code: 89 df e8 69 c1 5e 00 48 8b 73 38 48 c7 c7 7b ef 9f 81 31 c0 e8 98 c9 5e 00 48 8b 73 10 48 c7 c7 98 ef 9f 81 31 c0 e8 86 c9 5e 00 <0f> 0b 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 10 48 89 [ 4389.481407] RIP [<ffffffff81050cd5>] pick_next_entity+0x105/0x110 [ 4389.484006] RSP <ffff8801d0cdd818> [ 4389.484006] ---[ end trace 7ee7cea7516c9821 ]---
And another one from the debug kernel on EC2, with a slightly different call stack:
[ 4389.480352] ------------[ cut here ]------------ precise- amd64/ubuntu- 2.6/kernel/ sched_fair. c:1239! ffffffff81050cd 5>] [<ffffffff81050 cd5>] pick_next_ entity+ 0x105/0x110 cdd818 EFLAGS: 00010092 0(0000) GS:ffff8801dff7 3000(0000) knlGS:000000000 0000000 4e8>] pick_next_ task_fair+ 0x38/0x70 f3c>] __schedule+ 0x14c/0x6f0 5af>] schedule+0x3f/0x60 6cd>] schedule_ hrtimeout_ range_clock+ 0x14d/0x170 a1f>] ? xen_restore_ fl_direct_ reloc+0x4/ 0x4 64e>] ? _raw_spin_ unlock_ irqrestore+ 0x1e/0x30 35d>] ? add_wait_ queue+0x4d/ 0x60 703>] schedule_ hrtimeout_ range+0x13/ 0x20 7d9>] poll_schedule_ timeout+ 0x49/0x70 356>] do_select+ 0x4d6/0x600 8e0>] ? poll_freewait+ 0xe0/0xe0 9d0>] ? __pollwait+ 0xf0/0xf0 191>] ? __raw_callee_ save_xen_ pte_val+ 0x11/0x1e c3e>] ? update_ curr+0x21e/ 0x230 c65>] ? pvclock_ clocksource_ read+0x55/ 0xf0 3db>] ? check_preempt_ wakeup+ 0x15b/0x230 d74>] ? check_preempt_ curr+0x84/ 0xa0 dcd>] ? ttwu_do_ wakeup+ 0x3d/0x120 d49>] ? put_dec+0x59/0x60 c1f>] ? number. isra.2+ 0x31f/0x350 436>] ? nla_parse+0x86/0xe0 64e>] ? _raw_spin_ unlock_ irqrestore+ 0x1e/0x30 610>] ? try_to_ wake_up+ 0x190/0x200 641>] core_sys_ select+ 0x1c1/0x330 a32>] ? check_events+ 0x12/0x20 25d>] ? xen_force_ evtchn_ callback+ 0xd/0x10 a32>] ? check_events+ 0x12/0x20 a1f>] ? xen_restore_ fl_direct_ reloc+0x4/ 0x4 c62>] ? xen_mc_ flush+0xb2/ 0x1c0 a1f>] ? xen_restore_ fl_direct_ reloc+0x4/ 0x4 9eb>] sys_select+ 0xbb/0x100 df7>] ? schedule_ tail+0x27/ 0xb0 a42>] system_ call_fastpath+ 0x16/0x1b cd5>] pick_next_ entity+ 0x105/0x110
[ 4389.480884] kernel BUG at /home/smb/
[ 4389.480894] invalid opcode: 0000 [#1] SMP
[ 4389.480902] CPU 0
[ 4389.480905] Modules linked in: ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables bnep rfcomm bluetooth parport_pc ppdev lp parport isofs acpiphp
[ 4389.480945]
[ 4389.480949] Pid: 24612, comm: ohai Not tainted 3.2.0-24-virtual #37+lp999755dbg1
[ 4389.480958] RIP: e030:[<
[ 4389.480972] RSP: e02b:ffff8801d0
[ 4389.480977] RAX: 000000000000002c RBX: ffff8801d1106e00 RCX: 000000000003ffff
[ 4389.480983] RDX: 0000000000000000 RSI: ffffffff81e000a0 RDI: 0000000000000004
[ 4389.480988] RBP: ffff8801d0cdd838 R08: 000000000000000a R09: 0000000000000000
[ 4389.480994] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801dff866c0
[ 4389.481000] R13: ffff8801d1107a00 R14: 0000000000000008 R15: ffff8801d1e97800
[ 4389.481009] FS: 00007f09fcc8a70
[ 4389.481015] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 4389.481021] CR2: 00000000012dbf30 CR3: 00000001d0702000 CR4: 0000000000002660
[ 4389.481028] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4389.481035] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 4389.481041] Process ohai (pid: 24612, threadinfo ffff8801d0cdc000, task ffff8801d2905b80)
[ 4389.481047] Stack:
[ 4389.481051] ffff8801d1106e00 ffff8801dff866c0 ffff8801d1107a00 0000000000000008
[ 4389.481063] ffff8801d0cdd868 ffffffff810544e8 ffff8801d0cdd868 ffff8801dff866c0
[ 4389.481075] 0000000000000000 ffff8801d2905f28 ffff8801d0cdd8e8 ffffffff81652f3c
[ 4389.481086] Call Trace:
[ 4389.481093] [<ffffffff81054
[ 4389.481101] [<ffffffff81652
[ 4389.481108] [<ffffffff81653
[ 4389.481114] [<ffffffff81654
[ 4389.481124] [<ffffffff8100a
[ 4389.481131] [<ffffffff81655
[ 4389.481139] [<ffffffff81089
[ 4389.481146] [<ffffffff81654
[ 4389.481154] [<ffffffff81187
[ 4389.481160] [<ffffffff81188
[ 4389.481167] [<ffffffff81187
[ 4389.481173] [<ffffffff81187
[ 4389.481179] [<ffffffff81005
[ 4389.481186] [<ffffffff81054
[ 4389.481192] [<ffffffff8103c
[ 4389.481199] [<ffffffff81055
[ 4389.481207] [<ffffffff8104e
[ 4389.481214] [<ffffffff8104e
[ 4389.481222] [<ffffffff8130f
[ 4389.481228] [<ffffffff81310
[ 4389.481236] [<ffffffff81323
[ 4389.481242] [<ffffffff81655
[ 4389.481250] [<ffffffff8105e
[ 4389.481257] [<ffffffff81188
[ 4389.481263] [<ffffffff8100a
[ 4389.481269] [<ffffffff8100a
[ 4389.481276] [<ffffffff8100a
[ 4389.481282] [<ffffffff8100a
[ 4389.481289] [<ffffffff81004
[ 4389.481295] [<ffffffff8100a
[ 4389.481302] [<ffffffff81188
[ 4389.481308] [<ffffffff8105e
[ 4389.481314] [<ffffffff8165d
[ 4389.481319] Code: 89 df e8 69 c1 5e 00 48 8b 73 38 48 c7 c7 7b ef 9f 81 31 c0 e8 98 c9 5e 00 48 8b 73 10 48 c7 c7 98 ef 9f 81 31 c0 e8 86 c9 5e 00 <0f> 0b 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 10 48 89
[ 4389.481407] RIP [<ffffffff81050
[ 4389.484006] RSP <ffff8801d0cdd818>
[ 4389.484006] ---[ end trace 7ee7cea7516c9821 ]---