Hi Stefan, OK. Here's another stack trace:
[47708.053788] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 [47708.053810] IP: [<ffffffff8130d7f1>] rb_next+0x1/0x50 [47708.053824] PGD 1d0b83067 PUD 1d0e64067 PMD 0 [47708.053833] Oops: 0000 [#1] SMP [47708.053840] CPU 1 [47708.053843] Modules linked in: ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables parport_pc ppdev lp parport isofs acpiphp [47708.053874] [47708.053878] Pid: 1190, comm: chef-client Not tainted 3.2.0-24-virtual #37-Ubuntu [47708.053888] RIP: e030:[<ffffffff8130d7f1>] [<ffffffff8130d7f1>] rb_next+0x1/0x50 [47708.053896] RSP: e02b:ffff8801d2e99c18 EFLAGS: 00010046 [47708.053901] RAX: 0000000000000000 RBX: ffff8801d2745800 RCX: 0000000000000000 [47708.053907] RDX: fffffffffffffff0 RSI: 0000000000000000 RDI: 0000000000000010 [47708.053913] RBP: ffff8801d2e99c48 R08: 0000000000000000 R09: 0000000000000000 [47708.053919] R10: ffff8801dff866c0 R11: 0000000000000001 R12: 0000000000000000 [47708.053925] R13: 0000000000000000 R14: 0000000000000000 R15: 000000000240db78 [47708.053935] FS: 00007fc872779700(0000) GS:ffff8801dff8f000(0000) knlGS:0000000000000000 [47708.053943] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b [47708.053948] CR2: 0000000000000010 CR3: 00000001d0b3d000 CR4: 0000000000002660 [47708.053955] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [47708.053961] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [47708.053967] Process chef-client (pid: 1190, threadinfo ffff8801d2e98000, task ffff8801d24d2dc0) [47708.053974] Stack: [47708.053978] ffff8801d2e99c48 ffffffff8104ece9 ffff8801d2745800 ffff8801dffa26c0 [47708.053990] ffff8801d2744a00 0000000000000000 ffff8801d2e99c78 ffffffff810544b8 [47708.054001] ffff8801d2e99c78 ffff8801dffa26c0 0000000000000001 ffff8801d24d3168 [47708.054013] Call Trace: [47708.054022] [<ffffffff8104ece9>] ? pick_next_entity+0xb9/0xe0 [47708.054030] [<ffffffff810544b8>] pick_next_task_fair+0x38/0x70 [47708.054038] [<ffffffff81652ddc>] __schedule+0x14c/0x6f0 [47708.054045] [<ffffffff816554ee>] ? _raw_spin_unlock_irqrestore+0x1e/0x30 [47708.054052] [<ffffffff8165344f>] schedule+0x3f/0x60 [47708.054060] [<ffffffff8117e119>] pipe_wait+0x59/0x80 [47708.054067] [<ffffffff81089340>] ? add_wait_queue+0x60/0x60 [47708.054074] [<ffffffff8117e87a>] pipe_read+0x1da/0x330 [47708.054081] [<ffffffff81174522>] do_sync_read+0xd2/0x110 [47708.054088] [<ffffffff8100a25d>] ? xen_force_evtchn_callback+0xd/0x10 [47708.054095] [<ffffffff8100aa32>] ? check_events+0x12/0x20 [47708.054103] [<ffffffff81298d33>] ? security_file_permission+0x93/0xb0 [47708.054110] [<ffffffff811749a1>] ? rw_verify_area+0x61/0xf0 [47708.054116] [<ffffffff81174e80>] vfs_read+0xb0/0x180 [47708.054122] [<ffffffff81174f9a>] sys_read+0x4a/0x90 [47708.054129] [<ffffffff8165d8c2>] system_call_fastpath+0x16/0x1b [47708.054134] Code: 89 06 48 8b 47 08 48 89 46 08 48 8b 47 10 48 89 46 10 c3 0f 1f 80 00 00 00 00 48 89 32 eb b2 0f 1f 00 48 89 70 10 eb a9 66 90 55 <48> 8b 17 48 89 e5 48 89 d0 48 83 e0 fc 48 39 c7 74 34 48 8b 47 [47708.054224] RIP [<ffffffff8130d7f1>] rb_next+0x1/0x50 [47708.054231] RSP <ffff8801d2e99c18> [47708.054235] CR2: 0000000000000010 [47708.054253] ---[ end trace 3fc2767cd5259b41 ]---
This is, of course still on EC2. Here's the Xen version info from the system log:
Xen version: 3.4.3-2.6.18 (preserve-AD)
Hope that helps!
Hi Stefan, OK. Here's another stack trace:
[47708.053788] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 7f1>] rb_next+0x1/0x50 ffffffff8130d7f 1>] [<ffffffff8130d 7f1>] rb_next+0x1/0x50 e99c18 EFLAGS: 00010046 0(0000) GS:ffff8801dff8 f000(0000) knlGS:000000000 0000000 ce9>] ? pick_next_ entity+ 0xb9/0xe0 4b8>] pick_next_ task_fair+ 0x38/0x70 ddc>] __schedule+ 0x14c/0x6f0 4ee>] ? _raw_spin_ unlock_ irqrestore+ 0x1e/0x30 44f>] schedule+0x3f/0x60 119>] pipe_wait+0x59/0x80 340>] ? add_wait_ queue+0x60/ 0x60 87a>] pipe_read+ 0x1da/0x330 522>] do_sync_ read+0xd2/ 0x110 25d>] ? xen_force_ evtchn_ callback+ 0xd/0x10 a32>] ? check_events+ 0x12/0x20 d33>] ? security_ file_permission +0x93/0xb0 9a1>] ? rw_verify_ area+0x61/ 0xf0 e80>] vfs_read+0xb0/0x180 f9a>] sys_read+0x4a/0x90 8c2>] system_ call_fastpath+ 0x16/0x1b 7f1>] rb_next+0x1/0x50
[47708.053810] IP: [<ffffffff8130d
[47708.053824] PGD 1d0b83067 PUD 1d0e64067 PMD 0
[47708.053833] Oops: 0000 [#1] SMP
[47708.053840] CPU 1
[47708.053843] Modules linked in: ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables parport_pc ppdev lp parport isofs acpiphp
[47708.053874]
[47708.053878] Pid: 1190, comm: chef-client Not tainted 3.2.0-24-virtual #37-Ubuntu
[47708.053888] RIP: e030:[<
[47708.053896] RSP: e02b:ffff8801d2
[47708.053901] RAX: 0000000000000000 RBX: ffff8801d2745800 RCX: 0000000000000000
[47708.053907] RDX: fffffffffffffff0 RSI: 0000000000000000 RDI: 0000000000000010
[47708.053913] RBP: ffff8801d2e99c48 R08: 0000000000000000 R09: 0000000000000000
[47708.053919] R10: ffff8801dff866c0 R11: 0000000000000001 R12: 0000000000000000
[47708.053925] R13: 0000000000000000 R14: 0000000000000000 R15: 000000000240db78
[47708.053935] FS: 00007fc87277970
[47708.053943] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[47708.053948] CR2: 0000000000000010 CR3: 00000001d0b3d000 CR4: 0000000000002660
[47708.053955] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[47708.053961] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[47708.053967] Process chef-client (pid: 1190, threadinfo ffff8801d2e98000, task ffff8801d24d2dc0)
[47708.053974] Stack:
[47708.053978] ffff8801d2e99c48 ffffffff8104ece9 ffff8801d2745800 ffff8801dffa26c0
[47708.053990] ffff8801d2744a00 0000000000000000 ffff8801d2e99c78 ffffffff810544b8
[47708.054001] ffff8801d2e99c78 ffff8801dffa26c0 0000000000000001 ffff8801d24d3168
[47708.054013] Call Trace:
[47708.054022] [<ffffffff8104e
[47708.054030] [<ffffffff81054
[47708.054038] [<ffffffff81652
[47708.054045] [<ffffffff81655
[47708.054052] [<ffffffff81653
[47708.054060] [<ffffffff8117e
[47708.054067] [<ffffffff81089
[47708.054074] [<ffffffff8117e
[47708.054081] [<ffffffff81174
[47708.054088] [<ffffffff8100a
[47708.054095] [<ffffffff8100a
[47708.054103] [<ffffffff81298
[47708.054110] [<ffffffff81174
[47708.054116] [<ffffffff81174
[47708.054122] [<ffffffff81174
[47708.054129] [<ffffffff8165d
[47708.054134] Code: 89 06 48 8b 47 08 48 89 46 08 48 8b 47 10 48 89 46 10 c3 0f 1f 80 00 00 00 00 48 89 32 eb b2 0f 1f 00 48 89 70 10 eb a9 66 90 55 <48> 8b 17 48 89 e5 48 89 d0 48 83 e0 fc 48 39 c7 74 34 48 8b 47
[47708.054224] RIP [<ffffffff8130d
[47708.054231] RSP <ffff8801d2e99c18>
[47708.054235] CR2: 0000000000000010
[47708.054253] ---[ end trace 3fc2767cd5259b41 ]---
This is, of course still on EC2. Here's the Xen version info from the system log:
Xen version: 3.4.3-2.6.18 (preserve-AD)
Hope that helps!