Comment 8 for bug 1056746

Revision history for this message
sles (slesru) wrote : Re: kernel panic on iscsi target disconnect

I had no chance to test mailine yet, but -31 and -32 has the same issue:

ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs]
[ 463.624735] CPU 1
[ 463.624737] Modules linked in: scsi_dh_rdac ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs]
[ 463.624775]
[ 463.624778] Pid: 83, comm: kworker/u:2 Not tainted 3.2.0-32-generic #51-Ubuntu DEPO Computers X8STi/X8STi
[ 463.624784] RIP: 0010:[<ffffffff8142f6b0>] [<ffffffff8142f6b0>] __scsi_remove_target+0x20/0xf0
[ 463.624792] RSP: 0018:ffff880307067d80 EFLAGS: 00000246
[ 463.624795] RAX: 0000000000000000 RBX: ffffffff81cddaa0 RCX: 0000000000000010
[ 463.624798] RDX: ffff8802cc1a9008 RSI: 0000000000000286 RDI: ffff88030a75d930
[ 463.624800] RBP: ffff880307067da0 R08: ffffffff81cddaa0 R09: 0000000000000100
[ 463.624803] R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000010
[ 463.624806] R13: 0000000000000000 R14: 0000000000000010 R15: ffff880307067d70
[ 463.624809] FS: 0000000000000000(0000) GS:ffff88031fc20000(0000) knlGS:0000000000000000
[ 463.624812] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 463.624815] CR2: 00007f94fc247000 CR3: 0000000001c05000 CR4: 00000000000006e0
[ 463.624818] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 463.624821] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 463.624824] Process kworker/u:2 (pid: 83, threadinfo ffff880307066000, task ffff880307032e00)
[ 463.624826] Stack:
[ 463.652982] BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u:3:86]
[ 463.652985] Modules linked in: scsi_dh_rdac ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs]
[ 463.653018] CPU 2
[ 463.653019] Modules linked in: scsi_dh_rdac ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs]
[ 463.653045]
[ 463.653048] Pid: 86, comm: kworker/u:3 Not tainted 3.2.0-32-generic #51-Ubuntu DEPO Computers X8STi/X8STi
[ 463.653052] RIP: 0010:[<ffffffff8165b0ac>] [<ffffffff8165b0ac>] _raw_spin_lock_irqsave+0xc/0x40
[ 463.653060] RSP: 0018:ffff8803070b3d60 EFLAGS: 00000286
[ 463.653062] RAX: 0000000000000001 RBX: 0000000000000010 RCX: 0000000000000010
[ 463.653064] RDX: ffff8802cc1ab008 RSI: 0000000000000286 RDI: ffff88030a75b050
[ 463.653067] RBP: ffff8803070b3d70 R08: ffffffff81cddaa0 R09: 0000000000000100
[ 463.653069] R10: 0000000000000010 R11: 0000000000000000 R12: ffff88028cac8b48
[ 463.653071] R13: ffff8803070b3d70 R14: ffff88028cac8b3c R15: ffff88028cac8758
[ 463.653074] FS: 0000000000000000(0000) GS:ffff88031fc40000(0000) knlGS:0000000000000000
[ 463.653077] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 463.653079] CR2: 00007f8385fef7b0 CR3: 0000000001c05000 CR4: 00000000000006e0
[ 463.653081] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 463.653084] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 463.653087] Process kworker/u:3 (pid: 86, threadinfo ffff8803070b2000, task ffff8803070b8000)
[ 463.653088] Stack:
[ 463.653090] ffff88030a75e130 ffff8802cc1ab000 ffff8803070b3da0 ffffffff8142f6d9
[ 463.653094] ffff88030a75e130 ffff88030a75b010 ffff88030a75b000 ffff8802cc1ab000
[ 463.653098] ffff8803070b3dd0 ffffffff8142f841 ffff88030a75e130 ffff88030a75e000
[ 463.653102] Call Trace:
[ 463.653107] [<ffffffff8142f6d9>] __scsi_remove_target+0x49/0xf0
[ 463.653111] [<ffffffff8142f841>] scsi_remove_target+0xc1/0xe0
[ 463.653119] [<ffffffffa0139fec>] __iscsi_unbind_session+0xbc/0x190 [scsi_transport_iscsi]
[ 463.653126] [<ffffffff81084b1a>] process_one_work+0x11a/0x480
[ 463.653130] [<ffffffff810858c4>] worker_thread+0x164/0x370
[ 463.653134] [<ffffffff81085760>] ? manage_workers.isra.30+0x130/0x130
[ 463.653138] [<ffffffff8108a0dc>] kthread+0x8c/0xa0
[ 463.653144] [<ffffffff816655b4>] kernel_thread_helper+0x4/0x10
[ 463.653148] [<ffffffff8108a050>] ? flush_kthread_worker+0xa0/0xa0
[ 463.653151] [<ffffffff816655b0>] ? gs_change+0x13/0x13
[ 463.653153] Code: 48 89 e5 66 66 66 66 90 f0 81 07 00 00 10 00 48 89 f7 57 9d 66 66 90 66 90 5d c3 0f 1f 40 00 55 48 89 e5 48 83 ec 10 48 89 5d f0 <4c> 89 65 f8 66 66 66 66 90 9c 58 66 66 90 66 90 48 89 c3 fa 66
[ 463.653177] Call Trace:
[ 463.653180] [<ffffffff8142f6d9>] __scsi_remove_target+0x49/0xf0
[ 463.653183] [<ffffffff8142f841>] scsi_remove_target+0xc1/0xe0
[ 463.653190] [<ffffffffa0139fec>] __iscsi_unbind_session+0xbc/0x190 [scsi_transport_iscsi]
[ 463.653194] [<ffffffff81084b1a>] process_one_work+0x11a/0x480
[ 463.653198] [<ffffffff810858c4>] worker_thread+0x164/0x370
[ 463.653202] [<ffffffff81085760>] ? manage_workers.isra.30+0x130/0x130
[ 463.653205] [<ffffffff8108a0dc>] kthread+0x8c/0xa0
[ 463.653209] [<ffffffff816655b4>] kernel_thread_helper+0x4/0x10
[ 463.653212] [<ffffffff8108a050>] ? flush_kthread_worker+0xa0/0xa0
[ 463.653216] [<ffffffff816655b0>] ? gs_change+0x13/0x13
[ 463.819157] ffff88030a75d930 ffff88030a759810 ffff88030a759800 ffff8802cc1a9000
[ 463.819163] ffff880307067dd0 ffffffff8142f841 ffff88030a75d930 ffff88030a75d800
[ 463.819168] ffff88030a75d818 0000000000000000 ffff880307067e00 ffffffffa0139fec
[ 463.819173] Call Trace:
[ 463.852454] [<ffffffff8142f841>] scsi_remove_target+0xc1/0xe0
[ 463.852464] [<ffffffffa0139fec>] __iscsi_unbind_session+0xbc/0x190 [scsi_transport_iscsi]
[ 463.852470] [<ffffffff81084b1a>] process_one_work+0x11a/0x480
[ 463.852475] [<ffffffff810858c4>] worker_thread+0x164/0x370
[ 463.852479] [<ffffffff81085760>] ? manage_workers.isra.30+0x130/0x130
[ 463.852484] [<ffffffff8108a0dc>] kthread+0x8c/0xa0
[ 463.852489] [<ffffffff816655b4>] kernel_thread_helper+0x4/0x10
[ 463.852493] [<ffffffff8108a050>] ? flush_kthread_worker+0xa0/0xa0
[ 463.852497] [<ffffffff816655b0>] ? gs_change+0x13/0x13
[ 463.852499] Code: 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 56 41 55 41 54 53 66 66 66 66 90 4c 8b 6f 28 49 89 fc eb 14 0f 1f 80 00 00 00 00 <4d> 8b 6d 00 4d 85 ed 0f 84 b3 00 00 00 4c 89 ef e8 cb 34 ff ff
[ 463.924424] Call Trace:
[ 463.924429] [<ffffffff8142f841>] scsi_remove_target+0xc1/0xe0
[ 463.924438] [<ffffffffa0139fec>] __iscsi_unbind_session+0xbc/0x190 [scsi_transport_iscsi]
[ 463.924445] [<ffffffff81084b1a>] process_one_work+0x11a/0x480
[ 463.924452] [<ffffffff810858c4>] worker_thread+0x164/0x370
[ 463.924458] [<ffffffff81085760>] ? manage_workers.isra.30+0x130/0x130
[ 463.924464] [<ffffffff8108a0dc>] kthread+0x8c/0xa0
[ 463.924470] [<ffffffff816655b4>] kernel_thread_helper+0x4/0x10
[ 463.924474] [<ffffffff8108a050>] ? flush_kthread_worker+0xa0/0xa0
[ 463.924478] [<ffffffff816655b0>] ? gs_change+0x13/0x13

And yes, we use multipath.
Could you tell me right sequence in init scripst to implement this procedure -
 multipath -F
etc?