I had no chance to test mailine yet, but -31 and -32 has the same issue:
ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs] [ 463.624735] CPU 1 [ 463.624737] Modules linked in: scsi_dh_rdac ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs] [ 463.624775] [ 463.624778] Pid: 83, comm: kworker/u:2 Not tainted 3.2.0-32-generic #51-Ubuntu DEPO Computers X8STi/X8STi [ 463.624784] RIP: 0010:[<ffffffff8142f6b0>] [<ffffffff8142f6b0>] __scsi_remove_target+0x20/0xf0 [ 463.624792] RSP: 0018:ffff880307067d80 EFLAGS: 00000246 [ 463.624795] RAX: 0000000000000000 RBX: ffffffff81cddaa0 RCX: 0000000000000010 [ 463.624798] RDX: ffff8802cc1a9008 RSI: 0000000000000286 RDI: ffff88030a75d930 [ 463.624800] RBP: ffff880307067da0 R08: ffffffff81cddaa0 R09: 0000000000000100 [ 463.624803] R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000010 [ 463.624806] R13: 0000000000000000 R14: 0000000000000010 R15: ffff880307067d70 [ 463.624809] FS: 0000000000000000(0000) GS:ffff88031fc20000(0000) knlGS:0000000000000000 [ 463.624812] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 463.624815] CR2: 00007f94fc247000 CR3: 0000000001c05000 CR4: 00000000000006e0 [ 463.624818] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 463.624821] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 463.624824] Process kworker/u:2 (pid: 83, threadinfo ffff880307066000, task ffff880307032e00) [ 463.624826] Stack: [ 463.652982] BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u:3:86] [ 463.652985] Modules linked in: scsi_dh_rdac ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs] [ 463.653018] CPU 2 [ 463.653019] Modules linked in: scsi_dh_rdac ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs] [ 463.653045] [ 463.653048] Pid: 86, comm: kworker/u:3 Not tainted 3.2.0-32-generic #51-Ubuntu DEPO Computers X8STi/X8STi [ 463.653052] RIP: 0010:[<ffffffff8165b0ac>] [<ffffffff8165b0ac>] _raw_spin_lock_irqsave+0xc/0x40 [ 463.653060] RSP: 0018:ffff8803070b3d60 EFLAGS: 00000286 [ 463.653062] RAX: 0000000000000001 RBX: 0000000000000010 RCX: 0000000000000010 [ 463.653064] RDX: ffff8802cc1ab008 RSI: 0000000000000286 RDI: ffff88030a75b050 [ 463.653067] RBP: ffff8803070b3d70 R08: ffffffff81cddaa0 R09: 0000000000000100 [ 463.653069] R10: 0000000000000010 R11: 0000000000000000 R12: ffff88028cac8b48 [ 463.653071] R13: ffff8803070b3d70 R14: ffff88028cac8b3c R15: ffff88028cac8758 [ 463.653074] FS: 0000000000000000(0000) GS:ffff88031fc40000(0000) knlGS:0000000000000000 [ 463.653077] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 463.653079] CR2: 00007f8385fef7b0 CR3: 0000000001c05000 CR4: 00000000000006e0 [ 463.653081] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 463.653084] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 463.653087] Process kworker/u:3 (pid: 86, threadinfo ffff8803070b2000, task ffff8803070b8000) [ 463.653088] Stack: [ 463.653090] ffff88030a75e130 ffff8802cc1ab000 ffff8803070b3da0 ffffffff8142f6d9 [ 463.653094] ffff88030a75e130 ffff88030a75b010 ffff88030a75b000 ffff8802cc1ab000 [ 463.653098] ffff8803070b3dd0 ffffffff8142f841 ffff88030a75e130 ffff88030a75e000 [ 463.653102] Call Trace: [ 463.653107] [<ffffffff8142f6d9>] __scsi_remove_target+0x49/0xf0 [ 463.653111] [<ffffffff8142f841>] scsi_remove_target+0xc1/0xe0 [ 463.653119] [<ffffffffa0139fec>] __iscsi_unbind_session+0xbc/0x190 [scsi_transport_iscsi] [ 463.653126] [<ffffffff81084b1a>] process_one_work+0x11a/0x480 [ 463.653130] [<ffffffff810858c4>] worker_thread+0x164/0x370 [ 463.653134] [<ffffffff81085760>] ? manage_workers.isra.30+0x130/0x130 [ 463.653138] [<ffffffff8108a0dc>] kthread+0x8c/0xa0 [ 463.653144] [<ffffffff816655b4>] kernel_thread_helper+0x4/0x10 [ 463.653148] [<ffffffff8108a050>] ? flush_kthread_worker+0xa0/0xa0 [ 463.653151] [<ffffffff816655b0>] ? gs_change+0x13/0x13 [ 463.653153] Code: 48 89 e5 66 66 66 66 90 f0 81 07 00 00 10 00 48 89 f7 57 9d 66 66 90 66 90 5d c3 0f 1f 40 00 55 48 89 e5 48 83 ec 10 48 89 5d f0 <4c> 89 65 f8 66 66 66 66 90 9c 58 66 66 90 66 90 48 89 c3 fa 66 [ 463.653177] Call Trace: [ 463.653180] [<ffffffff8142f6d9>] __scsi_remove_target+0x49/0xf0 [ 463.653183] [<ffffffff8142f841>] scsi_remove_target+0xc1/0xe0 [ 463.653190] [<ffffffffa0139fec>] __iscsi_unbind_session+0xbc/0x190 [scsi_transport_iscsi] [ 463.653194] [<ffffffff81084b1a>] process_one_work+0x11a/0x480 [ 463.653198] [<ffffffff810858c4>] worker_thread+0x164/0x370 [ 463.653202] [<ffffffff81085760>] ? manage_workers.isra.30+0x130/0x130 [ 463.653205] [<ffffffff8108a0dc>] kthread+0x8c/0xa0 [ 463.653209] [<ffffffff816655b4>] kernel_thread_helper+0x4/0x10 [ 463.653212] [<ffffffff8108a050>] ? flush_kthread_worker+0xa0/0xa0 [ 463.653216] [<ffffffff816655b0>] ? gs_change+0x13/0x13 [ 463.819157] ffff88030a75d930 ffff88030a759810 ffff88030a759800 ffff8802cc1a9000 [ 463.819163] ffff880307067dd0 ffffffff8142f841 ffff88030a75d930 ffff88030a75d800 [ 463.819168] ffff88030a75d818 0000000000000000 ffff880307067e00 ffffffffa0139fec [ 463.819173] Call Trace: [ 463.852454] [<ffffffff8142f841>] scsi_remove_target+0xc1/0xe0 [ 463.852464] [<ffffffffa0139fec>] __iscsi_unbind_session+0xbc/0x190 [scsi_transport_iscsi] [ 463.852470] [<ffffffff81084b1a>] process_one_work+0x11a/0x480 [ 463.852475] [<ffffffff810858c4>] worker_thread+0x164/0x370 [ 463.852479] [<ffffffff81085760>] ? manage_workers.isra.30+0x130/0x130 [ 463.852484] [<ffffffff8108a0dc>] kthread+0x8c/0xa0 [ 463.852489] [<ffffffff816655b4>] kernel_thread_helper+0x4/0x10 [ 463.852493] [<ffffffff8108a050>] ? flush_kthread_worker+0xa0/0xa0 [ 463.852497] [<ffffffff816655b0>] ? gs_change+0x13/0x13 [ 463.852499] Code: 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 56 41 55 41 54 53 66 66 66 66 90 4c 8b 6f 28 49 89 fc eb 14 0f 1f 80 00 00 00 00 <4d> 8b 6d 00 4d 85 ed 0f 84 b3 00 00 00 4c 89 ef e8 cb 34 ff ff [ 463.924424] Call Trace: [ 463.924429] [<ffffffff8142f841>] scsi_remove_target+0xc1/0xe0 [ 463.924438] [<ffffffffa0139fec>] __iscsi_unbind_session+0xbc/0x190 [scsi_transport_iscsi] [ 463.924445] [<ffffffff81084b1a>] process_one_work+0x11a/0x480 [ 463.924452] [<ffffffff810858c4>] worker_thread+0x164/0x370 [ 463.924458] [<ffffffff81085760>] ? manage_workers.isra.30+0x130/0x130 [ 463.924464] [<ffffffff8108a0dc>] kthread+0x8c/0xa0 [ 463.924470] [<ffffffff816655b4>] kernel_thread_helper+0x4/0x10 [ 463.924474] [<ffffffff8108a050>] ? flush_kthread_worker+0xa0/0xa0 [ 463.924478] [<ffffffff816655b0>] ? gs_change+0x13/0x13
And yes, we use multipath. Could you tell me right sequence in init scripst to implement this procedure - multipath -F etc?
I had no chance to test mailine yet, but -31 and -32 has the same issue:
ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_ iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs] iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs] ffffffff8142f6b 0>] [<ffffffff8142f 6b0>] __scsi_ remove_ target+ 0x20/0xf0 067d80 EFLAGS: 00000246 0(0000) GS:ffff88031fc2 0000(0000) knlGS:000000000 0000000 iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs] iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs] ffffffff8165b0a c>] [<ffffffff8165b 0ac>] _raw_spin_ lock_irqsave+ 0xc/0x40 0b3d60 EFLAGS: 00000286 0(0000) GS:ffff88031fc4 0000(0000) knlGS:000000000 0000000 6d9>] __scsi_ remove_ target+ 0x49/0xf0 841>] scsi_remove_ target+ 0xc1/0xe0 fec>] __iscsi_ unbind_ session+ 0xbc/0x190 [scsi_transport _iscsi] b1a>] process_ one_work+ 0x11a/0x480 8c4>] worker_ thread+ 0x164/0x370 760>] ? manage_ workers. isra.30+ 0x130/0x130 0dc>] kthread+0x8c/0xa0 5b4>] kernel_ thread_ helper+ 0x4/0x10 050>] ? flush_kthread_ worker+ 0xa0/0xa0 5b0>] ? gs_change+0x13/0x13 6d9>] __scsi_ remove_ target+ 0x49/0xf0 841>] scsi_remove_ target+ 0xc1/0xe0 fec>] __iscsi_ unbind_ session+ 0xbc/0x190 [scsi_transport _iscsi] b1a>] process_ one_work+ 0x11a/0x480 8c4>] worker_ thread+ 0x164/0x370 760>] ? manage_ workers. isra.30+ 0x130/0x130 0dc>] kthread+0x8c/0xa0 5b4>] kernel_ thread_ helper+ 0x4/0x10 050>] ? flush_kthread_ worker+ 0xa0/0xa0 5b0>] ? gs_change+0x13/0x13 841>] scsi_remove_ target+ 0xc1/0xe0 fec>] __iscsi_ unbind_ session+ 0xbc/0x190 [scsi_transport _iscsi] b1a>] process_ one_work+ 0x11a/0x480 8c4>] worker_ thread+ 0x164/0x370 760>] ? manage_ workers. isra.30+ 0x130/0x130 0dc>] kthread+0x8c/0xa0 5b4>] kernel_ thread_ helper+ 0x4/0x10 050>] ? flush_kthread_ worker+ 0xa0/0xa0 5b0>] ? gs_change+0x13/0x13 841>] scsi_remove_ target+ 0xc1/0xe0 fec>] __iscsi_ unbind_ session+ 0xbc/0x190 [scsi_transport _iscsi] b1a>] process_ one_work+ 0x11a/0x480 8c4>] worker_ thread+ 0x164/0x370 760>] ? manage_ workers. isra.30+ 0x130/0x130 0dc>] kthread+0x8c/0xa0 5b4>] kernel_ thread_ helper+ 0x4/0x10 050>] ? flush_kthread_ worker+ 0xa0/0xa0 5b0>] ? gs_change+0x13/0x13
[ 463.624735] CPU 1
[ 463.624737] Modules linked in: scsi_dh_rdac ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_
[ 463.624775]
[ 463.624778] Pid: 83, comm: kworker/u:2 Not tainted 3.2.0-32-generic #51-Ubuntu DEPO Computers X8STi/X8STi
[ 463.624784] RIP: 0010:[<
[ 463.624792] RSP: 0018:ffff880307
[ 463.624795] RAX: 0000000000000000 RBX: ffffffff81cddaa0 RCX: 0000000000000010
[ 463.624798] RDX: ffff8802cc1a9008 RSI: 0000000000000286 RDI: ffff88030a75d930
[ 463.624800] RBP: ffff880307067da0 R08: ffffffff81cddaa0 R09: 0000000000000100
[ 463.624803] R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000010
[ 463.624806] R13: 0000000000000000 R14: 0000000000000010 R15: ffff880307067d70
[ 463.624809] FS: 000000000000000
[ 463.624812] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 463.624815] CR2: 00007f94fc247000 CR3: 0000000001c05000 CR4: 00000000000006e0
[ 463.624818] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 463.624821] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 463.624824] Process kworker/u:2 (pid: 83, threadinfo ffff880307066000, task ffff880307032e00)
[ 463.624826] Stack:
[ 463.652982] BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u:3:86]
[ 463.652985] Modules linked in: scsi_dh_rdac ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_
[ 463.653018] CPU 2
[ 463.653019] Modules linked in: scsi_dh_rdac ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_
[ 463.653045]
[ 463.653048] Pid: 86, comm: kworker/u:3 Not tainted 3.2.0-32-generic #51-Ubuntu DEPO Computers X8STi/X8STi
[ 463.653052] RIP: 0010:[<
[ 463.653060] RSP: 0018:ffff880307
[ 463.653062] RAX: 0000000000000001 RBX: 0000000000000010 RCX: 0000000000000010
[ 463.653064] RDX: ffff8802cc1ab008 RSI: 0000000000000286 RDI: ffff88030a75b050
[ 463.653067] RBP: ffff8803070b3d70 R08: ffffffff81cddaa0 R09: 0000000000000100
[ 463.653069] R10: 0000000000000010 R11: 0000000000000000 R12: ffff88028cac8b48
[ 463.653071] R13: ffff8803070b3d70 R14: ffff88028cac8b3c R15: ffff88028cac8758
[ 463.653074] FS: 000000000000000
[ 463.653077] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 463.653079] CR2: 00007f8385fef7b0 CR3: 0000000001c05000 CR4: 00000000000006e0
[ 463.653081] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 463.653084] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 463.653087] Process kworker/u:3 (pid: 86, threadinfo ffff8803070b2000, task ffff8803070b8000)
[ 463.653088] Stack:
[ 463.653090] ffff88030a75e130 ffff8802cc1ab000 ffff8803070b3da0 ffffffff8142f6d9
[ 463.653094] ffff88030a75e130 ffff88030a75b010 ffff88030a75b000 ffff8802cc1ab000
[ 463.653098] ffff8803070b3dd0 ffffffff8142f841 ffff88030a75e130 ffff88030a75e000
[ 463.653102] Call Trace:
[ 463.653107] [<ffffffff8142f
[ 463.653111] [<ffffffff8142f
[ 463.653119] [<ffffffffa0139
[ 463.653126] [<ffffffff81084
[ 463.653130] [<ffffffff81085
[ 463.653134] [<ffffffff81085
[ 463.653138] [<ffffffff8108a
[ 463.653144] [<ffffffff81665
[ 463.653148] [<ffffffff8108a
[ 463.653151] [<ffffffff81665
[ 463.653153] Code: 48 89 e5 66 66 66 66 90 f0 81 07 00 00 10 00 48 89 f7 57 9d 66 66 90 66 90 5d c3 0f 1f 40 00 55 48 89 e5 48 83 ec 10 48 89 5d f0 <4c> 89 65 f8 66 66 66 66 90 9c 58 66 66 90 66 90 48 89 c3 fa 66
[ 463.653177] Call Trace:
[ 463.653180] [<ffffffff8142f
[ 463.653183] [<ffffffff8142f
[ 463.653190] [<ffffffffa0139
[ 463.653194] [<ffffffff81084
[ 463.653198] [<ffffffff81085
[ 463.653202] [<ffffffff81085
[ 463.653205] [<ffffffff8108a
[ 463.653209] [<ffffffff81665
[ 463.653212] [<ffffffff8108a
[ 463.653216] [<ffffffff81665
[ 463.819157] ffff88030a75d930 ffff88030a759810 ffff88030a759800 ffff8802cc1a9000
[ 463.819163] ffff880307067dd0 ffffffff8142f841 ffff88030a75d930 ffff88030a75d800
[ 463.819168] ffff88030a75d818 0000000000000000 ffff880307067e00 ffffffffa0139fec
[ 463.819173] Call Trace:
[ 463.852454] [<ffffffff8142f
[ 463.852464] [<ffffffffa0139
[ 463.852470] [<ffffffff81084
[ 463.852475] [<ffffffff81085
[ 463.852479] [<ffffffff81085
[ 463.852484] [<ffffffff8108a
[ 463.852489] [<ffffffff81665
[ 463.852493] [<ffffffff8108a
[ 463.852497] [<ffffffff81665
[ 463.852499] Code: 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 56 41 55 41 54 53 66 66 66 66 90 4c 8b 6f 28 49 89 fc eb 14 0f 1f 80 00 00 00 00 <4d> 8b 6d 00 4d 85 ed 0f 84 b3 00 00 00 4c 89 ef e8 cb 34 ff ff
[ 463.924424] Call Trace:
[ 463.924429] [<ffffffff8142f
[ 463.924438] [<ffffffffa0139
[ 463.924445] [<ffffffff81084
[ 463.924452] [<ffffffff81085
[ 463.924458] [<ffffffff81085
[ 463.924464] [<ffffffff8108a
[ 463.924470] [<ffffffff81665
[ 463.924474] [<ffffffff8108a
[ 463.924478] [<ffffffff81665
And yes, we use multipath.
Could you tell me right sequence in init scripst to implement this procedure -
multipath -F
etc?