This has been our experience as well.

I can verify that flushing the paths using *sudo multipath -F* prior to
logging off the iSCSI nodes would avoid the hang.

However as under normal operations we have LVM volumes on the multipath
device it was necessary to perform a *sudo lvchange -an <path to logical
volume>* to mark the lv as unavailable, otherwise the the multipath -F
command fails sue to an active path.

Nevertheless, if we follow this procedure we can reliably logout of the
iscsi nodes without a hang.

Stephen

On 16 October 2012 15:20, Louis Bouchard <email address hidden>wrote:

> One thing that is important to know is that it will not happen on a
> system that does not use multipath.  On the latest kernel, one
> workaround is to flush the multipath tables before logging out the iscsi
> nodes :
>
>  $ sudo invoke-rc.d multipath-tools stop
>  $ sudo multipath -F
>  $ sudo iscsiadm -m node --logout
>
> This will work and avoid hanging the kernel.
>
> I'm now proceeding to bisect between those two kernel versions
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1056746
>
> Title:
>   kernel panic on iscsi target disconnect
>
> Status in “linux” package in Ubuntu:
>   Incomplete
>
> Bug description:
>   Hello!
>
>   Just rebooted two identical Ubuntu 12.04.1 LTS servers to set linux-
>   image-3.2.0-31-generic
>
>   And got the same panic on both. Don't think it is fixed in -31.
>
>   Servers are connected to IBM3512 over iscsi with multipath rdac and
> ocfs2.
>   Here is log from one of servers:
>
>
>
>   Sep 26 13:13:17 inetgw1 kernel: [1292980.531921] o2dlm: Leaving domain
> 68166ECC9E2A48208239B14D8218F0DA
>   Sep 26 13:13:18 inetgw1 kernel: [1292981.054231] o2net: No longer
> connected to node inetgw2 (num 1) at 192.168.19.254:7777
>   Sep 26 13:13:18 inetgw1 kernel: [1292981.055378] ocfs2: Unmounting
> device (252,6) on (node 0)
>   Sep 26 13:13:20 inetgw1 kernel: [1292982.774884] ocfs2: Unregistered
> cluster interface o2cb
>   Sep 26 13:13:24 inetgw1 kernel: [1292987.277805] sd 5:0:0:0: rdac:
> Detached
>   Sep 26 13:13:24 inetgw1 kernel: [1292987.277882] sd 6:0:0:0: rdac:
> Detached
>   Sep 26 13:13:24 inetgw1 kernel: [1292987.279549] sd 5:0:0:0: [sdb]
> Synchronizing SCSI cache
>   Sep 26 13:13:24 inetgw1 kernel: [1292987.279808] sd 6:0:0:0: [sdc]
> Synchronizing SCSI cache
>   Sep 26 13:13:26 inetgw1 kernel: [1292988.817740] sd 5:0:0:0: [sdb]
>  Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
>   Sep 26 13:13:26 inetgw1 kernel: [1292988.817746] sd 5:0:0:0: [sdb]
>  Sense Key : Illegal Request [current]
>   Sep 26 13:13:26 inetgw1 kernel: [1292988.817750] sd 5:0:0:0: [sdb]
>  <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
>
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970176] Modules linked in:
> btrfs zlib_deflate ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs
>   reiserfs ext2 ip_vs libcrc32c scsi_dh_rdac xt_conntrack nf_conntrack_ftp
> xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_c
>   onntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT
> xt_tcpudp iptable_filter ip_tables x_tables ib_iser rdma_cm ib_cm iw_cm i
>   b_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
> scsi_transport_iscsi vesafb dm_round_robin 8021q garp stp ioatdma psmouse
> dca seri
>   o_raw dm_multipath joydev i7core_edac edac_core mac_hid lp parport
> usbhid hid megaraid_sas e1000e [last unloaded: configfs]
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970234] CPU 0
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970236] Modules linked in:
> btrfs zlib_deflate ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs
>   reiserfs ext2 ip_vs libcrc32c scsi_dh_rdac xt_conntrack nf_conntrack_ftp
> xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_c
>   onntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT
> xt_tcpudp iptable_filter ip_tables x_tables ib_iser rdma_cm ib_cm iw_cm i
>   b_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
> scsi_transport_iscsi vesafb dm_round_robin 8021q garp stp ioatdma psmouse
> dca seri
>   o_raw dm_multipath joydev i7core_edac edac_core mac_hid lp parport
> usbhid hid megaraid_sas e1000e [last unloaded: configfs]
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970283]
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970286] Pid: 30450, comm:
> kworker/u:2 Not tainted 3.2.0-30-generic #48-Ubuntu DEPO Computers X8STi/X8
>   STi
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970292] RIP:
> 0010:[<ffffffff8165a629>]  [<ffffffff8165a629>]
> _raw_spin_unlock_irqrestore+0x19/0x30
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970301] RSP:
> 0018:ffff880113bf1d68  EFLAGS: 00000286
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970303] RAX: 0000000000000286
> RBX: 0000000000000010 RCX: 0000000000000010
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970306] RDX: ffff880309c47418
> RSI: 0000000000000286 RDI: 0000000000000286
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970309] RBP: ffff880113bf1d70
> R08: ffffffff81cddaa0 R09: 0000000000000100
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970312] R10: 0000000000000010
> R11: 0000000000000000 R12: ffff880306fb92a0
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970314] R13: ffff880113bf1d70
> R14: ffff880306fb9294 R15: ffff880306fb8eb0
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970318] FS:
>  0000000000000000(0000) GS:ffff88031fc00000(0000) knlGS:0000000000000000
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970321] CS:  0010 DS: 0000 ES:
> 0000 CR0: 000000008005003b
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970323] CR2: 00007fc2c6b46000
> CR3: 0000000001c05000 CR4: 00000000000006f0
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970326] DR0: 0000000000000000
> DR1: 0000000000000000 DR2: 0000000000000000
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970329] DR3: 0000000000000000
> DR6: 00000000ffff0ff0 DR7: 0000000000000400
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.970332] Process kworker/u:2
> (pid: 30450, threadinfo ffff880113bf0000, task ffff8802b88f4500)
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.978403]  ffff880309c47400
> ffff880113bf1da0 ffffffff8142d1e7 ffff880305c64930
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.978412]  ffff880305c64930
> ffff88030ac0b010 ffff88030ac0b000 ffff880113bf1dd0
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.978419]  ffffffff8142f3c9
> ffff880305c64930 ffff880305c64800 ffff880305c64818
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.986383]  [<ffffffff8142d1e7>]
> scsi_target_reap+0x77/0xf0
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.986389]  [<ffffffff8142f3c9>]
> scsi_remove_target+0xc9/0xe0
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.986404]  [<ffffffffa00f9fec>]
> __iscsi_unbind_session+0xbc/0x190 [scsi_transport_iscsi]
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.986413]  [<ffffffff81084a6a>]
> process_one_work+0x11a/0x480
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.986423]  [<ffffffff81085814>]
> worker_thread+0x164/0x370
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.986428]  [<ffffffff810856b0>] ?
> manage_workers.isra.30+0x130/0x130
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.986432]  [<ffffffff8108a03c>]
> kthread+0x8c/0xa0
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.986438]  [<ffffffff81664b74>]
> kernel_thread_helper+0x4/0x10
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.986442]  [<ffffffff81089fb0>] ?
> flush_kthread_worker+0xa0/0xa0
>   Sep 26 13:13:54 inetgw1 kernel: [1293013.986447]  [<ffffffff81664b70>] ?
> gs_change+0x13/0x13
>   Sep 26 13:13:54 inetgw1 kernel: [1293014.010878] Call Trace:
>   Sep 26 13:13:54 inetgw1 kernel: [1293014.010884]  [<ffffffff8142d1e7>]
> scsi_target_reap+0x77/0xf0
>   Sep 26 13:13:54 inetgw1 kernel: [1293014.010888]  [<ffffffff8142f3c9>]
> scsi_remove_target+0xc9/0xe0
>   Sep 26 13:13:54 inetgw1 kernel: [1293014.010897]  [<ffffffffa00f9fec>]
> __iscsi_unbind_session+0xbc/0x190 [scsi_transport_iscsi]
>   Sep 26 13:13:54 inetgw1 kernel: [1293014.010902]  [<ffffffff81084a6a>]
> process_one_work+0x11a/0x480
>   Sep 26 13:13:54 inetgw1 kernel: [1293014.010906]  [<ffffffff81085814>]
> worker_thread+0x164/0x370
>   Sep 26 13:13:54 inetgw1 kernel: [1293014.010911]  [<ffffffff810856b0>] ?
> manage_workers.isra.30+0x130/0x130
>   Sep 26 13:13:54 inetgw1 kernel: [1293014.010915]  [<ffffffff8108a03c>]
> kthread+0x8c/0xa0
>   Sep 26 13:13:54 inetgw1 kernel: [1293014.010919]  [<ffffffff81664b74>]
> kernel_thread_helper+0x4/0x10
>   Sep 26 13:13:54 inetgw1 kernel: [1293014.010923]  [<ffffffff81089fb0>] ?
> flush_kthread_worker+0xa0/0xa0
>   Sep 26 13:13:54 inetgw1 kernel: [1293014.010927]  [<ffffffff81664b70>] ?
> gs_change+0x13/0x13
>   Sep 26 13:13:54 inetgw1 kernel: [1293014.054272] Modules linked in:
> btrfs zlib_deflate ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs
>   reiserfs ext2 ip_vs libcrc32c scsi_dh_rdac xt_conntrack nf_conntrack_ftp
> xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_c
>   onntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT
> xt_tcpudp iptable_filter ip_tables x_tables ib_iser rdma_cm ib_cm iw_cm i
>   b_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
> scsi_transport_iscsi vesafb dm_round_robin 8021q garp stp ioatdma psmouse
> dca seri
>   o_raw dm_multipath joydev i7core_edac edac_core mac_hid lp parport
> usbhid hid megaraid_sas e1000e [last unloaded: configfs]
>
>
>   server console was not available at this time , ssh too, although I can
> ping server, so I had to reset both servers.
>
>   There was no such problem with -29 kernel...
>
>   Thank you!
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1056746/+subscriptions
>
>


-- 
*
Stephen Buckley
Senior Systems Administrator

Hogarth Worldwide Limited
Direct: +44 207 420 8360
Mobile: +44 789 4788351

www.hogarthww.com*

-- 

------------------------------

This email contains information from Hogarth Worldwide Limited which may be 
privileged or confidential. The information is intended to be for the use 
of the individual(s) or entity named above. If you are not the intended 
recipient be aware that any disclosure, copying, distribution or use of the 
contents of this information is prohibited. If you have received this email 
in error, please contact the sender.

Headquarters/registered office: 164 Shaftesbury Avenue, London WC2H 8HL 
United Kingdom T: +44 20 7240 6400 W:www.hogarthww.com

Registered in England and Wales. Company No. 06872427. VAT No. 973 7879 46