kernel thread hang on iscsi target disconnect

Bug #1056746 reported by sles on 2012-09-26
82
This bug affects 15 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
High
Louis Bouchard
Precise
High
Louis Bouchard
Quantal
High
Louis Bouchard

Bug Description

Hello!

Just rebooted two identical Ubuntu 12.04.1 LTS servers to set linux-image-3.2.0-31-generic

And got the same panic on both. Don't think it is fixed in -31.

Servers are connected to IBM3512 over iscsi with multipath rdac and ocfs2.
Here is log from one of servers:

Sep 26 13:13:17 inetgw1 kernel: [1292980.531921] o2dlm: Leaving domain 68166ECC9E2A48208239B14D8218F0DA
Sep 26 13:13:18 inetgw1 kernel: [1292981.054231] o2net: No longer connected to node inetgw2 (num 1) at 192.168.19.254:7777
Sep 26 13:13:18 inetgw1 kernel: [1292981.055378] ocfs2: Unmounting device (252,6) on (node 0)
Sep 26 13:13:20 inetgw1 kernel: [1292982.774884] ocfs2: Unregistered cluster interface o2cb
Sep 26 13:13:24 inetgw1 kernel: [1292987.277805] sd 5:0:0:0: rdac: Detached
Sep 26 13:13:24 inetgw1 kernel: [1292987.277882] sd 6:0:0:0: rdac: Detached
Sep 26 13:13:24 inetgw1 kernel: [1292987.279549] sd 5:0:0:0: [sdb] Synchronizing SCSI cache
Sep 26 13:13:24 inetgw1 kernel: [1292987.279808] sd 6:0:0:0: [sdc] Synchronizing SCSI cache
Sep 26 13:13:26 inetgw1 kernel: [1292988.817740] sd 5:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Sep 26 13:13:26 inetgw1 kernel: [1292988.817746] sd 5:0:0:0: [sdb] Sense Key : Illegal Request [current]
Sep 26 13:13:26 inetgw1 kernel: [1292988.817750] sd 5:0:0:0: [sdb] <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1

Sep 26 13:13:54 inetgw1 kernel: [1293013.970176] Modules linked in: btrfs zlib_deflate ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs
reiserfs ext2 ip_vs libcrc32c scsi_dh_rdac xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_c
onntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables ib_iser rdma_cm ib_cm iw_cm i
b_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vesafb dm_round_robin 8021q garp stp ioatdma psmouse dca seri
o_raw dm_multipath joydev i7core_edac edac_core mac_hid lp parport usbhid hid megaraid_sas e1000e [last unloaded: configfs]
Sep 26 13:13:54 inetgw1 kernel: [1293013.970234] CPU 0
Sep 26 13:13:54 inetgw1 kernel: [1293013.970236] Modules linked in: btrfs zlib_deflate ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs
reiserfs ext2 ip_vs libcrc32c scsi_dh_rdac xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_c
onntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables ib_iser rdma_cm ib_cm iw_cm i
b_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vesafb dm_round_robin 8021q garp stp ioatdma psmouse dca seri
o_raw dm_multipath joydev i7core_edac edac_core mac_hid lp parport usbhid hid megaraid_sas e1000e [last unloaded: configfs]
Sep 26 13:13:54 inetgw1 kernel: [1293013.970283]
Sep 26 13:13:54 inetgw1 kernel: [1293013.970286] Pid: 30450, comm: kworker/u:2 Not tainted 3.2.0-30-generic #48-Ubuntu DEPO Computers X8STi/X8
STi
Sep 26 13:13:54 inetgw1 kernel: [1293013.970292] RIP: 0010:[<ffffffff8165a629>] [<ffffffff8165a629>] _raw_spin_unlock_irqrestore+0x19/0x30
Sep 26 13:13:54 inetgw1 kernel: [1293013.970301] RSP: 0018:ffff880113bf1d68 EFLAGS: 00000286
Sep 26 13:13:54 inetgw1 kernel: [1293013.970303] RAX: 0000000000000286 RBX: 0000000000000010 RCX: 0000000000000010
Sep 26 13:13:54 inetgw1 kernel: [1293013.970306] RDX: ffff880309c47418 RSI: 0000000000000286 RDI: 0000000000000286
Sep 26 13:13:54 inetgw1 kernel: [1293013.970309] RBP: ffff880113bf1d70 R08: ffffffff81cddaa0 R09: 0000000000000100
Sep 26 13:13:54 inetgw1 kernel: [1293013.970312] R10: 0000000000000010 R11: 0000000000000000 R12: ffff880306fb92a0
Sep 26 13:13:54 inetgw1 kernel: [1293013.970314] R13: ffff880113bf1d70 R14: ffff880306fb9294 R15: ffff880306fb8eb0
Sep 26 13:13:54 inetgw1 kernel: [1293013.970318] FS: 0000000000000000(0000) GS:ffff88031fc00000(0000) knlGS:0000000000000000
Sep 26 13:13:54 inetgw1 kernel: [1293013.970321] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Sep 26 13:13:54 inetgw1 kernel: [1293013.970323] CR2: 00007fc2c6b46000 CR3: 0000000001c05000 CR4: 00000000000006f0
Sep 26 13:13:54 inetgw1 kernel: [1293013.970326] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 26 13:13:54 inetgw1 kernel: [1293013.970329] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Sep 26 13:13:54 inetgw1 kernel: [1293013.970332] Process kworker/u:2 (pid: 30450, threadinfo ffff880113bf0000, task ffff8802b88f4500)
Sep 26 13:13:54 inetgw1 kernel: [1293013.978403] ffff880309c47400 ffff880113bf1da0 ffffffff8142d1e7 ffff880305c64930
Sep 26 13:13:54 inetgw1 kernel: [1293013.978412] ffff880305c64930 ffff88030ac0b010 ffff88030ac0b000 ffff880113bf1dd0
Sep 26 13:13:54 inetgw1 kernel: [1293013.978419] ffffffff8142f3c9 ffff880305c64930 ffff880305c64800 ffff880305c64818
Sep 26 13:13:54 inetgw1 kernel: [1293013.986383] [<ffffffff8142d1e7>] scsi_target_reap+0x77/0xf0
Sep 26 13:13:54 inetgw1 kernel: [1293013.986389] [<ffffffff8142f3c9>] scsi_remove_target+0xc9/0xe0
Sep 26 13:13:54 inetgw1 kernel: [1293013.986404] [<ffffffffa00f9fec>] __iscsi_unbind_session+0xbc/0x190 [scsi_transport_iscsi]
Sep 26 13:13:54 inetgw1 kernel: [1293013.986413] [<ffffffff81084a6a>] process_one_work+0x11a/0x480
Sep 26 13:13:54 inetgw1 kernel: [1293013.986423] [<ffffffff81085814>] worker_thread+0x164/0x370
Sep 26 13:13:54 inetgw1 kernel: [1293013.986428] [<ffffffff810856b0>] ? manage_workers.isra.30+0x130/0x130
Sep 26 13:13:54 inetgw1 kernel: [1293013.986432] [<ffffffff8108a03c>] kthread+0x8c/0xa0
Sep 26 13:13:54 inetgw1 kernel: [1293013.986438] [<ffffffff81664b74>] kernel_thread_helper+0x4/0x10
Sep 26 13:13:54 inetgw1 kernel: [1293013.986442] [<ffffffff81089fb0>] ? flush_kthread_worker+0xa0/0xa0
Sep 26 13:13:54 inetgw1 kernel: [1293013.986447] [<ffffffff81664b70>] ? gs_change+0x13/0x13
Sep 26 13:13:54 inetgw1 kernel: [1293014.010878] Call Trace:
Sep 26 13:13:54 inetgw1 kernel: [1293014.010884] [<ffffffff8142d1e7>] scsi_target_reap+0x77/0xf0
Sep 26 13:13:54 inetgw1 kernel: [1293014.010888] [<ffffffff8142f3c9>] scsi_remove_target+0xc9/0xe0
Sep 26 13:13:54 inetgw1 kernel: [1293014.010897] [<ffffffffa00f9fec>] __iscsi_unbind_session+0xbc/0x190 [scsi_transport_iscsi]
Sep 26 13:13:54 inetgw1 kernel: [1293014.010902] [<ffffffff81084a6a>] process_one_work+0x11a/0x480
Sep 26 13:13:54 inetgw1 kernel: [1293014.010906] [<ffffffff81085814>] worker_thread+0x164/0x370
Sep 26 13:13:54 inetgw1 kernel: [1293014.010911] [<ffffffff810856b0>] ? manage_workers.isra.30+0x130/0x130
Sep 26 13:13:54 inetgw1 kernel: [1293014.010915] [<ffffffff8108a03c>] kthread+0x8c/0xa0
Sep 26 13:13:54 inetgw1 kernel: [1293014.010919] [<ffffffff81664b74>] kernel_thread_helper+0x4/0x10
Sep 26 13:13:54 inetgw1 kernel: [1293014.010923] [<ffffffff81089fb0>] ? flush_kthread_worker+0xa0/0xa0
Sep 26 13:13:54 inetgw1 kernel: [1293014.010927] [<ffffffff81664b70>] ? gs_change+0x13/0x13
Sep 26 13:13:54 inetgw1 kernel: [1293014.054272] Modules linked in: btrfs zlib_deflate ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs
reiserfs ext2 ip_vs libcrc32c scsi_dh_rdac xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_c
onntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables ib_iser rdma_cm ib_cm iw_cm i
b_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vesafb dm_round_robin 8021q garp stp ioatdma psmouse dca seri
o_raw dm_multipath joydev i7core_edac edac_core mac_hid lp parport usbhid hid megaraid_sas e1000e [last unloaded: configfs]

server console was not available at this time , ssh too, although I can ping server, so I had to reset both servers.

There was no such problem with -29 kernel...

Thank you!

sles (slesru) wrote :

btw, I don't know why it is attached to pwernap, I choosed linux-image-3.2.0-30-generic as package.
I don't know how change this :-(

affects: powernap (Ubuntu) → linux (Ubuntu)

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1056746

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: precise

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.6 kernel[0] (Not a kernel in the daily directory) and install both the linux-image and linux-image-extra .deb packages.

Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. Please only remove that one tag and leave the other tags. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.6-quantal/

Changed in linux (Ubuntu):
importance: Undecided → High
tags: added: kernel-da-key raw-spin-unlock-irqrestore
tags: added: needs-upstream-testing
mynet (technie) wrote :
Download full text (4.7 KiB)

Hello!

I have exactly the same problem.
The Server hangs on iscsiadm logouts.
My ISCSI Target ist a HP P4500

Here is my stack trace:

[ 2360.832232] sd 3:0:0:0: [sdb] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
[ 2360.833438] sd 3:0:0:0: [sdb] Write Protect is off
[ 2360.833445] sd 3:0:0:0: [sdb] Mode Sense: 77 00 00 08
[ 2360.834112] sd 3:0:0:0: [sdb] Write cache: disabled, read cache: disabled, doesn't support DPO or FUA
[ 2360.850500] sdb: sdb1 sdb2 < sdb5 >
[ 2360.852989] sd 3:0:0:0: [sdb] Attached SCSI disk
[ 2403.635452] BUG: soft lockup - CPU#6 stuck for 22s! [kworker/u:0:5]
[ 2403.635805] Modules linked in: ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi usbhid hid veth ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables openvswitch_mod(O) bridge stp kvm_intel kvm nbd vesafb bonding dm_multipath psmouse serio_raw sb_edac edac_core hpilo ioatdma dca acpi_power_meter mac_hid lp parport hpsa tg3 [last unloaded: scsi_transport_iscsi]
[ 2403.635841] CPU 6
[ 2403.635842] Modules linked in: ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi usbhid hid veth ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables openvswitch_mod(O) bridge stp kvm_intel kvm nbd vesafb bonding dm_multipath psmouse serio_raw sb_edac edac_core hpilo ioatdma dca acpi_power_meter mac_hid lp parport hpsa tg3 [last unloaded: scsi_transport_iscsi]
[ 2403.635864]
[ 2403.635866] Pid: 5, comm: kworker/u:0 Tainted: G W O 3.2.0-31-generic #50-Ubuntu HP ProLiant DL360p Gen8
[ 2403.635869] RIP: 0010:[<ffffffff8165a70f>] [<ffffffff8165a70f>] _raw_spin_unlock_irqrestore+0x1f/0x30
[ 2403.635878] RSP: 0018:ffff88080c0e5da0 EFLAGS: 00000286
[ 2403.635880] RAX: 0000000000000286 RBX: 0000000000000286 RCX: 0000000000000001
[ 2403.635881] RDX: ffff880fffbc3c08 RSI: 0000000000000286 RDI: 0000000000000286
[ 2403.635883] RBP: ffff88080c0e5da0 R08: ffffffff81cddaa0 R09: 0000000000000100
[ 2403.635884] R10: 0000000000000040 R11: 0000000000000000 R12: 0000000000000001
[ 2403.635885] R13: 0000000000000001 R14: ffffffff81cddaa0 R15: 0000000000000100
[ 2403.635887] FS: 0000000000000000(0000) GS:ffff88101f200000(0000) knlGS:0000000000000000
[ 2403.635889] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2403.635890] CR2: 00007f43c5c87cd0 CR3: 0000000001c05000 CR4: 00000000000426e0
[ 2403.635892] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2403.635894] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 2403.635896] Process kworker/u:0 (pid: 5, threadinfo ffff88080c0e4000, task ffff88080c0c5c00)
[ 2403.635897] Stack:
[ 2403.635992] ffff88080c0e5dd0 ffffffff8142f3b4 ffff881001bff130 ffff881001bff000
[ 2403.635996] ffff881001bff018 0000000000000000 ffff88080c0e5e00 ffffffffa0106fec
[ 2...

Read more...

tags: added: regression-update
Louis Bouchard (louis) wrote :

@Joe

I'm able to reproduce the issue systematically in a KVM vm so I ran the following tests :

Kernel
3.2.0-31-virtual HANG
3.2.0-29-virtual OK
3.2.28-030228-generic HANG (this is the mainline kernel corresponding to 3.2.0-31-virtual I think)
3.5.0-17-generic OK (Quantal latest)

So looks like the issue is not seen in Quantal but do happen with the mainline kernel.

One important thing is that running 3.2.0-29-virtual DOES WORKAROUND this issue !

summary: - kernel panic on iscsi tagret disconnect
+ kernel panic on iscsi target disconnect

One thing that is important to know is that it will not happen on a system that does not use multipath. On the latest kernel, one workaround is to flush the multipath tables before logging out the iscsi nodes :

 $ sudo invoke-rc.d multipath-tools stop
 $ sudo multipath -F
 $ sudo iscsiadm -m node --logout

This will work and avoid hanging the kernel.

I'm now proceeding to bisect between those two kernel versions

Download full text (10.3 KiB)

This has been our experience as well.

I can verify that flushing the paths using *sudo multipath -F* prior to
logging off the iSCSI nodes would avoid the hang.

However as under normal operations we have LVM volumes on the multipath
device it was necessary to perform a *sudo lvchange -an <path to logical
volume>* to mark the lv as unavailable, otherwise the the multipath -F
command fails sue to an active path.

Nevertheless, if we follow this procedure we can reliably logout of the
iscsi nodes without a hang.

Stephen

On 16 October 2012 15:20, Louis Bouchard <email address hidden>wrote:

> One thing that is important to know is that it will not happen on a
> system that does not use multipath. On the latest kernel, one
> workaround is to flush the multipath tables before logging out the iscsi
> nodes :
>
> $ sudo invoke-rc.d multipath-tools stop
> $ sudo multipath -F
> $ sudo iscsiadm -m node --logout
>
> This will work and avoid hanging the kernel.
>
> I'm now proceeding to bisect between those two kernel versions
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1056746
>
> Title:
> kernel panic on iscsi target disconnect
>
> Status in “linux” package in Ubuntu:
> Incomplete
>
> Bug description:
> Hello!
>
> Just rebooted two identical Ubuntu 12.04.1 LTS servers to set linux-
> image-3.2.0-31-generic
>
> And got the same panic on both. Don't think it is fixed in -31.
>
> Servers are connected to IBM3512 over iscsi with multipath rdac and
> ocfs2.
> Here is log from one of servers:
>
>
>
> Sep 26 13:13:17 inetgw1 kernel: [1292980.531921] o2dlm: Leaving domain
> 68166ECC9E2A48208239B14D8218F0DA
> Sep 26 13:13:18 inetgw1 kernel: [1292981.054231] o2net: No longer
> connected to node inetgw2 (num 1) at 192.168.19.254:7777
> Sep 26 13:13:18 inetgw1 kernel: [1292981.055378] ocfs2: Unmounting
> device (252,6) on (node 0)
> Sep 26 13:13:20 inetgw1 kernel: [1292982.774884] ocfs2: Unregistered
> cluster interface o2cb
> Sep 26 13:13:24 inetgw1 kernel: [1292987.277805] sd 5:0:0:0: rdac:
> Detached
> Sep 26 13:13:24 inetgw1 kernel: [1292987.277882] sd 6:0:0:0: rdac:
> Detached
> Sep 26 13:13:24 inetgw1 kernel: [1292987.279549] sd 5:0:0:0: [sdb]
> Synchronizing SCSI cache
> Sep 26 13:13:24 inetgw1 kernel: [1292987.279808] sd 6:0:0:0: [sdc]
> Synchronizing SCSI cache
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817740] sd 5:0:0:0: [sdb]
> Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817746] sd 5:0:0:0: [sdb]
> Sense Key : Illegal Request [current]
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817750] sd 5:0:0:0: [sdb]
> <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
>
> Sep 26 13:13:54 inetgw1 kernel: [1293013.970176] Modules linked in:
> btrfs zlib_deflate ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs
> reiserfs ext2 ip_vs libcrc32c scsi_dh_rdac xt_conntrack nf_conntrack_ftp
> xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_c
> onntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT
> xt_tcpudp iptable_filter ip_tables x_tables ib_iser rdm...

Download full text (8.3 KiB)

I had no chance to test mailine yet, but -31 and -32 has the same issue:

ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs]
[ 463.624735] CPU 1
[ 463.624737] Modules linked in: scsi_dh_rdac ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs]
[ 463.624775]
[ 463.624778] Pid: 83, comm: kworker/u:2 Not tainted 3.2.0-32-generic #51-Ubuntu DEPO Computers X8STi/X8STi
[ 463.624784] RIP: 0010:[<ffffffff8142f6b0>] [<ffffffff8142f6b0>] __scsi_remove_target+0x20/0xf0
[ 463.624792] RSP: 0018:ffff880307067d80 EFLAGS: 00000246
[ 463.624795] RAX: 0000000000000000 RBX: ffffffff81cddaa0 RCX: 0000000000000010
[ 463.624798] RDX: ffff8802cc1a9008 RSI: 0000000000000286 RDI: ffff88030a75d930
[ 463.624800] RBP: ffff880307067da0 R08: ffffffff81cddaa0 R09: 0000000000000100
[ 463.624803] R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000010
[ 463.624806] R13: 0000000000000000 R14: 0000000000000010 R15: ffff880307067d70
[ 463.624809] FS: 0000000000000000(0000) GS:ffff88031fc20000(0000) knlGS:0000000000000000
[ 463.624812] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 463.624815] CR2: 00007f94fc247000 CR3: 0000000001c05000 CR4: 00000000000006e0
[ 463.624818] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 463.624821] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 463.624824] Process kworker/u:2 (pid: 83, threadinfo ffff880307066000, task ffff880307032e00)
[ 463.624826] Stack:
[ 463.652982] BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u:3:86]
[ 463.652985] Modules linked in: scsi_dh_rdac ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables scsi_transport_iscsi vesafb dm_round_robin lp ioatdma joydev dm_multipath parport psmouse serio_raw dca i7core_edac mac_hid edac_core 8021q garp stp usbhid hid megaraid_sas e1000e [last unloaded: configfs]
[ 463.653018] CPU 2
[ 463.653019] Modules linked in: scsi_dh_rdac ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi ip_vs libcrc32c xt_conntrack nf_conntrack_ftp xt_NOTRACK iptable_raw xt_mark iptable_mangle ...

Read more...

Louis Bouchard (louis) wrote :

I must correct one affirmation I made earlier. The issue is also visible on latest Quantal kernel, so the table should read :

Kernel
3.2.0-31-virtual HANG
3.2.0-29-virtual OK
3.2.28-030228-generic HANG (this is the mainline kernel corresponding to 3.2.0-31-virtual I think)
3.5.0-17-generic HANG (Quantal latest)

Kernel bisection outlines the following commit as the first bad :

$ git bisect good [0/1917]
1675b80790e5e373535d87b5ffbfafe7e532b9b9 is the first bad commit
commit 1675b80790e5e373535d87b5ffbfafe7e532b9b9
Author: Dan Williams <email address hidden>
Date: Thu Jun 21 23:47:28 2012 -0700

    fix hot unplug vs async scan race

    BugLink: http://bugs.launchpad.net/bugs/1032640

    commit 3b661a92e869ebe2358de8f4b3230ad84f7fce51 upstream.

    The following crash results from cases where the end_device has been
    removed before scsi_sysfs_add_sdev has had a chance to run.

     BUG: unable to handle kernel NULL pointer dereference at 0000000000000098
     IP: [<ffffffff8115e100>] sysfs_create_dir+0x32/0xb6
     ...
     Call Trace:
      [<ffffffff8125e4a8>] kobject_add_internal+0x120/0x1e3
      [<ffffffff81075149>] ? trace_hardirqs_on+0xd/0xf
      [<ffffffff8125e641>] kobject_add_varg+0x41/0x50
      [<ffffffff8125e70b>] kobject_add+0x64/0x66
      [<ffffffff8131122b>] device_add+0x12d/0x63a
      [<ffffffff814b65ea>] ? _raw_spin_unlock_irqrestore+0x47/0x56
      [<ffffffff8107de15>] ? module_refcount+0x89/0xa0
      [<ffffffff8132f348>] scsi_sysfs_add_sdev+0x4e/0x28a
      [<ffffffff8132dcbb>] do_scan_async+0x9c/0x145

    ...teach scsi_sysfs_add_devices() to check for deleted devices() before
    trying to add them, and teach scsi_remove_target() how to remove targets
    that have not been added via device_add().

    Reported-by: Dariusz Majchrzak <email address hidden>
    Signed-off-by: Dan Williams <email address hidden>
    Signed-off-by: James Bottomley <email address hidden>
    Signed-off-by: Ben Hutchings <email address hidden>
    Signed-off-by: Herton Ronaldo Krzesinski <email address hidden>

:040000 040000 8d739d0b793a09849cc280e781cb3613391ed8f9 ff60255ac7ac2eced39510bec9caf066b431ce7a M drivers

Louis Bouchard (louis) on 2012-10-18
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
assignee: nobody → Louis Bouchard (louis-bouchard)
sles (slesru) wrote :

OK, I added this

   #sort of workaround
        /usr/sbin/invoke-rc.d multipath-tools stop
        /sbin/multipath -F

to stop section in open-iscsi
and
  /sbin/multipath -F

to multipath-tools after daemon stop.

tested on one of two servers, reboot is OK :-)

Download full text (9.9 KiB)

Thanks for th update louis, I'm a bit busy until next Tuesday but I will
test this in our configuration.

Does that multipath-tools stop have the same effect as the lvchange -an
command I was using to ensure that there were no active paths preventing
multipath -F from completing successfully?

Stephen

On 18 October 2012 09:58, sles <email address hidden> wrote:

> OK, I added this
>
> #sort of workaround
> /usr/sbin/invoke-rc.d multipath-tools stop
> /sbin/multipath -F
>
> to stop section in open-iscsi
> and
> /sbin/multipath -F
>
> to multipath-tools after daemon stop.
>
> tested on one of two servers, reboot is OK :-)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1056746
>
> Title:
> kernel panic on iscsi target disconnect
>
> Status in “linux” package in Ubuntu:
> Confirmed
>
> Bug description:
> Hello!
>
> Just rebooted two identical Ubuntu 12.04.1 LTS servers to set linux-
> image-3.2.0-31-generic
>
> And got the same panic on both. Don't think it is fixed in -31.
>
> Servers are connected to IBM3512 over iscsi with multipath rdac and
> ocfs2.
> Here is log from one of servers:
>
>
>
> Sep 26 13:13:17 inetgw1 kernel: [1292980.531921] o2dlm: Leaving domain
> 68166ECC9E2A48208239B14D8218F0DA
> Sep 26 13:13:18 inetgw1 kernel: [1292981.054231] o2net: No longer
> connected to node inetgw2 (num 1) at 192.168.19.254:7777
> Sep 26 13:13:18 inetgw1 kernel: [1292981.055378] ocfs2: Unmounting
> device (252,6) on (node 0)
> Sep 26 13:13:20 inetgw1 kernel: [1292982.774884] ocfs2: Unregistered
> cluster interface o2cb
> Sep 26 13:13:24 inetgw1 kernel: [1292987.277805] sd 5:0:0:0: rdac:
> Detached
> Sep 26 13:13:24 inetgw1 kernel: [1292987.277882] sd 6:0:0:0: rdac:
> Detached
> Sep 26 13:13:24 inetgw1 kernel: [1292987.279549] sd 5:0:0:0: [sdb]
> Synchronizing SCSI cache
> Sep 26 13:13:24 inetgw1 kernel: [1292987.279808] sd 6:0:0:0: [sdc]
> Synchronizing SCSI cache
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817740] sd 5:0:0:0: [sdb]
> Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817746] sd 5:0:0:0: [sdb]
> Sense Key : Illegal Request [current]
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817750] sd 5:0:0:0: [sdb]
> <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
>
> Sep 26 13:13:54 inetgw1 kernel: [1293013.970176] Modules linked in:
> btrfs zlib_deflate ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs
> reiserfs ext2 ip_vs libcrc32c scsi_dh_rdac xt_conntrack nf_conntrack_ftp
> xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_c
> onntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT
> xt_tcpudp iptable_filter ip_tables x_tables ib_iser rdma_cm ib_cm iw_cm i
> b_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
> scsi_transport_iscsi vesafb dm_round_robin 8021q garp stp ioatdma psmouse
> dca seri
> o_raw dm_multipath joydev i7core_edac edac_core mac_hid lp parport
> usbhid hid megaraid_sas e1000e [last unloaded: configfs]
> Sep 26 13:13:54 inetgw1 kernel: [1293013.970234] CPU 0
> Sep 26 13:13:54 inetgw1 ke...

Louis Bouchard (louis) on 2012-10-18
summary: - kernel panic on iscsi target disconnect
+ kernel thread hang on iscsi target disconnect when multipath is active

Hi!

The multipath -F workaround does not really work in my enviroment, beacause i use openstack nova which does not
use this command. so if i terminate a nova instance, nova tries to logout and the whole compute-node hangs after some time :(

Alex

Louis Bouchard (louis) wrote :

FYI, turns out, this is fixed in upstream 3.7 :

commit bc3f02a795d3b4faa99d37390174be2a75d091bd
Author: Dan Williams <email address hidden>
Date: Tue Aug 28 22:12:10 2012 -0700

    [SCSI] scsi_remove_target: fix softlockup regression on hot remove

    John reports:
     BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u:8:2202]
     [..]
     Call Trace:
      [<ffffffff8141782a>] scsi_remove_target+0xda/0x1f0
      [<ffffffff81421de5>] sas_rphy_remove+0x55/0x60
      [<ffffffff81421e01>] sas_rphy_delete+0x11/0x20
      [<ffffffff81421e35>] sas_port_delete+0x25/0x160
      [<ffffffff814549a3>] mptsas_del_end_device+0x183/0x270

    ...introduced by commit 3b661a9 "[SCSI] fix hot unplug vs async scan race".

    Don't restart lookup of more stargets in the multi-target case, just
    arrange to traverse the list once, on the assumption that new targets
    are always added at the end. There is no guarantee that the target will
    change state in scsi_target_reap() so we can end up spinning if we
    restart.

Louis Bouchard (louis) on 2012-10-18
Changed in linux (Ubuntu):
status: Confirmed → In Progress
Louis Bouchard (louis) wrote :

Cherry-pick of the commit has been tested on the Quantal & Precise kernels & submitted to the Ubuntu kernel ML.

summary: - kernel thread hang on iscsi target disconnect when multipath is active
+ kernel thread hang on iscsi target disconnect
Andy Whitcroft (apw) wrote :

This is fixed in upstream so marking raring fix committed as we have a 3.7 snapshot in our tree.

Changed in linux (Ubuntu Precise):
assignee: nobody → Louis Bouchard (louis-bouchard)
Changed in linux (Ubuntu Quantal):
assignee: nobody → Louis Bouchard (louis-bouchard)
importance: Undecided → High
Changed in linux (Ubuntu Precise):
importance: Undecided → High
status: New → In Progress
Changed in linux (Ubuntu Quantal):
status: New → In Progress
Changed in linux (Ubuntu):
status: In Progress → Fix Committed
Tim Gardner (timg-tpi) on 2012-10-19
Changed in linux (Ubuntu Precise):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Quantal):
status: In Progress → Fix Released
Tais P. Hansen (taisph) wrote :

This is still an issue in with linux-image-3.2.0-33-generic (3.2.0-33.52) in precise-proposed.

Louis Bouchard (louis) wrote :

I have tested the latest PPA pre-proposed kernel that contains the fix. The tested version is :

 - linux - 3.2.0-33.52pre201210190400

So unless I'm mistaken, the fix is queued up for the next -proposed package.

Tais P. Hansen (taisph) wrote :

@Louis

I've tested the ppa you pointed to on one affected host and it seems like it fixes the problem. Thanks.

... I guess it's not my day as it apparently masked another problem. Back to debugging.

sles (slesru) wrote :

Hello!

Is there any chance to get fix in 12.04 updates ? :-)

Thank you!

Louis Bouchard (louis) wrote :

@sles

It will make it to the -updates pocket eventually. It's not in the latest 3.2.0-33.52 kernel currently in -updates, but since the commit is in the ubuntu-precise git tree and made it to the PPA, it will appear in -updates in the near future.

I'll put a note here once it gets there.

Download full text (9.2 KiB)

@sles

FWIW The pre-proposed package (linux-image-3.2.0-34-generic) listed earlier
is working well in our environment on 12.04.

On Thu, Nov 15, 2012 at 9:20 AM, Louis Bouchard <
<email address hidden>> wrote:

> @sles
>
> It will make it to the -updates pocket eventually. It's not in the
> latest 3.2.0-33.52 kernel currently in -updates, but since the commit is
> in the ubuntu-precise git tree and made it to the PPA, it will appear in
> -updates in the near future.
>
> I'll put a note here once it gets there.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1056746
>
> Title:
> kernel thread hang on iscsi target disconnect
>
> Status in “linux” package in Ubuntu:
> Fix Committed
> Status in “linux” source package in Precise:
> Fix Committed
> Status in “linux” source package in Quantal:
> Fix Released
>
> Bug description:
> Hello!
>
> Just rebooted two identical Ubuntu 12.04.1 LTS servers to set linux-
> image-3.2.0-31-generic
>
> And got the same panic on both. Don't think it is fixed in -31.
>
> Servers are connected to IBM3512 over iscsi with multipath rdac and
> ocfs2.
> Here is log from one of servers:
>
>
>
> Sep 26 13:13:17 inetgw1 kernel: [1292980.531921] o2dlm: Leaving domain
> 68166ECC9E2A48208239B14D8218F0DA
> Sep 26 13:13:18 inetgw1 kernel: [1292981.054231] o2net: No longer
> connected to node inetgw2 (num 1) at 192.168.19.254:7777
> Sep 26 13:13:18 inetgw1 kernel: [1292981.055378] ocfs2: Unmounting
> device (252,6) on (node 0)
> Sep 26 13:13:20 inetgw1 kernel: [1292982.774884] ocfs2: Unregistered
> cluster interface o2cb
> Sep 26 13:13:24 inetgw1 kernel: [1292987.277805] sd 5:0:0:0: rdac:
> Detached
> Sep 26 13:13:24 inetgw1 kernel: [1292987.277882] sd 6:0:0:0: rdac:
> Detached
> Sep 26 13:13:24 inetgw1 kernel: [1292987.279549] sd 5:0:0:0: [sdb]
> Synchronizing SCSI cache
> Sep 26 13:13:24 inetgw1 kernel: [1292987.279808] sd 6:0:0:0: [sdc]
> Synchronizing SCSI cache
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817740] sd 5:0:0:0: [sdb]
> Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817746] sd 5:0:0:0: [sdb]
> Sense Key : Illegal Request [current]
> Sep 26 13:13:26 inetgw1 kernel: [1292988.817750] sd 5:0:0:0: [sdb]
> <<vendor>> ASC=0x94 ASCQ=0x1ASC=0x94 ASCQ=0x1
>
> Sep 26 13:13:54 inetgw1 kernel: [1293013.970176] Modules linked in:
> btrfs zlib_deflate ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs
> reiserfs ext2 ip_vs libcrc32c scsi_dh_rdac xt_conntrack nf_conntrack_ftp
> xt_NOTRACK iptable_raw xt_mark iptable_mangle iptable_nat nf_nat nf_c
> onntrack_ipv4 nf_conntrack nf_defrag_ipv4 ts_bm xt_string ipt_REJECT
> xt_tcpudp iptable_filter ip_tables x_tables ib_iser rdma_cm ib_cm iw_cm i
> b_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi
> scsi_transport_iscsi vesafb dm_round_robin 8021q garp stp ioatdma psmouse
> dca seri
> o_raw dm_multipath joydev i7core_edac edac_core mac_hid lp parport
> usbhid hid megaraid_sas e1000e [last unloaded: configfs]
> Sep 26 13:13:54 inetgw1 kernel: [1293013.970234] CPU 0
> Sep 26 13:13:54 inetg...

Read more...

Herton R. Krzesinski (herton) wrote :

The commit for this issue came in via a stable upstream release (linux 3.2.32). As such it is not subject to the standard bug verification process.

tags: added: verification-done-precise
Louis Bouchard (louis) wrote :

As a reference to everyone, the current kernel in the -proposed pocket does contain the fix for this issue. Its version is 3.2.0-34-generic #53-Ubuntu.

This should become available publicly in -updates in the upcoming week.

cyberfarer (cyberfarer) wrote :

Hi

Just to add a twist. I am able to confirm that I could repeat this problem. The thing is, I found I didn't get the error until I added an entry for the iSCSI volume in fstab. And in fact, after removing the entry from fstab and restarting, I was able to again restart without an error.

cyberfarer (cyberfarer) wrote :

Ok, so I had not mounted the volume before restarting. So the issue isn't fstab, it is when the volume is mounted.

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :
Download full text (19.5 KiB)

This bug was fixed in the package linux - 3.2.0-34.53

---------------
linux (3.2.0-34.53) precise-proposed; urgency=low

  [Luis Henriques]

  * Release Tracking Bug
    - LP: #1078760

  [ Kamal Mostafa ]

  * SAUCE: input: Cypress PS/2 Trackpad list additional contributors

  [ Kyle Fazzari ]

  * SAUCE: input: Cypress PS/2 Trackpad fix multi-source, double-click
    - LP: #1055788
  * SAUCE: input: Cypress PS/2 Trackpad fix lost sync upon palm contact
    - LP: #1048258
  * SAUCE: input: Cypress PS/2 Trackpad fix taps turning into hardware
    clicks
    - LP: #1064086

  [ Tim Gardner ]

  * [Config] remove ndiswrapper from Provides:
    - LP: #1076395
  * [Config] CONFIG_HP_WATCHDOG=m for x86en
    - LP: #1076342

  [ Upstream Kernel Changes ]

  * Revert "SUNRPC: Ensure we close the socket on EPIPE errors too..."
    - LP: #1075355
  * Revert "ath9k_hw: Updated AR9003 tx gain table for 5GHz"
    - LP: #1075355
  * eCryptfs: check for eCryptfs cipher support at mount
    - LP: #338914
  * isci: fix isci_pci_probe() generates warning on efi failure path
    - LP: #1068162
  * mtd: nand: Use the mirror BBT descriptor when reading its version
    - LP: #1068162
  * drm/i915: prevent possible pin leak on error path
    - LP: #1068162
  * workqueue: add missing smp_wmb() in process_one_work()
    - LP: #1068162
  * TTY: ttyprintk, don't touch behind tty->write_buf
    - LP: #1068162
  * Remove BUG_ON from n_tty_read()
    - LP: #1068162
  * n_gsm.c: Implement 3GPP27.010 DLC start-up procedure in MUX
    - LP: #1068162
  * n_gsm: uplink SKBs accumulate on list
    - LP: #1068162
  * n_gsm : Flow control handling in Mux driver
    - LP: #1068162
  * char: n_gsm: remove message filtering for contipated DLCI
    - LP: #1068162
  * n_gsm: added interlocking for gsm_data_lock for certain code paths
    - LP: #1068162
  * n_gsm: avoid accessing freed memory during CMD_FCOFF condition
    - LP: #1068162
  * n_gsm: replace kfree_skb w/ appropriate dev_* versions
    - LP: #1068162
  * n_gsm: memory leak in uplink error path
    - LP: #1068162
  * UBI: fix autoresize handling in R/O mode
    - LP: #1068162
  * UBI: erase free PEB with bitflip in EC header
    - LP: #1068162
  * firmware: Add missing attributes to EFI variable attribute print out
    from sysfs
    - LP: #1068162
  * tools/hv: Fix exit() error code
    - LP: #1068162
  * slab: fix the DEADLOCK issue on l3 alien lock
    - LP: #1068162
  * gspca_pac7302: Add usb-id for 145f:013c
    - LP: #1068162
  * gspca_pac7302: add support for device 1ae7:2001 Speedlink Snappy
    Microphone SL-6825-SBK
    - LP: #1068162
  * xhci: Warn when hosts don't halt.
    - LP: #1068162
  * xHCI: add cmd_ring_state
    - LP: #1068162
  * xHCI: add aborting command ring function
    - LP: #1068162
  * xHCI: cancel command after command timeout
    - LP: #1068162
  * hpsa: Use LUN reset instead of target reset
    - LP: #1068162
  * rc: ite-cir: Initialise ite_dev::rdev earlier
    - LP: #1068162
  * staging: speakup_soft: Fix reading of init string
    - LP: #1068162
  * target: fix return code in target_core_init_configfs error path
    - LP: #1068162
  * powerpc/eeh: Lock module while handling EEH event
  ...

Changed in linux (Ubuntu Precise):
status: Fix Committed → Fix Released
Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
Marcus Klein (kleini76) wrote :
Download full text (5.3 KiB)

The same issue happens for me with the linux-image-server:

Dec 7 12:04:24 node1 multipathd: sdc: remove path (uevent)
Dec 7 12:04:24 node1 kernel: [65815.457105] sd 1:0:0:1: [sdc] Synchronizing SCSI cache
Dec 7 12:04:24 node1 kernel: [65815.458094] sd 1:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 7 12:04:24 node1 kernel: [65815.458101] sd 1:0:0:1: [sdc] Sense Key : Illegal Request [current]
Dec 7 12:04:24 node1 kernel: [65815.458106] sd 1:0:0:1: [sdc] Add. Sense: Logical unit not supported
Dec 7 12:04:48 node1 kernel: [65839.483571] BUG: soft lockup - CPU#10 stuck for 22s! [kworker/u:1:8014]
Dec 7 12:04:48 node1 kernel: [65839.486057] Modules linked in: dm_snapshot ipt_LOG ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vesafb xfs joydev i7core_edac edac_core ioatdma bridge stp tpm_tis mac_hid lp parport dm_round_robin usbhid hid mptsas mptscsih mptbase igb scsi_transport_sas dca dm_multipath [last unloaded: scsi_wait_scan]
Dec 7 12:04:48 node1 kernel: [65839.486109] CPU 10
Dec 7 12:04:48 node1 kernel: [65839.486110] Modules linked in: dm_snapshot ipt_LOG ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables kvm_intel kvm ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vesafb xfs joydev i7core_edac edac_core ioatdma bridge stp tpm_tis mac_hid lp parport dm_round_robin usbhid hid mptsas mptscsih mptbase igb scsi_transport_sas dca dm_multipath [last unloaded: scsi_wait_scan]
Dec 7 12:04:48 node1 kernel: [65839.486151]
Dec 7 12:04:48 node1 kernel: [65839.486154] Pid: 8014, comm: kworker/u:1 Not tainted 3.2.0-32-generic #51-Ubuntu INTEL MFS5520VI Compute Module/MFS5520VI Compute Module
Dec 7 12:04:48 node1 kernel: [65839.486160] RIP: 0010:[<ffffffff8165b049>] [<ffffffff8165b049>] _raw_spin_unlock_irqrestore+0x19/0x30
Dec 7 12:04:48 node1 kernel: [65839.486170] RSP: 0018:ffff880c44a73d98 EFLAGS: 00000286
Dec 7 12:04:48 node1 kernel: [65839.486172] RAX: 0000000000000286 RBX: 0000000000000018 RCX: 0000000000000018
Dec 7 12:04:48 node1 kernel: [65839.486175] RDX: ffff8806433e0408 RSI: 0000000000000286 RDI: 0000000000000286
Dec 7 12:04:48 node1 kernel: [65839.486178] RBP: ffff880c44a73da0 R08: ffffffff81cddaa0 R09: 0000000000000100
Dec 7 12:04:48 node1 kernel: [65839.486181] R10: 0000000000000018 R11: 0000000000000001 R12: ffffffff81cddaa0
Dec 7 12:04:48 node1 kernel: [65839.486183] R13: 0000000000000100 R14: 0000000000000018 R15: 0000000000000001
Dec 7 12:04:48 node1 kernel: [65839.486187] FS: 0000000000000000(0000) GS:ffff880657ca0000(0000) knlGS:0000000000000000
Dec 7 12:04:48 node1 kernel: [65839.486190] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Dec 7 12:04:48 node1 kernel: [6...

Read more...

Brian Candler (b-candler) wrote :

> This kernel is running here:
>
> root@node1:~# dpkg -s linux-image-server
> ...
> Version: 3.2.0.34.37

Just to eliminate the obvious, does "uname -a" agree on the kernel version?

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux-meta (Ubuntu Precise):
status: New → Confirmed
Changed in linux-meta (Ubuntu Quantal):
status: New → Confirmed
Changed in linux-meta (Ubuntu):
status: New → Confirmed
no longer affects: linux-meta (Ubuntu)
no longer affects: linux-meta (Ubuntu Precise)
no longer affects: linux-meta (Ubuntu Quantal)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers