Ubuntu
linux package

Comment 3 for bug 1670490

Revision history for this message

Michael Hohnbaum (hohnbaum) wrote on 2017-03-06: Re: [Bug 1670490] [NEW] FC Adapter (LPe32000-based) prints "iotag out of range", goes offline, and delays boot a lot (Ubuntu17.04/Emulex/lpfc))

Leann,

A few driver patches for the Kernel team to look at.

Michael

On 03/06/2017 01:10 PM, Launchpad Bug Tracker wrote:
> bugproxy (bugproxy) has assigned this bug to you for Ubuntu:
>
> ---Problem Description---
> FC Adapter goes offline and produces the call traces while booting into OS, on assigning the LUNs to it.
>
> ---uname output---
> Linux ltciofvtr-firestone1 4.9.0-12-generic #13-Ubuntu SMP Tue Jan 10 12:52:39 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
>
> FC Redfish Adapter(32 GB) is going offline on assigning the Luns to it. Adapter shows online till pettitboot and even Luns are also vissible in pettitboot.
> once selecting the OS from pettitboot and booting into it, it produces call traces and also few lpfc errors. attached the comple console logs FYR.
> The FC switch to which the Redfish adapter is connected is a 16 GB switch and the adapter is of 32 GB. According to me this parameter should not have any concern as the adapter has to support backward compatibility.
>
> ---Steps to Reproduce---
> 1. install the adapter in a server. and connect it to FC switch (16 GB)
> 2. create zone and assign the LUNS to it from V7000.
> 3. reboot the OS
>
>
> root@ltciofvtr-firestone1:~# ll /sys/class/fc_host/
> total 0
> drwxr-xr-x 2 root root 0 Jan 25 04:22 ./
> drwxr-xr-x 72 root root 0 Jan 25 03:50 ../
> lrwxrwxrwx 1 root root 0 Jan 25 04:13 host10 -> ../../devices/pci0001:00/0001:00:00.0/0001:01:00.1/host10/fc_host/host10/
> lrwxrwxrwx 1 root root 0 Jan 25 04:13 host6 -> ../../devices/pci0000:00/0000:00:00.0/0000:01:00.0/host6/fc_host/host6/
> lrwxrwxrwx 1 root root 0 Jan 25 04:13 host8 -> ../../devices/pci0000:00/0000:00:00.0/0000:01:00.1/host8/fc_host/host8/
> lrwxrwxrwx 1 root root 0 Jan 25 04:13 host9 -> ../../devices/pci0001:00/0001:00:00.0/0001:01:00.0/host9/fc_host/host9/
> root@ltciofvtr-firestone1:~#
> root@ltciofvtr-firestone1:~# cat /sys/class/fc_host/host6/port_state
> Offline
> root@ltciofvtr-firestone1:~# cat /sys/class/fc_host/host8/port_state
> Offline
> root@ltciofvtr-firestone1:~# cat /sys/class/fc_host/host9/port_state
> Online
> root@ltciofvtr-firestone1:~# cat /sys/class/fc_host/host10/port_state
> Online
> root@ltciofvtr-firestone1:~#
> root@ltciofvtr-firestone1:~# lspci -nn | grep -i fibre
> 0000:01:00.0 Fibre Channel [0c04]: Emulex Corporation Lancer Gen6: LPe32000 Fibre Channel Host Adapter [10df:e300] (rev 01)
> 0000:01:00.1 Fibre Channel [0c04]: Emulex Corporation Lancer Gen6: LPe32000 Fibre Channel Host Adapter [10df:e300] (rev 01)
> 0001:01:00.0 Fibre Channel [0c04]: Emulex Corporation Lancer-X: LightPulse Fibre Channel Host Adapter [10df:e200] (rev 10)
> 0001:01:00.1 Fibre Channel [0c04]: Emulex Corporation Lancer-X: LightPulse Fibre Channel Host Adapter [10df:e200] (rev 10)
> root@ltciofvtr-firestone1:~#
>
>
> Device driver error code:
> [ 537.317563] lpfc 0000:01:00.1: 1:0338 IOCB wait timeout error - no wake response Data x3c
> [ 537.317755] lpfc 0000:01:00.1: 1:(0):0727 TMF FCP_LUN_RESET to TGT 1 LUN 0 failed (0, 0) iocb_flag x206
> [ 537.317934] lpfc 0000:01:00.1: 1:(0):0713 SCSI layer issued Device Reset (1, 0) return x2007
> [ 537.318005] lpfc 0000:01:00.1: 1:0372 iotag x0 is out off range: max iotag (x880)
> [ 551.653563] lpfc 0000:01:00.0: 0:(0):0748 abort handler timed out waiting for abortng I/O (xri:x149) to complete: ret 0x2003, ID 1, LUN 1
> [ 551.653795] lpfc 0000:01:00.0: 0:0372 iotag x0 is out off range: max iotag (x880)
> [ 598.757557] lpfc 0000:01:00.1: 1:0338 IOCB wait timeout error - no wake response Data x3c
> [ 598.757766] lpfc 0000:01:00.1: 1:(0):0727 TMF FCP_LUN_RESET to TGT 1 LUN 1 failed (0, 0) iocb_flag x206
> [ 598.757946] lpfc 0000:01:00.1: 1:(0):0713 SCSI layer issued Device Reset (1, 1) return x2007
> [ 598.758017] lpfc 0000:01:00.1: 1:0372 iotag x0 is out off range: max iotag (x880)
> [ 613.093562] lpfc 0000:01:00.0: 0:(0):0748 abort handler timed out waiting for abortng I/O (xri:x14f) to complete: ret 0x2003, ID 1, LUN 0
> [ 613.093630] INFO: task systemd-udevd:1148 blocked for more than 120 seconds.
> [ 613.093631] Not tainted 4.9.0-12-generic #13-Ubuntu
> [ 613.093631] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 613.093632] systemd-udevd D 0 1148 1141 0x00040000
>
>
> Stack trace output:
> [ 613.093633] Call Trace:
> [ 613.093634] [c000001fd6b5b360] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
> [ 613.093636] [c000001fd6b5b530] [c00000000001c3a0] __switch_to+0x2e0/0x4c0
> [ 613.093637] [c000001fd6b5b590] [c000000000b188d8] __schedule+0x2f8/0x990
> [ 613.093638] [c000001fd6b5b670] [c000000000b18fb8] schedule+0x48/0xc0
> [ 613.093640] [c000001fd6b5b6a0] [c000000000b1d394] schedule_timeout+0x274/0x470
> [ 613.093641] [c000001fd6b5b790] [c000000000b19f8c] wait_for_common+0xec/0x240
> [ 613.093642] [c000001fd6b5b810] [c0000000000ea27c] flush_work+0x12c/0x270
> [ 613.093643] [c000001fd6b5b8a0] [c0000000000eca20] __cancel_work_timer+0xc0/0x220
> [ 613.093645] [c000001fd6b5b940] [c00000000059966c] disk_block_events+0xcc/0xe0
> [ 613.093646] [c000001fd6b5b990] [c00000000037124c] __blkdev_get+0x9c/0x490
> [ 613.093648] [c000001fd6b5ba00] [c000000000372830] blkdev_get+0x1a0/0x4a0
> [ 613.093649] [c000001fd6b5bab0] [c0000000003167e0] do_dentry_open+0x2d0/0x470
> [ 613.093651] [c000001fd6b5bb10] [c00000000032fee4] do_last+0x614/0x1070
> [ 613.093652] [c000001fd6b5bc00] [c000000000330a1c] path_openat+0xdc/0x480
> [ 613.093654] [c000001fd6b5bc80] [c00000000033268c] do_filp_open+0xec/0x160
> [ 613.093655] [c000001fd6b5bdb0] [c00000000031841c] do_sys_open+0x1cc/0x380
> [ 613.093656] [c000001fd6b5be30] [c00000000000bd84] system_call+0x38/0xe0
> [ 613.093657] INFO: task systemd-udevd:1155 blocked for more than 120 seconds.
> [ 613.093658] Not tainted 4.9.0-12-generic #13-Ubuntu
> [ 613.093658] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 613.093659] systemd-udevd D 0 1155 1141 0x00040002
>
> Hi Canonical,
>
> Can you please include these 2 commits to the lpfc driver on 17.04 and 16.04 HWE ?
> They've just been applied on mainline as of 4.11-rc1.
>
> The first resolves this problem, and the second prevents cache/DMA
> consistency problems which is likely to be hit in the future with this
> higher-performance adapter.
>
> I already asked for both patches to be flagged for stable kernels.
>
> [1] 8ea73db486cda442f0671f4bc9c03a76be398a28 lpfc: Correct WQ creation for pagesize
> [2] 6b3b3bdb83b4ad51252d21bb13596db879e51850 lpfc: Add missing memory barrier
>
> Thank you.
>
> [1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/drivers/scsi/lpfc?id=8ea73db486cda442f0671f4bc9c03a76be398a28
> [2] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/drivers/scsi/lpfc?id=6b3b3bdb83b4ad51252d21bb13596db879e51850
>
> ** Affects: ubuntu
> Importance: Undecided
> Assignee: Taco Screen team (taco-screen-team)
> Status: New
>
>
> ** Tags: architecture-ppc64le bugnameltc-150924 severity-high targetmilestone-inin1704

--
Michael Hohnbaum
OIL Program Manager
Power (ppc64el) Development Project Manager
Canonical, Ltd.

Leann,

A few driver patches for the Kernel team to look at.

Michael

On 03/06/2017 01:10 PM, Launchpad Bug Tracker wrote:
> bugproxy (bugproxy) has assigned this bug to you for Ubuntu:
>
> ---Problem Description---
> FC Adapter goes offline and produces the call traces while booting into OS, on assigning the LUNs to it.
>   
> ---uname output---
> Linux ltciofvtr-firestone1 4.9.0-12-generic #13-Ubuntu SMP Tue Jan 10 12:52:39 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux
>   
> FC Redfish Adapter(32 GB) is going offline on assigning the Luns to it. Adapter shows online till pettitboot and even Luns are also vissible in pettitboot.
> once selecting the OS from pettitboot and booting into it, it produces call traces and also few lpfc errors. attached the comple console logs FYR.
> The FC switch to which the Redfish adapter is connected is a 16 GB switch and the adapter is of 32 GB. According to me this parameter should not have any concern as the adapter has to support backward compatibility.
>
> ---Steps to Reproduce---
> 1. install the adapter in a server. and connect it to FC switch (16 GB)
> 2. create zone and assign the LUNS to it from V7000.
> 3. reboot the OS
>
>
> root@ltciofvtr-firestone1:~# ll /sys/class/fc_host/
> total 0
> drwxr-xr-x  2 root root 0 Jan 25 04:22 ./
> drwxr-xr-x 72 root root 0 Jan 25 03:50 ../
> lrwxrwxrwx  1 root root 0 Jan 25 04:13 host10 -> ../../devices/pci0001:00/0001:00:00.0/0001:01:00.1/host10/fc_host/host10/
> lrwxrwxrwx  1 root root 0 Jan 25 04:13 host6 -> ../../devices/pci0000:00/0000:00:00.0/0000:01:00.0/host6/fc_host/host6/
> lrwxrwxrwx  1 root root 0 Jan 25 04:13 host8 -> ../../devices/pci0000:00/0000:00:00.0/0000:01:00.1/host8/fc_host/host8/
> lrwxrwxrwx  1 root root 0 Jan 25 04:13 host9 -> ../../devices/pci0001:00/0001:00:00.0/0001:01:00.0/host9/fc_host/host9/
> root@ltciofvtr-firestone1:~#
> root@ltciofvtr-firestone1:~# cat /sys/class/fc_host/host6/port_state
> Offline
> root@ltciofvtr-firestone1:~# cat /sys/class/fc_host/host8/port_state
> Offline
> root@ltciofvtr-firestone1:~# cat /sys/class/fc_host/host9/port_state
> Online
> root@ltciofvtr-firestone1:~# cat /sys/class/fc_host/host10/port_state
> Online
> root@ltciofvtr-firestone1:~#
> root@ltciofvtr-firestone1:~# lspci -nn | grep -i fibre
> 0000:01:00.0 Fibre Channel [0c04]: Emulex Corporation Lancer Gen6: LPe32000 Fibre Channel Host Adapter [10df:e300] (rev 01)
> 0000:01:00.1 Fibre Channel [0c04]: Emulex Corporation Lancer Gen6: LPe32000 Fibre Channel Host Adapter [10df:e300] (rev 01)
> 0001:01:00.0 Fibre Channel [0c04]: Emulex Corporation Lancer-X: LightPulse Fibre Channel Host Adapter [10df:e200] (rev 10)
> 0001:01:00.1 Fibre Channel [0c04]: Emulex Corporation Lancer-X: LightPulse Fibre Channel Host Adapter [10df:e200] (rev 10)
> root@ltciofvtr-firestone1:~#
>
>
> Device driver error code:
>   [  537.317563] lpfc 0000:01:00.1: 1:0338 IOCB wait timeout error - no wake response Data x3c
> [  537.317755] lpfc 0000:01:00.1: 1:(0):0727 TMF FCP_LUN_RESET to TGT 1 LUN 0 failed (0, 0) iocb_flag x206
> [  537.317934] lpfc 0000:01:00.1: 1:(0):0713 SCSI layer issued Device Reset (1, 0) return x2007
> [  537.318005] lpfc 0000:01:00.1: 1:0372 iotag x0 is out off range: max iotag (x880)
> [  551.653563] lpfc 0000:01:00.0: 0:(0):0748 abort handler timed out waiting for abortng I/O (xri:x149) to complete: ret 0x2003, ID 1, LUN 1
> [  551.653795] lpfc 0000:01:00.0: 0:0372 iotag x0 is out off range: max iotag (x880)
> [  598.757557] lpfc 0000:01:00.1: 1:0338 IOCB wait timeout error - no wake response Data x3c
> [  598.757766] lpfc 0000:01:00.1: 1:(0):0727 TMF FCP_LUN_RESET to TGT 1 LUN 1 failed (0, 0) iocb_flag x206
> [  598.757946] lpfc 0000:01:00.1: 1:(0):0713 SCSI layer issued Device Reset (1, 1) return x2007
> [  598.758017] lpfc 0000:01:00.1: 1:0372 iotag x0 is out off range: max iotag (x880)
> [  613.093562] lpfc 0000:01:00.0: 0:(0):0748 abort handler timed out waiting for abortng I/O (xri:x14f) to complete: ret 0x2003, ID 1, LUN 0
> [  613.093630] INFO: task systemd-udevd:1148 blocked for more than 120 seconds.
> [  613.093631]       Not tainted 4.9.0-12-generic #13-Ubuntu
> [  613.093631] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [  613.093632] systemd-udevd   D    0  1148   1141 0x00040000
>
>   
> Stack trace output:
>   [  613.093633] Call Trace:
> [  613.093634] [c000001fd6b5b360] [ffffffffffffffff] 0xffffffffffffffff (unreliable)
> [  613.093636] [c000001fd6b5b530] [c00000000001c3a0] __switch_to+0x2e0/0x4c0
> [  613.093637] [c000001fd6b5b590] [c000000000b188d8] __schedule+0x2f8/0x990
> [  613.093638] [c000001fd6b5b670] [c000000000b18fb8] schedule+0x48/0xc0
> [  613.093640] [c000001fd6b5b6a0] [c000000000b1d394] schedule_timeout+0x274/0x470
> [  613.093641] [c000001fd6b5b790] [c000000000b19f8c] wait_for_common+0xec/0x240
> [  613.093642] [c000001fd6b5b810] [c0000000000ea27c] flush_work+0x12c/0x270
> [  613.093643] [c000001fd6b5b8a0] [c0000000000eca20] __cancel_work_timer+0xc0/0x220
> [  613.093645] [c000001fd6b5b940] [c00000000059966c] disk_block_events+0xcc/0xe0
> [  613.093646] [c000001fd6b5b990] [c00000000037124c] __blkdev_get+0x9c/0x490
> [  613.093648] [c000001fd6b5ba00] [c000000000372830] blkdev_get+0x1a0/0x4a0
> [  613.093649] [c000001fd6b5bab0] [c0000000003167e0] do_dentry_open+0x2d0/0x470
> [  613.093651] [c000001fd6b5bb10] [c00000000032fee4] do_last+0x614/0x1070
> [  613.093652] [c000001fd6b5bc00] [c000000000330a1c] path_openat+0xdc/0x480
> [  613.093654] [c000001fd6b5bc80] [c00000000033268c] do_filp_open+0xec/0x160
> [  613.093655] [c000001fd6b5bdb0] [c00000000031841c] do_sys_open+0x1cc/0x380
> [  613.093656] [c000001fd6b5be30] [c00000000000bd84] system_call+0x38/0xe0
> [  613.093657] INFO: task systemd-udevd:1155 blocked for more than 120 seconds.
> [  613.093658]       Not tainted 4.9.0-12-generic #13-Ubuntu
> [  613.093658] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [  613.093659] systemd-udevd   D    0  1155   1141 0x00040002
>   
> Hi Canonical,
>
> Can you please include these 2 commits to the lpfc driver on 17.04 and 16.04 HWE ?
> They've just been applied on mainline as of 4.11-rc1.
>
> The first resolves this problem, and the second prevents cache/DMA
> consistency problems which is likely to be hit in the future with this
> higher-performance adapter.
>
> I already asked for both patches to be flagged for stable kernels.
>
> [1] 8ea73db486cda442f0671f4bc9c03a76be398a28 lpfc: Correct WQ creation for pagesize
> [2] 6b3b3bdb83b4ad51252d21bb13596db879e51850 lpfc: Add missing memory barrier
>
> Thank you.
>
> [1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/drivers/scsi/lpfc?id=8ea73db486cda442f0671f4bc9c03a76be398a28
> [2] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/drivers/scsi/lpfc?id=6b3b3bdb83b4ad51252d21bb13596db879e51850
>
> ** Affects: ubuntu
>       Importance: Undecided
>       Assignee: Taco Screen team (taco-screen-team)
>           Status: New
>
>
> ** Tags: architecture-ppc64le bugnameltc-150924 severity-high targetmilestone-inin1704

-- 
Michael Hohnbaum
OIL Program Manager
Power (ppc64el) Development Project Manager
Canonical, Ltd.

Ubuntulinux package

Comment 3 for bug 1670490

Ubuntu
linux package