Leann, A few driver patches for the Kernel team to look at. Michael On 03/06/2017 01:10 PM, Launchpad Bug Tracker wrote: > bugproxy (bugproxy) has assigned this bug to you for Ubuntu: > > ---Problem Description--- > FC Adapter goes offline and produces the call traces while booting into OS, on assigning the LUNs to it. > > ---uname output--- > Linux ltciofvtr-firestone1 4.9.0-12-generic #13-Ubuntu SMP Tue Jan 10 12:52:39 UTC 2017 ppc64le ppc64le ppc64le GNU/Linux > > FC Redfish Adapter(32 GB) is going offline on assigning the Luns to it. Adapter shows online till pettitboot and even Luns are also vissible in pettitboot. > once selecting the OS from pettitboot and booting into it, it produces call traces and also few lpfc errors. attached the comple console logs FYR. > The FC switch to which the Redfish adapter is connected is a 16 GB switch and the adapter is of 32 GB. According to me this parameter should not have any concern as the adapter has to support backward compatibility. > > ---Steps to Reproduce--- > 1. install the adapter in a server. and connect it to FC switch (16 GB) > 2. create zone and assign the LUNS to it from V7000. > 3. reboot the OS > > > root@ltciofvtr-firestone1:~# ll /sys/class/fc_host/ > total 0 > drwxr-xr-x 2 root root 0 Jan 25 04:22 ./ > drwxr-xr-x 72 root root 0 Jan 25 03:50 ../ > lrwxrwxrwx 1 root root 0 Jan 25 04:13 host10 -> ../../devices/pci0001:00/0001:00:00.0/0001:01:00.1/host10/fc_host/host10/ > lrwxrwxrwx 1 root root 0 Jan 25 04:13 host6 -> ../../devices/pci0000:00/0000:00:00.0/0000:01:00.0/host6/fc_host/host6/ > lrwxrwxrwx 1 root root 0 Jan 25 04:13 host8 -> ../../devices/pci0000:00/0000:00:00.0/0000:01:00.1/host8/fc_host/host8/ > lrwxrwxrwx 1 root root 0 Jan 25 04:13 host9 -> ../../devices/pci0001:00/0001:00:00.0/0001:01:00.0/host9/fc_host/host9/ > root@ltciofvtr-firestone1:~# > root@ltciofvtr-firestone1:~# cat /sys/class/fc_host/host6/port_state > Offline > root@ltciofvtr-firestone1:~# cat /sys/class/fc_host/host8/port_state > Offline > root@ltciofvtr-firestone1:~# cat /sys/class/fc_host/host9/port_state > Online > root@ltciofvtr-firestone1:~# cat /sys/class/fc_host/host10/port_state > Online > root@ltciofvtr-firestone1:~# > root@ltciofvtr-firestone1:~# lspci -nn | grep -i fibre > 0000:01:00.0 Fibre Channel [0c04]: Emulex Corporation Lancer Gen6: LPe32000 Fibre Channel Host Adapter [10df:e300] (rev 01) > 0000:01:00.1 Fibre Channel [0c04]: Emulex Corporation Lancer Gen6: LPe32000 Fibre Channel Host Adapter [10df:e300] (rev 01) > 0001:01:00.0 Fibre Channel [0c04]: Emulex Corporation Lancer-X: LightPulse Fibre Channel Host Adapter [10df:e200] (rev 10) > 0001:01:00.1 Fibre Channel [0c04]: Emulex Corporation Lancer-X: LightPulse Fibre Channel Host Adapter [10df:e200] (rev 10) > root@ltciofvtr-firestone1:~# > > > Device driver error code: > [ 537.317563] lpfc 0000:01:00.1: 1:0338 IOCB wait timeout error - no wake response Data x3c > [ 537.317755] lpfc 0000:01:00.1: 1:(0):0727 TMF FCP_LUN_RESET to TGT 1 LUN 0 failed (0, 0) iocb_flag x206 > [ 537.317934] lpfc 0000:01:00.1: 1:(0):0713 SCSI layer issued Device Reset (1, 0) return x2007 > [ 537.318005] lpfc 0000:01:00.1: 1:0372 iotag x0 is out off range: max iotag (x880) > [ 551.653563] lpfc 0000:01:00.0: 0:(0):0748 abort handler timed out waiting for abortng I/O (xri:x149) to complete: ret 0x2003, ID 1, LUN 1 > [ 551.653795] lpfc 0000:01:00.0: 0:0372 iotag x0 is out off range: max iotag (x880) > [ 598.757557] lpfc 0000:01:00.1: 1:0338 IOCB wait timeout error - no wake response Data x3c > [ 598.757766] lpfc 0000:01:00.1: 1:(0):0727 TMF FCP_LUN_RESET to TGT 1 LUN 1 failed (0, 0) iocb_flag x206 > [ 598.757946] lpfc 0000:01:00.1: 1:(0):0713 SCSI layer issued Device Reset (1, 1) return x2007 > [ 598.758017] lpfc 0000:01:00.1: 1:0372 iotag x0 is out off range: max iotag (x880) > [ 613.093562] lpfc 0000:01:00.0: 0:(0):0748 abort handler timed out waiting for abortng I/O (xri:x14f) to complete: ret 0x2003, ID 1, LUN 0 > [ 613.093630] INFO: task systemd-udevd:1148 blocked for more than 120 seconds. > [ 613.093631] Not tainted 4.9.0-12-generic #13-Ubuntu > [ 613.093631] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 613.093632] systemd-udevd D 0 1148 1141 0x00040000 > > > Stack trace output: > [ 613.093633] Call Trace: > [ 613.093634] [c000001fd6b5b360] [ffffffffffffffff] 0xffffffffffffffff (unreliable) > [ 613.093636] [c000001fd6b5b530] [c00000000001c3a0] __switch_to+0x2e0/0x4c0 > [ 613.093637] [c000001fd6b5b590] [c000000000b188d8] __schedule+0x2f8/0x990 > [ 613.093638] [c000001fd6b5b670] [c000000000b18fb8] schedule+0x48/0xc0 > [ 613.093640] [c000001fd6b5b6a0] [c000000000b1d394] schedule_timeout+0x274/0x470 > [ 613.093641] [c000001fd6b5b790] [c000000000b19f8c] wait_for_common+0xec/0x240 > [ 613.093642] [c000001fd6b5b810] [c0000000000ea27c] flush_work+0x12c/0x270 > [ 613.093643] [c000001fd6b5b8a0] [c0000000000eca20] __cancel_work_timer+0xc0/0x220 > [ 613.093645] [c000001fd6b5b940] [c00000000059966c] disk_block_events+0xcc/0xe0 > [ 613.093646] [c000001fd6b5b990] [c00000000037124c] __blkdev_get+0x9c/0x490 > [ 613.093648] [c000001fd6b5ba00] [c000000000372830] blkdev_get+0x1a0/0x4a0 > [ 613.093649] [c000001fd6b5bab0] [c0000000003167e0] do_dentry_open+0x2d0/0x470 > [ 613.093651] [c000001fd6b5bb10] [c00000000032fee4] do_last+0x614/0x1070 > [ 613.093652] [c000001fd6b5bc00] [c000000000330a1c] path_openat+0xdc/0x480 > [ 613.093654] [c000001fd6b5bc80] [c00000000033268c] do_filp_open+0xec/0x160 > [ 613.093655] [c000001fd6b5bdb0] [c00000000031841c] do_sys_open+0x1cc/0x380 > [ 613.093656] [c000001fd6b5be30] [c00000000000bd84] system_call+0x38/0xe0 > [ 613.093657] INFO: task systemd-udevd:1155 blocked for more than 120 seconds. > [ 613.093658] Not tainted 4.9.0-12-generic #13-Ubuntu > [ 613.093658] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 613.093659] systemd-udevd D 0 1155 1141 0x00040002 > > Hi Canonical, > > Can you please include these 2 commits to the lpfc driver on 17.04 and 16.04 HWE ? > They've just been applied on mainline as of 4.11-rc1. > > The first resolves this problem, and the second prevents cache/DMA > consistency problems which is likely to be hit in the future with this > higher-performance adapter. > > I already asked for both patches to be flagged for stable kernels. > > [1] 8ea73db486cda442f0671f4bc9c03a76be398a28 lpfc: Correct WQ creation for pagesize > [2] 6b3b3bdb83b4ad51252d21bb13596db879e51850 lpfc: Add missing memory barrier > > Thank you. > > [1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/drivers/scsi/lpfc?id=8ea73db486cda442f0671f4bc9c03a76be398a28 > [2] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/drivers/scsi/lpfc?id=6b3b3bdb83b4ad51252d21bb13596db879e51850 > > ** Affects: ubuntu > Importance: Undecided > Assignee: Taco Screen team (taco-screen-team) > Status: New > > > ** Tags: architecture-ppc64le bugnameltc-150924 severity-high targetmilestone-inin1704 -- Michael Hohnbaum OIL Program Manager Power (ppc64el) Development Project Manager Canonical, Ltd.