QLogic Direct-Connect host can't discover SCSI-FC or NVMe/FC devices

Bug #1860724 reported by Jennifer Duong
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
Medium
Unassigned

Bug Description

My QLogic direct-connect host can't seem to SANboot or see any SCSI-FC devices in general. I'm also not able to discover any NVMe devices. I'm running with Ubuntu 20.04 kernel-5.4.0-9-generic.

There are the HBAs I'm running with:

root@ICTM1610S01H4:~# cat /sys/class/fc_host/host*/symbolic_name
QLE2742 FW:v8.08.231 DVR:v10.01.00.19-k
QLE2742 FW:v8.08.231 DVR:v10.01.00.19-k
QLE2692 FW:v8.08.231 DVR:v10.01.00.19-k
QLE2692 FW:v8.08.231 DVR:v10.01.00.19-k

lsscsi and multipath -ll don't seem to see my SCSI devices:

root@ICTM1610S01H4:/opt/iop/linux/scratch# multipath -ll
root@ICTM1610S01H4:/opt/iop/linux/scratch# lsscsi
[0:0:0:0] cd/dvd KVM vmDisk-CD 0.01 /dev/sr0
[1:0:0:0] cd/dvd HL-DT-ST DVDRAM GUD0N PF02 /dev/sr1
[3:0:0:0] disk ATA ST1000NX0313 SNA3 /dev/sda

It doesn't appear to be a configuration/hardware issue as installing Ubuntu 18.04 on the same exact server is able to SANboot and see my SCSI devices.

root@ICTM1610S01H4:/opt/iop/linux/scratch# lsb_release -rd
Description: Ubuntu Focal Fossa (development branch)
Release: 20.04
root@ICTM1610S01H4:/opt/iop/linux/scratch# apt-cache policy linux-image-generic
linux-image-generic:
  Installed: 5.4.0.9.11
  Candidate: 5.4.0.9.11
  Version table:
 *** 5.4.0.9.11 500
        500 http://repomirror-ict.eng.netapp.com/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: linux-image-generic 5.4.0.9.11
ProcVersionSignature: Ubuntu 5.4.0-9.12-generic 5.4.3
Uname: Linux 5.4.0-9-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Jan 23 11:32 seq
 crw-rw---- 1 root audio 116, 33 Jan 23 11:32 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.11-0ubuntu15
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Date: Thu Jan 23 15:04:42 2020
InstallationDate: Installed on 2020-01-23 (0 days ago)
InstallationMedia: Ubuntu-Server 20.04 LTS "Focal Fossa" - Alpha amd64 (20200107)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
MachineType: FUJITSU PRIMERGY RX2540 M4
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.4.0-9-generic root=UUID=9b9e6b1a-b8d9-4d9c-8782-36729d7f88a4 ro console=tty0 console=ttyS0,115200n8
RelatedPackageVersions:
 linux-restricted-modules-5.4.0-9-generic N/A
 linux-backports-modules-5.4.0-9-generic N/A
 linux-firmware 1.184
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/25/2019
dmi.bios.vendor: FUJITSU // American Megatrends Inc.
dmi.bios.version: V5.0.0.12 R1.35.0 for D3384-A1x
dmi.board.name: D3384-A1
dmi.board.vendor: FUJITSU
dmi.board.version: S26361-D3384-A13 WGS04 GS04
dmi.chassis.asset.tag: System Asset Tag
dmi.chassis.type: 23
dmi.chassis.vendor: FUJITSU
dmi.chassis.version: RX2540M4R4
dmi.modalias: dmi:bvnFUJITSU//AmericanMegatrendsInc.:bvrV5.0.0.12R1.35.0forD3384-A1x:bd06/25/2019:svnFUJITSU:pnPRIMERGYRX2540M4:pvr:rvnFUJITSU:rnD3384-A1:rvrS26361-D3384-A13WGS04GS04:cvnFUJITSU:ct23:cvrRX2540M4R4:
dmi.product.family: SERVER
dmi.product.name: PRIMERGY RX2540 M4
dmi.product.sku: S26361-K1567-Vxxx
dmi.sys.vendor: FUJITSU

CVE References

Revision history for this message
Jennifer Duong (jduong) wrote :
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Status changed to Confirmed

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
Revision history for this message
Jennifer Duong (jduong) wrote : Re: QLogic Direct-Connect host can't see SCSI-FC devices
Jennifer Duong (jduong)
summary: - QLogic Direct-Connect host can't see SCSI-FC devices
+ QLogic Direct-Connect host can't discover SCSI-FC or NVMe/FC devices
description: updated
Revision history for this message
Paolo Pisati (p-pisati) wrote :

A new Focal kernel that contains (among the other things) several scsi/qla2xx fixes is in -proposed now:

linux-image-generic | 5.4.0.13.16 | focal-proposed | amd64, arm64, armhf, ppc64el, s390x

could you test that?

In case that doesn't fix you issue, can you give us the output of "dmesg / cat /sys/class/fc_host/host*/symbolic_name / multipath -ll / lsscsi" of a working kernel?
You mentioned Bionic is working fine on this setup.

Revision history for this message
Jennifer Duong (jduong) wrote :

Paolo, it doesn't look like the newest Focal kernel resolves this issue. I've attached the output from a working kernel.

root@ICTM1610S01H2:~# cat /sys/class/fc_host/host*/symbolic_name
QLE2742 FW:v8.08.231 DVR:v10.01.00.19-k
QLE2742 FW:v8.08.231 DVR:v10.01.00.19-k
QLE2692 FW:v8.08.231 DVR:v10.01.00.19-k
QLE2692 FW:v8.08.231 DVR:v10.01.00.19-k

Revision history for this message
Jennifer Duong (jduong) wrote :

Paolo, what should my next steps be?

Revision history for this message
Jennifer Duong (jduong) wrote :

Paolo, I updated to kernel-5.4.0-14-generic and am still encountering this issue. Is there any additional logs you may need from me?

Revision history for this message
Jennifer Duong (jduong) wrote :

Any suggestions on what to try next?

Revision history for this message
Jennifer Duong (jduong) wrote :

Paolo, I've upgraded to kernel-5.4.0-16-generic and am still encountering this issue. Any suggestions on what to try next? Are there any additional logs that you'd like for me to grab?

Revision history for this message
Jennifer Duong (jduong) wrote :

Any update on this?

Revision history for this message
Jennifer Duong (jduong) wrote :

What should I try next?

Revision history for this message
Jennifer Duong (jduong) wrote :

Has anyone had a chance to look into this?

Revision history for this message
Jennifer Duong (jduong) wrote :

Any update on this?

Revision history for this message
Jennifer Duong (jduong) wrote :

I am still seeing this with Ubuntu 20.04 LTS

Revision history for this message
Kai-Heng Feng (kaihengfeng) wrote :

I've never used a complex setup like this, so I have to ask, are we sure it's kernel bug?

Revision history for this message
Jennifer Duong (jduong) wrote :

I believe it's a bug with the inbox Qlogic driver

Revision history for this message
Kai-Heng Feng (kaihengfeng) wrote :

If latest mainline kernel doesn't solve the issue, please raise the issue to maintainers and mailing list via email:
Nilesh Javali <email address hidden> (supporter:QLOGIC QLA2XXX FC-SCSI DRIVER)
<email address hidden> (supporter:QLOGIC QLA2XXX FC-SCSI DRIVER)
"James E.J. Bottomley" <email address hidden> (maintainer:SCSI SUBSYSTEM)
"Martin K. Petersen" <email address hidden> (maintainer:SCSI SUBSYSTEM)
<email address hidden> (open list:QLOGIC QLA2XXX FC-SCSI DRIVER)
<email address hidden> (open list)

Revision history for this message
Jennifer Duong (jduong) wrote :
Download full text (5.4 KiB)

I can now discover NVMe/FC devices on my Qlogic direct connect system after upgrading to kernel-5.4.0-70-generic. nvme-cli-1.9-1 is installed.

root@ICTM1608S01H4:~# nvme list
Node SN Model Namespace Usage Format FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 721838500080 NetApp E-Series 100 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n10 721838500080 NetApp E-Series 109 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n11 721838500080 NetApp E-Series 110 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n12 721838500080 NetApp E-Series 111 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n13 721838500080 NetApp E-Series 112 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n14 721838500080 NetApp E-Series 113 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n15 721838500080 NetApp E-Series 114 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n16 721838500080 NetApp E-Series 115 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n17 721838500080 NetApp E-Series 116 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n18 721838500080 NetApp E-Series 117 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n19 721838500080 NetApp E-Series 118 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n2 721838500080 NetApp E-Series 101 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n20 721838500080 NetApp E-Series 119 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n21 721838500080 NetApp E-Series 120 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n22 721838500080 NetApp E-Series 121 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n23 721838500080 NetApp E-Series 122 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n24 721838500080 NetApp E-Series 123 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n25 721838500080 NetApp E-Series 124 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n26 721838500080 NetApp E-Series 125 4.29 GB / 4.29 GB 512 B + 0 B 88714915
/dev/nvme0n27 721838500080 NetApp...

Read more...

Tim Gardner (timg-tpi)
Changed in linux (Ubuntu):
status: Confirmed → Fix Released
Stefan Bader (smb)
Changed in linux (Ubuntu Focal):
importance: Undecided → Medium
status: New → In Progress
Stefan Bader (smb)
Changed in linux (Ubuntu Focal):
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
Revision history for this message
Jennifer Duong (jduong) wrote :

After upgrading to kernel-5.4.0-74-generic, the Qlogic direct-connect host is able to discover both SCSI-FC and NVMe/FC devices. See attached logs.

Jennifer Duong (jduong)
tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (27.3 KiB)

This bug was fixed in the package linux - 5.4.0-80.90

---------------
linux (5.4.0-80.90) focal; urgency=medium

  * CVE-2021-33909
    - SAUCE: seq_file: Disallow extremely large seq buffer allocations

linux (5.4.0-79.88) focal; urgency=medium

  * focal/linux: 5.4.0-79.88 -proposed tracker (LP: #1934343)

  * lxd exec fails (LP: #1934187)
    - SAUCE: Revert "proc: Check /proc/$pid/attr/ writes against file opener"

linux (5.4.0-78.87) focal; urgency=medium

  * focal/linux: 5.4.0-78.87 -proposed tracker (LP: #1932478)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync getabis
    - [Packaging] update helper scripts
    - update dkms package versions

  * Disable hv-kvp-daemon.service on certain instance types (LP: #1932081)
    - [Packaging]: Add kernel command line condition to hv-kvp-daemon service

  * QLogic Direct-Connect host can't discover SCSI-FC or NVMe/FC devices
    (LP: #1860724)
    - scsi: qla2xxx: Serialize fc_port alloc in N2N
    - scsi: qla2xxx: Set Nport ID for N2N
    - scsi: qla2xxx: Fix point-to-point (N2N) device discovery issue
    - scsi: qla2xxx: Fix N2N and NVMe connect retry failure

  * [SRU] Add support for E810 NIC to Ice Driver in Focal (LP: #1912511)
    - ice: add additional E810 device id

  * Focal update: v5.4.124 upstream stable release (LP: #1931166)
    - ALSA: hda/realtek: Headphone volume is controlled by Front mixer
    - ALSA: usb-audio: scarlett2: Fix device hang with ehci-pci
    - ALSA: usb-audio: scarlett2: Improve driver startup messages
    - cifs: set server->cipher_type to AES-128-CCM for SMB3.0
    - NFSv4: Fix a NULL pointer dereference in pnfs_mark_matching_lsegs_return()
    - iommu/vt-d: Fix sysfs leak in alloc_iommu()
    - perf intel-pt: Fix sample instruction bytes
    - perf intel-pt: Fix transaction abort handling
    - perf scripts python: exported-sql-viewer.py: Fix copy to clipboard from Top
      Calls by elapsed Time report
    - perf scripts python: exported-sql-viewer.py: Fix Array TypeError
    - perf scripts python: exported-sql-viewer.py: Fix warning display
    - proc: Check /proc/$pid/attr/ writes against file opener
    - net: hso: fix control-request directions
    - ath10k: Validate first subframe of A-MSDU before processing the list
    - dm snapshot: properly fix a crash when an origin has no snapshots
    - drm/amdgpu/vcn1: add cancel_delayed_work_sync before power gate
    - drm/amdgpu/vcn2.0: add cancel_delayed_work_sync before power gate
    - drm/amdgpu/vcn2.5: add cancel_delayed_work_sync before power gate
    - selftests/gpio: Use TEST_GEN_PROGS_EXTENDED
    - selftests/gpio: Move include of lib.mk up
    - selftests/gpio: Fix build when source tree is read only
    - kgdb: fix gcc-11 warnings harder
    - Documentation: seccomp: Fix user notification documentation
    - serial: core: fix suspicious security_locked_down() call
    - misc/uss720: fix memory leak in uss720_probe
    - thunderbolt: dma_port: Fix NVM read buffer bounds and offset issue
    - mei: request autosuspend after sending rx flow control
    - staging: iio: cdc: ad7746: avoid overwrite of num_channels
    - iio: gyro: fxas21002c: balance runtime power in error path
    - ...

Changed in linux (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.