NVMe/RoCE I/O QID timeout during change volume ownership

[Jennifer Duong]

On my Ubuntu 20.04 kernel-5.4.0-24-generic nvme-cli 1.9-1 NVMe/RoCE config, I am seeing I/O QID timeouts when changing the ownership of the volumes on my E-Series array. From my understanding, this should not be occurring. My array is optimal while this is occurring and all of my NVMe/RoCE ports are up and optimal.

Apr 20 16:36:14 ICTM1611S01H4 kernel: [ 9819.553911] nvme nvme1: I/O 708 QID 1 timeout
Apr 20 16:36:14 ICTM1611S01H4 kernel: [ 9819.553914] nvme nvme1: I/O 29 QID 3 timeout
Apr 20 16:36:14 ICTM1611S01H4 kernel: [ 9819.553930] nvme nvme1: I/O 154 QID 4 timeout
Apr 20 16:36:14 ICTM1611S01H4 kernel: [ 9819.553931] nvme nvme1: I/O 695 QID 3 timeout
Apr 20 16:36:14 ICTM1611S01H4 kernel: [ 9819.553933] nvme nvme1: I/O 709 QID 1 timeout
Apr 20 16:36:14 ICTM1611S01H4 kernel: [ 9819.553935] nvme nvme1: I/O 155 QID 4 timeout
Apr 20 16:36:14 ICTM1611S01H4 kernel: [ 9819.553937] nvme nvme1: I/O 696 QID 3 timeout
Apr 20 16:36:14 ICTM1611S01H4 kernel: [ 9819.553938] nvme nvme1: I/O 710 QID 1 timeout
Apr 20 16:36:14 ICTM1611S01H4 kernel: [ 9819.553940] nvme nvme1: I/O 571 QID 4 timeout
Apr 20 16:36:14 ICTM1611S01H4 kernel: [ 9819.553942] nvme nvme1: I/O 697 QID 3 timeout
Apr 20 16:36:14 ICTM1611S01H4 kernel: [ 9819.553943] nvme nvme1: I/O 30 QID 3 timeout
Apr 20 16:36:14 ICTM1611S01H4 kernel: [ 9819.553946] nvme nvme1: I/O 156 QID 4 timeout
Apr 20 16:36:14 ICTM1611S01H4 kernel: [ 9819.553952] nvme nvme1: I/O 23 QID 3 timeout
Apr 20 16:36:14 ICTM1611S01H4 kernel: [ 9819.557842] nvme nvme1: I/O 965 QID 2 timeout
Apr 20 16:36:14 ICTM1611S01H4 kernel: [ 9819.557845] nvme nvme1: I/O 966 QID 2 timeout
Apr 20 16:36:14 ICTM1611S01H4 kernel: [ 9819.557847] nvme nvme1: I/O 967 QID 2 timeout

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: nvme-cli 1.9-1
ProcVersionSignature: Ubuntu 5.4.0-24.28-generic 5.4.30
Uname: Linux 5.4.0-24-generic x86_64
ApportVersion: 2.20.11-0ubuntu27
Architecture: amd64
CasperMD5CheckResult: skip
Date: Wed Apr 22 16:39:45 2020
InstallationDate: Installed on 2020-04-14 (8 days ago)
InstallationMedia: Ubuntu-Server 20.04 LTS "Focal Fossa" - Alpha amd64 (20200124)
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: nvme-cli
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.nvme.hostnqn: ictm1611s01h1-hostnqn
mtime.conffile..etc.nvme.hostnqn: 2020-04-14T16:03:41.867650

[Jennifer Duong (jduong)]

Message logs attached.

[Jennifer Duong (jduong)]

I'm running with the following cards:

MCX516A-GCAT FW 16.26.1040
MCX516A-CCAT FW 16.26.1040
QL45212H FW 8.37.7.0
MCX416A-CCAT FW 12.27.1016
MCX4121A-ACAT FW 14.27.1016

[Jennifer Duong (jduong)]

As a note, I am running without DA

[Jennifer Duong (jduong)]

I am still seeing this with Ubuntu 20.04 LTS

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.