NVMe connect fails with multiple NICs on storage subnet
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
os-brick |
New
|
Undecided
|
Unassigned |
Bug Description
If there are multiple NICs on a server that are connected to the storage subnet providing connectivity to the NVMe backend, but not all of these are NVMe capable interfaces, specifically NVMe-RoCE, which requires the NIC to be an approved Mellanox card, eg CX-5, then the nvme connect command will fail as the `-w` switch isn't specified to select the correct NIC for the RDNA transport.
Failure traceback:
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
Aug 21 16:23:27 np0000004186 cinder-
For this example the node running os-brick has these physical ports:
# ip a
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,
link/ether fa:16:3e:58:03:b7 brd ff:ff:ff:ff:ff:ff
inet 10.241.128.141/24 brd 10.241.128.255 scope global dynamic enp3s0
valid_lft 40152sec preferred_lft 40152sec
inet6 fe80::f816:
valid_lft forever preferred_lft forever
3: enp4s0: <BROADCAST,
link/ether fa:16:3e:56:fb:ee brd ff:ff:ff:ff:ff:ff
inet 192.168.1.146/24 brd 192.168.1.255 scope global dynamic enp4s0
valid_lft 40152sec preferred_lft 40152sec
inet6 fe80::f816:
valid_lft forever preferred_lft forever
4: enp6s0: <BROADCAST,
link/ether 6a:c2:78:28:09:50 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.246/24 scope global enp6s0
valid_lft forever preferred_lft forever
inet6 fe80::68c2:
valid_lft forever preferred_lft forever
Notice that enp4s0 and enp6s0 are both on the storage subnet, but only enp6s0 is a Mellanox (RoCE) capable NIC. enp4s0 is a virtio NIC.
# lspci
...
03:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
04:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
05:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01)
06:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
...
Routing table has routes for both storage subnet NICs to reach the NVMe ports on the backend:
# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 10.241.128.1 0.0.0.0 UG 0 0 0 enp3s0
10.241.128.0 0.0.0.0 255.255.255.0 U 0 0 0 enp3s0
169.254.169.254 192.168.1.100 255.255.255.255 UGH 0 0 0 enp4s0
172.24.5.0 0.0.0.0 255.255.255.0 U 0 0 0 br-ex
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 enp4s0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 enp6s0
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
To resolve this the nvme connect command needs to have the Mellanox IP used with the -w switch to correctly connect.
summary: |
- NVMe connect fails iwith multiple NICs on storage subnet + NVMe connect fails with multiple NICs on storage subnet |
description: | updated |
description: | updated |