Brief Description ----------------- Configuration of labs with mellanox interfaces fail. The interface is not visible in linux output (ifconfig -a).
Severity -------- Major
Steps to Reproduce ------------------ Interfaces (mellanox) do not appear to be visible in the linux output (while they do appear to be there).
This is preventing install/configuration from successfully completing The following error is reported during configure controller step on these labs.
Errors are being repored during configure controller indicates that it is unable to find the interface ens785f0 eg.
{"log":"E0117 20:18:30.697446 1 common.go:226] controller/host \"msg\"=\"user data error\" \"error\"=\"unable to find interface UUID for port: ens785f0\" \"request\"= {\"Namespace\":\"deployment\",\"Name\":\"controller-0\"} \n","stream":"stderr","time":"2020-01-17T20:18:30.704588677Z"} {"log":"E0117 20:19:31.405278 1 common.go:226] controller/host \"msg\"=\"user data error\" \"error\"=\"unable to find interface UUID for port: ens785f0\" \"request\"= {\"Namespace\":\"deployment\",\"Name\":\"controller-0\"} \n","stream":"stderr","time":"2020-01-17T20:19:31.405496902Z"}
The mellanox interfaces do not appear to be visible. Only these 2 interfaces are reported in the linux output.
$ system host-port-list controller-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+----------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+----------------------------------+ | 2b181f2c-f742-41f4-b718-c9d62da8067e | enp3s0f0 | ethernet | 0000:03:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | | | | | | | | | [1521] | | | | | | | | | | | f02d98df-86dc-4386-9707-b12aa7dff50a | enp3s0f3 | ethernet | 0000:03:00.3 | 0 | 0 | True | I350 Gigabit Network Connection | | | | | | | | | [1521]
controller-0:~$ ifconfig -a cali675b11b2b81: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::ecee:eeff:feee:eeee prefixlen 64 scopeid 0x20<link> ether ee:ee:ee:ee:ee:ee txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
cali8c7155b8c25: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::ecee:eeff:feee:eeee prefixlen 64 scopeid 0x20<link> ether ee:ee:ee:ee:ee:ee txqueuelen 0 (Ethernet) RX packets 40971661 bytes 14589139995 (13.5 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 40971661 bytes 14589139995 (13.5 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
calid2619c1ab5f: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::ecee:eeff:feee:eeee prefixlen 64 scopeid 0x20<link> ether ee:ee:ee:ee:ee:ee txqueuelen 0 (Ethernet) RX packets 40971661 bytes 14589139995 (13.5 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 40971661 bytes 14589139995 (13.5 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255 ether 02:42:70:69:02:fc txqueuelen 0 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp3s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 128.224.151.80 netmask 255.255.254.0 broadcast 128.224.151.255 inet6 2620:10a:a001:a103:a6bf:1ff:fe00:690 prefixlen 64 scopeid 0x0<global> inet6 fe80::a6bf:1ff:fe00:690 prefixlen 64 scopeid 0x20<link> ether a4:bf:01:00:06:90 txqueuelen 1000 (Ethernet) RX packets 11602842 bytes 3055925470 (2.8 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 423747 bytes 35184409 (33.5 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device memory 0x91920000-9193ffff
enp3s0f3: flags=4098<BROADCAST,MULTICAST> mtu 1500 ether a4:bf:01:00:06:91 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device memory 0x91900000-9191ffff
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 40971661 bytes 14589139995 (13.5 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 40971661 bytes 14589139995 (13.5 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo:1: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 192.168.204.2 netmask 255.255.255.0 loop txqueuelen 1000 (Local Loopback)
lo:5: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 192.168.206.2 netmask 255.255.255.0 loop txqueuelen 1000 (Local Loopback)
tunl0: flags=193<UP,RUNNING,NOARP> mtu 1440 inet 172.16.192.64 netmask 255.255.255.255 tunnel txqueuelen 1000 (IPIP Tunnel) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Expected Behavior ------------------ Expect the lab install/configuration to succeed (as previously load did 2019-12-13_19-03-42)
Actual Behavior ---------------- Configuration fails as interface expected is not visible in linux.
Reproducibility --------------- yes
System Configuration -------------------- wcp_61-62 wcp_63-66
Lab-name:
Branch/Pull Time/Commit ----------------------- 20200117T023005Z
Last Pass --------- The same configuration for lab wcp 63-66 worked on the following build 2019-12-13_19-03-42
Note: Output for december load (for lab wcp63-66) was as following. Two highlighted ports are missing on the latest master load. [sysadmin@controller-0 ~(keystone_admin)]$ system host-port-list controller-0 +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+----------------------------------------+ | uuid | name | type | pci address | device | processor | accelerated | device type | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+----------------------------------------+ | 6f857eaa-f2ca-4d38-9849-c0ca37d8184f | eno1 | ethernet | 0000:03:00.0 | 0 | 0 | False | I350 Gigabit Network Connection [1521] | | bf93c6d6-4127-44eb-bce8-460327d43082 | eno2 | ethernet | 0000:03:00.3 | 0 | 0 | False | I350 Gigabit Network Connection [1521] | | 12bc274e-3248-458b-ac6a-f051e7f8201a | ens785f0 | ethernet | 0000:05:00.0 | 0 | 0 | False | MT27710 Family [ConnectX-4 Lx] [1015] | | 0c72b5e3-e8b2-4abb-bb2b-fe15e34e4d0b | ens785f1 | ethernet | 0000:05:00.1 | 0 | 0 | False | MT27710 Family [ConnectX-4 Lx] [1015] | +--------------------------------------+----------+----------+--------------+--------+-----------+-------------+----------------------------------------+
Timestamp/Logs --------------
Test Activity ------------- Install/configuration
Brief Description
-----------------
Configuration of labs with mellanox interfaces fail. The interface is not visible in linux output (ifconfig -a).
Severity
--------
Major
Steps to Reproduce
------------------
Interfaces (mellanox) do not appear to be visible in the linux output (while they do appear to be there).
This is preventing install/ configuration from successfully completing
The following error is reported during configure controller step on these labs.
Errors are being repored during configure controller indicates that it is unable to find the interface ens785f0
eg.
{"log":"E0117 20:18:30.697446 1 common.go:226] controller/host \"msg\"=\"user data error\" \"error\"=\"unable to find interface UUID for port: ens785f0\" \"request\"= ":\"deployment\ ",\"Name\ ":\"controller- 0\"} :"stderr" ,"time" :"2020- 01-17T20: 18:30.704588677 Z"} ":\"deployment\ ",\"Name\ ":\"controller- 0\"} :"stderr" ,"time" :"2020- 01-17T20: 19:31.405496902 Z"}
{\"Namespace\
\n","stream"
{"log":"E0117 20:19:31.405278 1 common.go:226] controller/host \"msg\"=\"user data error\" \"error\"=\"unable to find interface UUID for port: ens785f0\" \"request\"=
{\"Namespace\
\n","stream"
The mellanox interfaces do not appear to be visible. Only these 2 interfaces are reported in the linux output.
$ system host-port-list controller-0 ------- ------- ------- ------- ----+-- ------- -+----- -----+- ------- ------+ ------- -+----- ------+ ------- ------+ ------- ------- ------- ------- ------+ ------- ------- ------- ------- ----+-- ------- -+----- -----+- ------- ------+ ------- -+----- ------+ ------- ------+ ------- ------- ------- ------- ------+ f742-41f4- b718-c9d62da806 7e | enp3s0f0 | ethernet | 0000:03:00.0 | 0 | 0 | True | I350 Gigabit Network Connection | 86dc-4386- 9707-b12aa7dff5 0a | enp3s0f3 | ethernet | 0000:03:00.3 | 0 | 0 | True | I350 Gigabit Network Connection |
+------
| uuid | name | type | pci address | device | processor | accelerated | device type |
+------
| 2b181f2c-
| | | | | | | | [1521] |
| | | | | | | | |
| f02d98df-
| | | | | | | | [1521]
controller-0:~$ ifconfig -a UP,BROADCAST, RUNNING, MULTICAST> mtu 1500 eeff:feee: eeee prefixlen 64 scopeid 0x20<link>
cali675b11b2b81: flags=4163<
inet6 fe80::ecee:
ether ee:ee:ee:ee:ee:ee txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
cali8c7155b8c25: flags=4163< UP,BROADCAST, RUNNING, MULTICAST> mtu 1500 eeff:feee: eeee prefixlen 64 scopeid 0x20<link>
inet6 fe80::ecee:
ether ee:ee:ee:ee:ee:ee txqueuelen 0 (Ethernet)
RX packets 40971661 bytes 14589139995 (13.5 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 40971661 bytes 14589139995 (13.5 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
calid2619c1ab5f: flags=4163< UP,BROADCAST, RUNNING, MULTICAST> mtu 1500 eeff:feee: eeee prefixlen 64 scopeid 0x20<link>
inet6 fe80::ecee:
ether ee:ee:ee:ee:ee:ee txqueuelen 0 (Ethernet)
RX packets 40971661 bytes 14589139995 (13.5 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 40971661 bytes 14589139995 (13.5 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
docker0: flags=4099< UP,BROADCAST, MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:70:69:02:fc txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp3s0f0: flags=4163< UP,BROADCAST, RUNNING, MULTICAST> mtu 1500 a001:a103: a6bf:1ff: fe00:690 prefixlen 64 scopeid 0x0<global> 1ff:fe00: 690 prefixlen 64 scopeid 0x20<link>
inet 128.224.151.80 netmask 255.255.254.0 broadcast 128.224.151.255
inet6 2620:10a:
inet6 fe80::a6bf:
ether a4:bf:01:00:06:90 txqueuelen 1000 (Ethernet)
RX packets 11602842 bytes 3055925470 (2.8 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 423747 bytes 35184409 (33.5 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0x91920000-9193ffff
enp3s0f3: flags=4098< BROADCAST, MULTICAST> mtu 1500
ether a4:bf:01:00:06:91 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0x91900000-9191ffff
lo: flags=73< UP,LOOPBACK, RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 40971661 bytes 14589139995 (13.5 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 40971661 bytes 14589139995 (13.5 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo:1: flags=73< UP,LOOPBACK, RUNNING> mtu 65536
inet 192.168.204.2 netmask 255.255.255.0
loop txqueuelen 1000 (Local Loopback)
lo:5: flags=73< UP,LOOPBACK, RUNNING> mtu 65536
inet 192.168.206.2 netmask 255.255.255.0
loop txqueuelen 1000 (Local Loopback)
tunl0: flags=193< UP,RUNNING, NOARP> mtu 1440
inet 172.16.192.64 netmask 255.255.255.255
tunnel txqueuelen 1000 (IPIP Tunnel)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Expected Behavior configuration to succeed (as previously load did 2019-12- 13_19-03- 42)
------------------
Expect the lab install/
Actual Behavior
----------------
Configuration fails as interface expected is not visible in linux.
Reproducibility
---------------
yes
System Configuration ------- ------
-------
wcp_61-62
wcp_63-66
Lab-name:
Branch/Pull Time/Commit ------- ------- --
-------
20200117T023005Z
Last Pass
---------
The same configuration for lab wcp 63-66 worked on the following build
2019-12-13_19-03-42
Note: Output for december load (for lab wcp63-66) was as following. controller- 0 ~(keystone_admin)]$ system host-port-list controller-0 ------- ------- ------- ------- ----+-- ------- -+----- -----+- ------- ------+ ------- -+----- ------+ ------- ------+ ------- ------- ------- ------- ------- -----+ ------- ------- ------- ------- ----+-- ------- -+----- -----+- ------- ------+ ------- -+----- ------+ ------- ------+ ------- ------- ------- ------- ------- -----+ f2ca-4d38- 9849-c0ca37d818 4f | eno1 | ethernet | 0000:03:00.0 | 0 | 0 | False | I350 Gigabit Network Connection [1521] | 4127-44eb- bce8-460327d430 82 | eno2 | ethernet | 0000:03:00.3 | 0 | 0 | False | I350 Gigabit Network Connection [1521] | 3248-458b- ac6a-f051e7f820 1a | ens785f0 | ethernet | 0000:05:00.0 | 0 | 0 | False | MT27710 Family [ConnectX-4 Lx] [1015] | e8b2-4abb- bb2b-fe15e34e4d 0b | ens785f1 | ethernet | 0000:05:00.1 | 0 | 0 | False | MT27710 Family [ConnectX-4 Lx] [1015] | ------- ------- ------- ------- ----+-- ------- -+----- -----+- ------- ------+ ------- -+----- ------+ ------- ------+ ------- ------- ------- ------- ------- -----+
Two highlighted ports are missing on the latest master load.
[sysadmin@
+------
| uuid | name | type | pci address | device | processor | accelerated | device type |
+------
| 6f857eaa-
| bf93c6d6-
| 12bc274e-
| 0c72b5e3-
+------
Timestamp/Logs
--------------
Test Activity configuration
-------------
Install/