10G 2P X520 Times Out During Bootstrap

[Michael Petersen]

Bug Description:

10G interfaces are being configured with the correct driver, however they are not being set to up after the node is bootstrapped. The interfaces can be brought up manually after the node has been bootstrapped.

The other 1G interfaces come up.

Steps to Reproduce:

Bootstrap a node with 10G 2P x520 interfaces

Expected Results:

The interfaces will be set to UP so that they can be configured via Fuel.

Actual Results:

The 10G interfaces are down.

Additional:

#fuel --version
9.0.0

Network card model : Intel(R) 10G 2P X520 Adapter

It looks like there are issues with bringing up the interfaces on 9.1. The --version shows 9.0.0, but the user is on 9.1.

Logs:

2017-01-06T13:58:03.303176+00:00 err: kernel: [ 139.137174] systemd-udevd[863]: worker [872] /devices/pci0000:00/0000:00:03.2/0000:05:00.0 timeout; kill it
2017-01-06T13:58:03.303251+00:00 err: kernel: [ 139.137181] systemd-udevd[863]: seq 5059 '/devices/pci0000:00/0000:00:03.2/0000:05:00.0' killed
2017-01-06T13:58:03.303292+00:00 err: kernel: [ 139.137190] systemd-udevd[863]: worker [877] /devices/pci0000:00/0000:00:02.2/0000:04:00.0 timeout; kill it
2017-01-06T13:58:03.303349+00:00 err: kernel: [ 139.137194] systemd-udevd[863]: seq 5047 '/devices/pci0000:00/0000:00:02.2/0000:04:00.0' killed
2017-01-06T13:58:03.303406+00:00 err: kernel: [ 139.137201] systemd-udevd[863]: worker [878] /devices/pci0000:00/0000:00:02.2/0000:04:00.1 timeout; kill it
2017-01-06T13:58:03.303463+00:00 err: kernel: [ 139.137205] systemd-udevd[863]: seq 5048 '/devices/pci0000:00/0000:00:02.2/0000:04:00.1' killed
2017-01-06T13:58:03.303520+00:00 err: kernel: [ 139.137210] systemd-udevd[863]: worker [881] /devices/pci0000:00/0000:00:03.2/0000:05:00.1 timeout; kill it
2017-01-06T13:58:03.303576+00:00 err: kernel: [ 139.137214] systemd-udevd[863]: seq 5060 '/devices/pci0000:00/0000:00:03.2/0000:05:00.1' killed
2017-01-06T13:58:03.303634+00:00 err: kernel: [ 139.137220] systemd-udevd[863]: worker [885] /devices/pci0000:80/0000:80:03.2/0000:84:00.0 timeout; kill it
2017-01-06T13:58:03.303708+00:00 err: kernel: [ 139.137224] systemd-udevd[863]: seq 5197 '/devices/pci0000:80/0000:80:03.2/0000:84:00.0' killed
2017-01-06T13:58:03.303748+00:00 err: kernel: [ 139.137230] systemd-udevd[863]: worker [889] /devices/pci0000:80/0000:80:03.0/0000:83:00.0 timeout; kill it
2017-01-06T13:58:03.303804+00:00 err: kernel: [ 139.137234] systemd-udevd[863]: seq 5192 '/devices/pci0000:80/0000:80:03.0/0000:83:00.0' killed
2017-01-06T13:58:03.303862+00:00 err: kernel: [ 139.137240] systemd-udevd[863]: worker [890] /devices/pci0000:80/0000:80:03.0/0000:83:00.1 timeout; kill it
2017-01-06T13:58:03.303920+00:00 err: kernel: [ 139.137244] systemd-udevd[863]: seq 5193 '/devices/pci0000:80/0000:80:03.0/0000:83:00.1' killed
2017-01-06T13:58:03.303976+00:00 err: kernel: [ 139.137250] systemd-udevd[863]: worker [893] /devices/pci0000:80/0000:80:03.2/0000:84:00.1 timeout; kill it
2017-01-06T13:58:03.304032+00:00 err: kernel: [ 139.137254] systemd-udevd[863]: seq 5198 '/devices/pci0000:80/0000:80:03.2/0000:84:00.1' killed

Interfaces to match with logs:

04:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
        Subsystem: Intel Corporation 10GbE 2P X520 Adapter
        Kernel driver in use: ixgbe
04:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
        Subsystem: Intel Corporation 10GbE 2P X520 Adapter
        Kernel driver in use: ixgbe
05:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
        Subsystem: Intel Corporation 10GbE 2P X520 Adapter
        Kernel driver in use: ixgbe
05:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
        Subsystem: Intel Corporation 10GbE 2P X520 Adapter
        Kernel driver in use: ixgbe
83:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
        Subsystem: Intel Corporation 10GbE 2P X520 Adapter
        Kernel driver in use: ixgbe
83:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
        Subsystem: Intel Corporation 10GbE 2P X520 Adapter
        Kernel driver in use: ixgbe
84:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
        Subsystem: Intel Corporation 10GbE 2P X520 Adapter
        Kernel driver in use: ixgbe
84:00.1 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
        Subsystem: Intel Corporation 10GbE 2P X520 Adapter
        Kernel driver in use: ixgbe

If additional information is required, please request it and I will provide it.

[Ivan Suzdal (isuzdal)]

Hi Michael!
Sorry for the late response.

First of all, I'd like to ask for remote access to affected environment, if it possible. It would simplify debugging process.

If not, I'd ask to attach /var/log/syslog, /var/log/kern.log, /var/log/udev, /etc/udev/udev.conf.
In additional to these log files, please execute next command "modinfo ixgbe 2>&1 > /tmp/ixgbe.info" and attach /tmp/ixgbe.info into bug.

[Dmitry Teselkin (teselkin-d)]

During debug session we found that systemd-udevd timed out because of multiple CPU stuck issues, like in the LP bug [1]. We tried bootstrap image with 4.4 kernel and it solved this issue, but network interfaces still were not configured by default. There was no configuration files in /etc/network/interfaces.d that describe any interface with 'iface XXX auto' so it's expected.

The reason why there is no configuration for interfaces looks not related to NIC card, kernel, but related to our configuration tools.

[1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1442706

[Alexander Rubtsov (arubtsov)]

> The reason why there is no configuration for interfaces looks not related to NIC card, kernel, but related to our configuration tools

It seems to be true - I was able to reproduce the same behavior even on my virtual lab.
I've filed a separate bug regarding this matter: https://bugs.launchpad.net/fuel/+bug/1655900

[Dmitry Teselkin (teselkin-d)]

There is an internal ticket SF#26739, so closing this bug as invalid.