Ubuntu 5.4.0-81 stops bnx2x loading hardware

[Pieter]

Server: HPE BL460c Gen9
Network: HP FlexFabric 10Gb 2-port 536FLB Adapter (Broadcom/Qlogic Net Extreme II 10/20GBE)

Using ubuntu 20.04.2 with linux kernel 5.4.0-80 all functionality worked however upon upgrades via apt to 20.04.3 which uses linux kernel 5.4.0-81, ubuntu is unable to load the drivers for the network device. Below is the output for both kernels when the system is booted.

###################################
##### Kernel 5.4.0-80 working #####
###################################

ubuntu@server:$ dmesg | grep bnx2x

[ 6.648833] bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.713.36-0 (2014/02/10)
[ 6.695219] bnx2x 0000:06:00.0: msix capability found
[ 6.719107] bnx2x 0000:06:00.0: part number 0-0-0-0
[ 7.112390] bnx2x 0000:06:00.0: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[ 7.112504] bnx2x 0000:06:00.1: msix capability found
[ 7.112889] bnx2x 0000:06:00.1: part number 0-0-0-0
[ 7.320779] bnx2x 0000:06:00.1: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[ 7.323549] bnx2x 0000:06:00.0 eno49: renamed from eth0
[ 7.343281] bnx2x 0000:06:00.1 eno50: renamed from eth1
[ 16.321102] bnx2x 0000:06:00.1 eno50: using MSI-X IRQs: sp 130 fp[0] 132 ... fp[7] 139
[ 16.770833] bnx2x 0000:06:00.1 eno50: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit
[ 17.136896] bnx2x 0000:06:00.0 eno49: using MSI-X IRQs: sp 120 fp[0] 122 ... fp[7] 129
[ 17.526831] bnx2x 0000:06:00.0 eno49: NIC Link is Up, 10000 Mbps full duplex, Flow control: ON - receive & transmit

ubuntu@server:$ ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno49: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 9e:b2:88:1c:ea:40 brd ff:ff:ff:ff:ff:ff
3: eno50: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 9e:b2:88:1c:ea:40 brd ff:ff:ff:ff:ff:ff
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9e:b2:88:1c:ea:40 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.68/24 brd 10.0.0.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet6 fe80::9cb2:88ff:fe1c:ea40/64 scope link
       valid_lft forever preferred_lft forever

ubuntu@server:$ sudo lshw -C network

  *-network:0
       description: Ethernet interface
       product: BCM57840 NetXtreme II 10/20-Gigabit Ethernet
       vendor: Broadcom Inc. and subsidiaries
       physical id: 0
       bus info: pci@0000:06:00.0
       logical name: eno49
       version: 11
       serial: 9e:b2:88:1c:ea:40
       size: 10Gbit/s
       capacity: 10Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msix pciexpress bus_master cap_list rom ethernet physical fibre 1000bt-fd 10000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=bnx2x driverversion=1.713.36-0 storm 7.13.11.0 duplex=full firmware=mbi 7.18.77 bc 7.15.80 latency=0 link=yes multicast=yes port=fibre slave=yes speed=10Gbit/s
       resources: irq:16 memory:94000000-947fffff memory:94800000-94ffffff memory:95010000-9501ffff memory:92b00000-92b7ffff
  *-network:1
       description: Ethernet interface
       product: BCM57840 NetXtreme II 10/20-Gigabit Ethernet
       vendor: Broadcom Inc. and subsidiaries
       physical id: 0.1
       bus info: pci@0000:06:00.1
       logical name: eno50
       version: 11
       serial: 9e:b2:88:1c:ea:40
       size: 10Gbit/s
       capacity: 10Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msix pciexpress bus_master cap_list rom ethernet physical fibre 1000bt-fd 10000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=bnx2x driverversion=1.713.36-0 storm 7.13.11.0 duplex=full firmware=mbi 7.18.77 bc 7.15.80 latency=0 link=yes multicast=yes port=fibre slave=yes speed=10Gbit/s
       resources: irq:17 memory:93000000-937fffff memory:93800000-93ffffff memory:95000000-9500ffff memory:92b80000-92bfffff
  *-network
       description: Ethernet interface
       physical id: 1
       logical name: bond0
       serial: 9e:b2:88:1c:ea:40
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=bonding driverversion=3.7.1 duplex=full firmware=2 ip=10.0.0.68 link=yes master=yes multicast=yes speed=10Gbit/s

###################################
##### Kernel 5.4.0-81 failure #####
###################################

ubuntu@server: $ dmesg | grep bnx2x

[ 6.605719] bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.713.36-0 (2014/02/10)
[ 6.726579] bnx2X 0000:06:00.0: msix capability found
[ 6.741708] bnx2x 0000:06:00.0: part number 0-0-0-0
[ 7.319238] bnx2x: probe of 0000:06:00.0 failed with error -22
[ 7.319268] bnx2x 0000:06:00.1: msix capability found
[ 7.319475] bnx2x 0000:06:00.1: part number 0-0-0-0
[ 7.542188] bnx2x: probe of 0000:06:00.1 failed with error -22

ubuntu@server:$ ip a

1: lo: <LOOPBACK, UP, LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 inet 127.0.0.1/8 scope host lo
  valid_ift forever preferred_ift forever
 inet6 ::1/128 scope host
  valid_lft forever preferred_lft forever
2: bond0: <NO-CARRIER, BROADCAST, MULTICAST, MASTER, UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
 link/ether 9e:62:88:1c:ea:40 brd ff:ff:ff:ff:ff:ff
 inet 10.0.0.68/24 brd 10.0.0.255 scope global bond0
  valid_ift forever preferred_ift forever

ubuntu@server:~$ sudo lshw -C network

*-network:0 UNCLAIMED
 description: Ethernet controller
 product: BCM57840 NetXtreme II 10/20-Gigabit Ethernet
 vendor: Broadcom Inc. and subsidiaries
 physical id: 0
 bus info: pcic0000:06:00.0
 version: 11
 width: 64 bits
 clock: 33MHZ
 capabilities: pm vpd msix pciexpress cap_list
 configuration: latency=0
 resources: memory:94000000-947fffff memory:94800000-94ffffff memory:95010000-9501ffff memory: 92b00000-92b7ffff
*-network:1 UNCLAIMED
 description: Ethernet controller
 product: BCM57840 NetXtreme II 10/20-Gigabit Ethernet
 vendor: Broadcom Inc. and subsidiaries
 physical id: 0.1
 bus info: pci@0000:06:00.1
 version: 11
 width: 64 bits
 clock: 33MHZ
 capabilities: pm vpd msix pciexpress cap_list
 configuration: latency=0
 resources: memory:93000000-937fffff memory:93800000-93ffffff memory:95000000-9500ffff memory:92b80000-92bfffff
*-network
 description: Ethernet interface
 physical id: 1
 logical name: bond0
 serial: 9e:52:88:10:ea:40
 capabilities: ethernet physical
 configuration: autonegotiation=off broadcast=yes driver=bonding driverversion=3.7.1 firmware=2 ip=10.0.0.68 link=no maste
 r=yes multicast=yes
ubuntu@qs47:~$

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: linux-image-5.4.0-81-generic 5.4.0-81.91
ProcVersionSignature: Ubuntu 5.4.0-80.90-generic 5.4.124
Uname: Linux 5.4.0-80-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Sep 3 08:01 seq
 crw-rw---- 1 root audio 116, 33 Sep 3 08:01 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu27.18
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CasperMD5CheckResult: pass
Date: Fri Sep 3 08:05:06 2021
InstallationDate: Installed on 2021-09-01 (1 days ago)
InstallationMedia: Ubuntu-Server 20.04.2 LTS "Focal Fossa" - Release amd64 (20210201.2)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: HP ProLiant BL460c Gen9
PciMultimedia:

ProcEnviron:
 SHELL=/bin/bash
 LANG=en_GB.UTF-8
 TERM=xterm-256color
 PATH=(custom, no user)
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.4.0-80-generic root=/dev/mapper/maas-root ro
RelatedPackageVersions:
 linux-restricted-modules-5.4.0-80-generic N/A
 linux-backports-modules-5.4.0-80-generic N/A
 linux-firmware 1.187.16
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 10/16/2020
dmi.bios.vendor: HP
dmi.bios.version: I36
dmi.chassis.type: 28
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrI36:bd10/16/2020:svnHP:pnProLiantBL460cGen9:pvr:cvnHP:ct28:cvr:
dmi.product.family: ProLiant
dmi.product.name: ProLiant BL460c Gen9
dmi.product.sku: 681844-B21
dmi.sys.vendor: HP

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Martin Jansa (martin-jansa)]

Discussion about a fix for this in upstream:
https://lkml.org/lkml/2021/9/13/432

[Martin Jansa (martin-jansa)]

Looks like this might be fixed in 5.4.0-90 with:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1945707

[Martin Jansa (martin-jansa)]

I've tested 5.4.0-90 from focal-proposed and the issue is no longer reproducible and bnx2x works as in 5.4.0-80 before:

$ uname -a
Linux jackrabbit-build 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ dmesg | grep bnx
[ 2.227656] bnx2x: QLogic 5771x/578xx 10/20-Gigabit Ethernet Driver bnx2x 1.713.36-0 (2014/02/10)
[ 2.227942] bnx2x 0000:06:00.0: msix capability found
[ 2.228274] bnx2x 0000:06:00.0: part number 0-0-0-0
[ 2.366450] bnx2x 0000:06:00.0: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[ 2.366558] bnx2x 0000:06:00.1: msix capability found
[ 2.366874] bnx2x 0000:06:00.1: part number 0-0-0-0
[ 2.502096] bnx2x 0000:06:00.1: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[ 2.502201] bnx2x 0000:06:00.2: msix capability found
[ 2.502452] bnx2x 0000:06:00.2: part number 0-0-0-0
[ 2.629572] bnx2x 0000:06:00.2: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[ 2.629679] bnx2x 0000:06:00.3: msix capability found
[ 2.629930] bnx2x 0000:06:00.3: part number 0-0-0-0
[ 2.753882] bnx2x 0000:06:00.3: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[ 2.753987] bnx2x 0000:06:00.4: msix capability found
[ 2.754226] bnx2x 0000:06:00.4: part number 0-0-0-0
[ 2.881840] bnx2x 0000:06:00.4: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[ 2.881946] bnx2x 0000:06:00.5: msix capability found
[ 2.882202] bnx2x 0000:06:00.5: part number 0-0-0-0
[ 3.010083] bnx2x 0000:06:00.5: 63.008 Gb/s available PCIe bandwidth (8 GT/s x8 link)
[ 3.013799] bnx2x 0000:06:00.1 eno50: renamed from eth1
[ 3.044278] bnx2x 0000:06:00.0 eno49: renamed from eth0
[ 3.076775] bnx2x 0000:06:00.2 eno51: renamed from eth2
[ 3.108745] bnx2x 0000:06:00.4 eno53: renamed from eth4
[ 3.140563] bnx2x 0000:06:00.3 eno52: renamed from eth3
[ 3.164303] bnx2x 0000:06:00.5 eno54: renamed from eth5