HP NC375T / phanfw.bin: firmware is older than flash / Direct firmware load failed with error -2 / HP firmware is newer in cards

[JSE]

HP has a newer firmware on their NC375T server nics: 4.0.593
This generates an error at the startup of Ubuntu Server 14.04 LTS
The version of phanfw.bin seems to be older.

[ 16.573260] QLogic/NetXen Network Driver v4.0.82
[ 16.811117] netxen_nic 0000:07:00.0: 2MB memory map

[ 16.812455] netxen_nic 0000:07:00.0: phanfw.bin: firmware is older than flash
[ 16.818949] netxen_nic 0000:07:00.0: Direct firmware load failed with error -2
[ 16.818950] netxen_nic 0000:07:00.0: Falling back to user helper

[ 23.138390] netxen_nic 0000:07:00.0: loading firmware from flash
[ 27.586070] netxen_nic 0000:07:00.0: Gen2 strapping detected
[ 30.185964] netxen_nic 0000:07:00.0: using 64-bit dma mask
[ 30.373956] netxen_nic: Quad Gig LP Board S/N TI13BK1960 Chip rev 0x42
[ 30.450969] netxen_nic 0000:07:00.0: Driver v4.0.82, firmware v4.0.593 [legacy]
[ 39.718557] netxen_nic 0000:07:00.0: irq 67 for MSI/MSI-X
[ 39.718561] netxen_nic 0000:07:00.0: irq 68 for MSI/MSI-X
[ 39.718564] netxen_nic 0000:07:00.0: irq 69 for MSI/MSI-X
[ 39.718567] netxen_nic 0000:07:00.0: irq 70 for MSI/MSI-X
[ 39.718605] netxen_nic 0000:07:00.0: using msi-x interrupts
[ 39.780399] netxen_nic 0000:07:00.0: non ULA adapter
[ 39.841095] netxen_nic 0000:07:00.0: eth0: GbE port initialized
[ 39.901172] netxen_nic 0000:07:00.1: 2MB memory map
[ 39.959403] netxen_nic 0000:07:00.1: using 64-bit dma mask
[ 40.117279] netxen_nic 0000:07:00.1: Driver v4.0.82, firmware v4.0.593 [legacy]
[ 40.174172] netxen_nic 0000:07:00.1: irq 71 for MSI/MSI-X
[ 40.174175] netxen_nic 0000:07:00.1: irq 72 for MSI/MSI-X
[ 40.174178] netxen_nic 0000:07:00.1: irq 73 for MSI/MSI-X
[ 40.174181] netxen_nic 0000:07:00.1: irq 74 for MSI/MSI-X
[ 40.174209] netxen_nic 0000:07:00.1: using msi-x interrupts
[ 40.229992] netxen_nic 0000:07:00.1: eth1: GbE port initialized
[ 40.284766] netxen_nic 0000:07:00.2: 2MB memory map
[ 40.339836] netxen_nic 0000:07:00.2: using 64-bit dma mask
[ 40.465258] netxen_nic 0000:07:00.2: Driver v4.0.82, firmware v4.0.593 [legacy]
[ 40.522359] netxen_nic 0000:07:00.2: irq 75 for MSI/MSI-X
[ 40.522363] netxen_nic 0000:07:00.2: irq 76 for MSI/MSI-X
[ 40.522365] netxen_nic 0000:07:00.2: irq 77 for MSI/MSI-X
[ 40.522368] netxen_nic 0000:07:00.2: irq 78 for MSI/MSI-X
[ 40.522392] netxen_nic 0000:07:00.2: using msi-x interrupts
[ 40.578742] netxen_nic 0000:07:00.2: eth2: GbE port initialized
[ 40.634802] netxen_nic 0000:07:00.3: 2MB memory map
[ 40.690423] netxen_nic 0000:07:00.3: using 64-bit dma mask
[ 40.825233] netxen_nic 0000:07:00.3: Driver v4.0.82, firmware v4.0.593 [legacy]
[ 40.882915] netxen_nic 0000:07:00.3: irq 79 for MSI/MSI-X
[ 40.882919] netxen_nic 0000:07:00.3: irq 80 for MSI/MSI-X
[ 40.882922] netxen_nic 0000:07:00.3: irq 81 for MSI/MSI-X
[ 40.882925] netxen_nic 0000:07:00.3: irq 82 for MSI/MSI-X
[ 40.882951] netxen_nic 0000:07:00.3: using msi-x interrupts
[ 40.939825] netxen_nic 0000:07:00.3: eth3: GbE port initialized

# ethtool -i p4p1
driver: netxen_nic
version: 4.0.82
firmware-version: 4.0.593
bus-info: 0000:07:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

# ./nxflash.bin -i p4p1 --info
Board Type: 0x24 (NC375T PCI Express Quad Port Gigabit Server Adapter)
Board Chip Rev: B2
Serial Id: TI13BK1960
Efuse ID: 0x0000000000000000
Firmware version(on Flash): 4.0.593
Active Firmware version: 4.0.593
Driver version: 0.0.0
BIOS version: 2.1.0
MAC Addr 0: 78:AC:C0:10:F9:DC
Subsystem ID: 0x1740103c
PXE Status 0: Disabled
WOL Status 0: Disabled

# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C222 Series Chipset Family Server Essential SKU LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)
02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
03:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
04:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
07:00.0 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42)
07:00.1 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42)
07:00.2 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42)
07:00.3 Ethernet controller: NetXen Incorporated NX3031 Multifunction 1/10-Gigabit Server Adapter (rev 42)

---
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
DistroRelease: Ubuntu 14.04
InstallationDate: Installed on 2014-02-14 (116 days ago)
InstallationMedia: Ubuntu-Server 14.04 LTS "Trusty Tahr" - Alpha amd64 (20140214)
Package: linux (not installed)
Tags: trusty
Uname: Linux 3.15.0-031500-generic x86_64
UnreportableReason: The running kernel is not an Ubuntu kernel
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True
---
ApportVersion: 2.20.1-0ubuntu2.10
Architecture: amd64
Dependencies:

DistroRelease: Ubuntu 16.04
Package: linux-firmware 1.157.11
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 4.4.0-89.112-generic 4.4.76
Tags: xenial
Uname: Linux 4.4.0-89-generic x86_64
UpgradeStatus: Upgraded to xenial on 2016-12-14 (237 days ago)
UserGroups:

_MarkForUpload: True

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1327908

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[penalvch (penalvch)]

JSE, could you please boot into a default Ubuntu kernel (not a mainline one) and execute the following in a terminal:
apport-collect 1327908

[JSE (jse.nl)]

Christopher, unfortunately I can't, as these systems have an issue booting normally with kernel linux-image-3.13.0-29-generic. They are running perfectly well with 3.15.0-031500-generic.

Is there anything else I can do except booting up a 3.13 kernel?

The issue is with cards using the latest firmware, as made available by HP on November 19th 2013:

Products Affected
==================
HP NC510F PCIe 10 Gigabit Server Adapter
HP NC510C PCIe 10 Gigabit Server Adapter
HP NC522m Dual Port 10GbE Multifunction BL-c Adapter
HP NC522SFP Dual Port 10GbE Server Adapter
HP NC375i Quad Port Multifunction Gigabit Server Adapter
HP NC375i 1Gb w/HP NC524 10Gb
HP NC375T PCI Express Quad Port Gigabit Server Adapter

Pre-requisites
===============
HP NC-Series QLogic (nx_nic) driver version >= 4.0.540
Firmware version >= 3.4.336 for NC510C and NC510F
Firmware version >= 4.0.230 for all other Products

Description
============
Binaries for upgrading HP ProLiant QLogic P2 and P3 10 Gigabit
Ethernet NICs ROM images running under Linux x86_64 platform.

Enhancements/Fixes
===================
- Enhancements Description:
  - Firmware Enhancements:
   - This component upgrades NIC fimware v4.0.593.

- Fixes Description:
  - Firmware Fixes:
   - FW changes w.r.t to 4.0.590 :
      * Fix Receive Side Scaling distribution on W2K12 systems.
      * Fix incorrect reporting of checksum status for packets
         with TCP checksum error in promiscuous mode.

More on the support website of HP:

http://h20565.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdHome/?sp4ts.oid=4038767&spf_p.tpst=swdMain&spf_p.prp_swdMain=wsrp-navigationalState%3DswEnvOID%253D4103%257CswLang%253D%257Caction%253DlistDriver&javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken

[JSE (jse.nl)]

These systems are only running well on at least a 3.14-rc2 kernel, as mentioned in another bug report: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1280353

So I have to stay above 3.14-rc2 to have these systems working well.

[penalvch (penalvch)]

JSE, could you please test https://wiki.ubuntu.com/Kernel/MainlineBuilds and advise to the results?

[JSE (jse.nl)]

Christopher, the test are done with the latest mainline kernel 3.15.0-031500-generic as mentioned earlier. With that kernel I get the messages as above.

I think the netxen driver should be updated, so it can handle the latest firmware of HP too. The issue is there is no source code of the HP drivers available and the HP drivers are compiled for RedHat 4/5/6, Suse 10/11 and Solaris 10.

Anybody on the Ubuntu Kernel team doing the netxen driver (I guess: https://github.com/Canonical-kernel/Ubuntu-kernel/tree/master/drivers/net/ethernet/qlogic/netxen) who can find out what the 2 fixes by HP contain? I guess they should know at QLogic too right? Or it this only known at HP?
 * Fix Receive Side Scaling distribution on W2K12 systems.
 * Fix incorrect reporting of checksum status for packets with TCP checksum error in promiscuous mode.

(btw. the updated description was only the lspci info added now)

[JSE (jse.nl)]

Or am I wrong here and does this "phanfw.bin' belong to the linux-kernel team and do they need to update that file, so it has a newer version compatible with the HP cards?

[JSE (jse.nl)]

Is this helpful?

http://ldriver.qlogic.com/driver-srpms/nx_nic/

[JSE (jse.nl)]

I can't boot in any default 3.13 kernel (tried all: 3.13.0-24, 3.13.0-27 and 3.13.0-29), as the system doesn't have USB and network up and running. So I have no direct access with the keyboard (USB, there is no other port on these boards) and no remote access using SSH (as the network is not working properly). Booting in a 3.13 kernel gives me a system I can't access...

Any other suggestion?

[penalvch (penalvch)]

JSE, just to clarify, it would appear you have two separate problems:
1) The OS doesn't boot unless on the mainline kernels you mentioned previously?
2) The firmware provided by Ubuntu in the mainline kernel doesn't work at all, and you personally tested the latest version and it works on the mainline kernels?

Given this, it would be best to split this up into two different reports, a new report that focuses solely on the OS not booting with the default kernel, and the other being this one focusing only on the HP firmware.

[JSE (jse.nl)]

1) The system boots incorrectly on the 3.13 kernels. The USB keyboard is not working, neither is the Ethernet connection.
The systems boots correctly on 3.14-rc+ kernels, so also on 3.15 it works ok.
This is a separate issue, which I can setup as a separate bug.

With 14.04.1 around the corner (which is running a 3.15 kernel) I rather make it work with that one (but that is my personal choice, I understand that). The reason is the deployment contains a huge amount of these servers and for 1 month I better can focus on 3.15 timing wise.

So let's focus on the other part, where this bug report IS for.

2) The firmware provided by Ubuntu is older than the one from HP. Ubuntu 4.0.590, HP 4.0.593.
The system loads the firmware from the NC375T flash at boot and reports an error.
That firmware is also added to this bug as an attachment.

With this firmware from the card loaded at boot by the system, the card works. I tested that personally on multiple (exactly the same) systems. The main issue is that the firmware included in the mainline kernel is not up to date enough.

Hope I explained it more clearly and correctly else please shout.

[penalvch (penalvch)]

JSE, regarding just the firmware piece, this would be bundled as a separate package in Ubuntu (linux-firmware) which contains phanfw.bin.

The next step would be to test this out in a working Ubuntu kernel. Hence, feel free to subscribe me to the new bug report you create regarding booting up and the keyboard doesn't work, and we can take it from there.

[Tim Gardner (timg-tpi)]

JSE - you could request inclusion of the correct version by emailing <email address hidden>. See the README in git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git

[JSE (jse.nl)]

Still getting error:

netxen_nic 0000:07:00.0: Direct firmware load for nx3fwct.bin failed with error -2

on Ubuntu 16.04 (4.4.0-89-generic)

The NC375T is currently on firmware 4.0.596

Can someone solve this? Is more info needed?

[JSE (jse.nl)]

apport information

[JSE (jse.nl)]

apport information

[JSE (jse.nl)]

Executed in Ubuntu to flash the NIC

Installation of the package:
dpkg -i hp-firmware-nic-netxen_5.6.3-1.1_amd64.deb
cd /usr/lib/x86_64-linux-gnu/hp-firmware-nic-netxen-5.6.3-1.1

Gathering info (for checking purposes):
./nxflash.bin -i p2p1 --info

Actual flashing of the NIC:
./nxflash.bin -i p2p1 -a nic_fw/phantom_romimage

[Seth Forshee (sforshee)]

In order to distribute any of this firmware in Ubuntu we need the individual firmware files and an open-source compatible license from the vendor for redistribution of the firmware files. Lacking that we are not able to distribute the firmware with Ubuntu.

There is a version of phanfw.bin in the upstream linux-firmware repository, which is the version we ship in Ubuntu. According to the history this is version 4.0.590, last updated Apr 7 2013. There is no nx3fwct.bin file there.

The best thing would still be to get this firmware updated/included in upstream linux-firmware, but that will also require an appropriate license for redistribution.

Changing this to wishlist since as far as I know there's nothing we can do about it right now. If the firmware files become available with an appropriate license, please let me know.

[Juerg Haefliger (juergh)]

I'm closing this bug due to old age. Please open a new bug if you think the issue still exists.