qla2xxx takes ~one minute to initialize per device

[Timo Aaltonen]

I've got a HP BL460c blade server, with QLogic HBA's:

13:00.0 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
13:00.1 Fibre Channel: QLogic Corp. ISP2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)

but it takes roughly one minute to initialize per device, which means that the system drops in initramfs before the rootfs is mounted (rootdelay defaults to 30s).

[Timo Aaltonen (tjaalton)]

Ok, got a reply from QLogic suggesting that it's a bug in udev:

"This is an issue with the initrd (initramfs) infrastructure not
supporting the request_firmware() interface. Basically, the 60 second
lag time is the request_firmware() call timing out due to udev being
unable to satisfy the call to load firmware:

        [ 3.031797] qla2xxx 0000:13:00.0: Verifying loaded RISC code...[
        [ 3.050062] qla2xxx 0000:13:00.0: firmware: requesting ql2400_fw.bin
        ...
        [ 63.040094] scsi(0): Failed to load firmware image (ql2400_fw.bin).
        [ 63.040160] qla2xxx 0000:13:00.0: Firmware image unavailable.

If you are going to continue using the distro, I'd suggest you load
the driver after the initrd image has loaded and the root-FS has been
mounted."

[Scott James Remnant (Canonical) (canonical-scott)]

We deliberately don't support firmware loading in the initramfs right now, because it would almost double the size of it!

You can trivially add support by creating your own file in /etc/initramfs-tools/hooks that copies /lib/udev/firmware.sh and the /lib/firmware/ql2400_fw.bin file into the initramfs

[Timo Aaltonen (tjaalton)]

That sounds ok, if it only worked :)

rogue tmp # mkdir foo
rogue tmp # cd foo/
rogue foo # zcat /boot/initrd.img-2.6.28-7-server |cpio -id
rogue foo # ls -l lib/udev/firmware.sh lib/firmware/ql2400_fw.bin
-rw-r--r-- 1 root root 206500 2009-02-13 09:34 lib/firmware/ql2400_fw.bin
-rwxr-xr-x 1 root root 654 2009-02-13 09:34 lib/udev/firmware.sh*

the driver still fails to load the firmware.

[Scott James Remnant (Canonical) (canonical-scott)]

On Fri, 2009-02-13 at 07:41 +0000, Timo Aaltonen wrote:

> That sounds ok, if it only worked :)
>
> rogue tmp # mkdir foo
> rogue tmp # cd foo/
> rogue foo # zcat /boot/initrd.img-2.6.28-7-server |cpio -id
> rogue foo # ls -l lib/udev/firmware.sh lib/firmware/ql2400_fw.bin
> -rw-r--r-- 1 root root 206500 2009-02-13 09:34 lib/firmware/ql2400_fw.bin
> -rwxr-xr-x 1 root root 654 2009-02-13 09:34 lib/udev/firmware.sh*
>
> the driver still fails to load the firmware.
>
Sorry, you also need to copy lib/udev/rules.d/50-firmware.rules ;)

Scott
--
Scott James Remnant
<email address hidden>

[Timo Aaltonen (tjaalton)]

Thanks! Now it boots up in 18s to gdm :)

before:
http://users.tkk.fi/~tjaalton/foo/jaunty-20090213-1.png
after:
http://users.tkk.fi/~tjaalton/foo/jaunty-20090213-2.png

[Andy Hauser (andy-ubuntu-bugzilla)]

There should be an easier way to get this working.
Make it wishlist?

[Scott James Remnant (Canonical) (canonical-scott)]

On Wed, 2009-07-15 at 14:35 +0000, Andy Hauser wrote:

> There should be an easier way to get this working.
> Make it wishlist?
>
At this time, it's not something we support.

Scott
--
Scott James Remnant
<email address hidden>

[Andy Hauser (andy-ubuntu-bugzilla)]

Hence the wishlist?

[Scott James Remnant (Canonical) (canonical-scott)]

On Wed, 2009-07-15 at 15:10 +0000, Andy Hauser wrote:

> Hence the wishlist?
>
My personal preference is to close wishlist bugs that we don't intend to
address.

Scott
--
Scott James Remnant
<email address hidden>

[Jens Maus (jens.maus)]

Let me add that I would also be glad if the firmware loading mechanism could be directly integrated into Ubuntu without having to create an own initramfs-tools hooks. However, as many people were already asking in other fora how to create such a hook, here is the one I created as /etc/initdramfs-tools/hooks/ql2xxx:

-- cut here --
#!/bin/sh -e
# initramfs hook for ql2xxx firmware loading

PREREQ="udev"

# Output pre-requisites
prereqs()
{
   echo "$PREREQ"
}

case "$1" in
    prereqs)
   prereqs
   exit 0
   ;;
esac

. /usr/share/initramfs-tools/hook-functions

mkdir ${DESTDIR}/lib/firmware
cp -p /lib/firmware/ql2400_fw.bin ${DESTDIR}/lib/firmware/
cp -p /lib/udev/firmware.sh ${DESTDIR}/lib/udev/
cp -p /lib/udev/rules.d/50-firmware.rules ${DESTDIR}/lib/udev/rules.d
-- cut here --

Make sure to create that file with executable right and then run "update-initramfs -u" to recreate the initrd file. In addition, please note that this hook only copies the ql2400_fw.bin file. So if you have another card replace that in the hooks file.

[Thomy23 (thomas-stather)]

Hi

Is there any reason why this worked out of the box in Hardy LTS (8.04.2) 64 Bit? Heres the dmesg:

[ 4.923483] qla2xxx 0000:03:00.0: Found an ISP2432, irq 16, iobase 0xffffc2000003e000
[ 4.924118] qla2xxx 0000:03:00.0: Configuring PCI space...
[ 4.927146] qla2xxx 0000:03:00.0: Configure NVRAM parameters...
[ 4.963535] qla2xxx 0000:03:00.0: Verifying loaded RISC code...
[ 1.395643] qla2xxx 0000:03:00.0: Allocated (64 KB) for EFT...
[ 1.396014] qla2xxx 0000:03:00.0: Allocated (1413 KB) for firmware dump...
[ 1.411674] qla2xxx 0000:03:00.0: MSI-X: Failed to enable support -- 2/-1
[ 1.411811] qla2xxx 0000:03:00.0: MSI-X: Falling back-to INTa mode -- -1.
[ 1.411949] scsi1 : qla2xxx
[ 1.412394] qla2xxx 0000:03:00.0:
[ 5.756065] qla2xxx 0000:03:00.0: LOOP UP detected (4 Gbps).

Seems that the FW was here loaded from the initramfs or am i wrong?

Greets Thomas

[Matt Pitts (mpitts)]

RE: Comment #11. Jens Langner you are awesome! I just got bit by this problem on a fresh Jaunty server install on an HP BL35 blade. Your hook script worked beautifully. Thanks!

[Rob Dyke (emailme-robdyke)]

RE: Comment #11. Thanks Jens. I'm adding this into my preseed for 9.x server builds this evening.

I'd like to add my voice to this bug as 'me too'. It is pesky and frustrating to have to work this out for the 'minor' releases. I hope we'll see the firmware included in the next LTS... or perhaps some hooks in advanced/server builds for specific/detected cards?

[Rob Dyke (emailme-robdyke)]

RE: Comment #11. Jens Langner. Many thanks Jens for your helpful script. I've not quite got it integrated with my preseed, but I confirm your script works well for me.

[Rob Dyke (emailme-robdyke)]

My pressed late_command for including Jens's script

d-i preseed/late_command string mkdir -p /target/etc/initdramfs-tools/hooks/ql2xxx ; wget http://<my webserver>/files/ql2xxx -o /target/etc/initdramfs-tools/hooks/ql2xxx ; chmod +x /target/etc/initdramfs-tools/hooks/ql2xxx ; in-target update-initramfs -u

[Steve Langasek (vorlon)]

I'm reopening this bug. This is not a wishlist at all; it's a significant regression for users of this hardware, that will affect both new installs and LTS->LTS upgrades. While I agree that doubling the size of all initramfses unreasonably penalizes users without this hardware, we do need to have a solution that works out of the box for users /with/ this hardware - if always including this firmware in the initramfs isn't reasonable, then we should entertain other options, such as using a different policy for server kernels or opportunistically adding the firmware to the initramfs *if* it's detected that the module is needed on the hardware where the initramfs is built (or if it's detected that the root device uses it).

Now, in the udev in Ubuntu 9.10, I see that /usr/share/initramfs-tools/hooks/udev *does* include the following code:

for rules in 50-udev-default.rules 50-firmware.rules 60-persistent-storage.rules 61-persistent-storage-edd.rules 64-device-mapper.rules 80-drivers.rules 95-udev-late.rules; do
    cp -p /lib/udev/rules.d/$rules ${DESTDIR}/lib/udev/rules.d
done
[...]
copy_exec /lib/udev/firmware.sh /lib/udev

So I don't believe udev in 9.10 is actually buggy - the firmware loading hooks are present in the initramfs!

It looks like only initramfs-tools itself has a bug, because only firmware for modules that are explicitly singled out in /usr/share/initramfs-tools/hook-functions are included in the initramfs. There, firmware is *only* copied if it's found in the /lib/modules/$(uname -r) directory - not if it's found in /lib/firmware. This is inconsistent, and appears to be a straightforward bug due to an oversight rather than a design decision.

Correcting this pulls in only 6 more firmware files, adds 684k (unpacked; 315k packed) to the initramfs size, and the change is also included already in the newer version of the update-initramfs package upstream in Debian. So I think this is the obviously correct change to make here.

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.92bubuntu54

---------------
initramfs-tools (0.92bubuntu54) lucid; urgency=low

  * hook-functions: check both /lib/firmware/$(uname -r) and /lib/firmware
    for the firmware that we wish to copy; in any cases where we want to
    suppress a firmware-using module from the initramfs, we should do that
    explicitly and not by accidentally failing to include the firmware.
    LP: #328550.
  * hook-functions: cherry pick fix from Debian upstream version to not warn
    about missing firmware for modules whose names are merely /like/ those of
    modules listed in /proc/modules.
 -- Steve Langasek <email address hidden> Thu, 19 Nov 2009 00:47:55 -0600

[Tim Gardner (timg-tpi)]

This does not appear to be a linux-firmware bug.