Ubuntu

Doesn't include qla2xxx firmware

Reported by Matt Zimmerman on 2006-12-01
14
Affects Status Importance Assigned to Milestone
initramfs-tools (Ubuntu)
High
Ben Collins
Dapper
Wishlist
Ben Collins
Edgy
High
Matt Zimmerman
Feisty
High
Ben Collins
linux-source-2.6.19 (Ubuntu)
High
Ben Collins
Dapper
Undecided
Unassigned
Feisty
High
Ben Collins
udev (Ubuntu)
High
Kyle McMartin
Dapper
High
Ben Collins
Edgy
Undecided
Unassigned
Feisty
High
Kyle McMartin

Bug Description

The qla2xxx driver is included in the initramfs, but the firmware upon which it depends is not. This causes this driver to be non-functional for all Edgy users; in a case where it is used for the root filesystem it would cause the system to fail to boot.

This has been addressed by patching udev to ship the firmware helper in initramfs and initramfs-tools to include the firmware in initramfs.

Matt Zimmerman (mdz) wrote :

Assigning to me since I've written a patch already

Changed in initramfs-tools:
assignee: nobody → mdz
importance: Undecided → High
status: Unconfirmed → Confirmed
Matt Zimmerman (mdz) wrote :
Matt Zimmerman (mdz) wrote :

Colin, please review for SRU

Changed in initramfs-tools:
importance: Undecided → High
status: Unconfirmed → Confirmed
Matt Zimmerman (mdz) wrote :

Assigning non-backport task to Ben

Changed in initramfs-tools:
assignee: nobody → ben-collins
Matt Zimmerman (mdz) wrote :

This fix also requires that udev provide firmware_helper in the initramfs

Matt Zimmerman (mdz) wrote :

This fix also requires that udev provide firmware_helper and the corresponding rules in the initramfs

Matt Zimmerman (mdz) wrote :
Ben Collins (ben-collins) wrote :

Added an initramfs-tools hook script in the kernel. Versioned filename, and checks to make sure that the version of kernel it is being run for is the one it was installed for. Should keep the script cleaner, and keep us from having to retain backward compatibility in it.

Changed in initramfs-tools:
status: Confirmed → Fix Committed
Matt Zimmerman (mdz) wrote :

initramfs-tools uploaded to edgy-proposed

Changed in udev:
importance: Undecided → High
status: Unconfirmed → Confirmed
Colin Watson (cjwatson) wrote :

I opened an initramfs-tools task for the development trunk, but on reflection I'm not sure that that's needed if it's going to be fixed in the kernel instead. Please reject this task if it's not needed.

Changed in initramfs-tools:
importance: Undecided → High
status: Unconfirmed → Confirmed
Colin Watson (cjwatson) wrote :

initramfs-tools accepted into edgy-proposed. I'm going to wait to add verification-needed tags etc. until the udev fix is also available.

description: updated
Changed in initramfs-tools:
status: Confirmed → Fix Committed
Ben Collins (ben-collins) wrote :

initramfs-tools doesn't like my idea of using hooks files with kernel version name as part of the file name. It seems it evals this filename for a var name.

Way too much work to fix that in initramfs, so I'll just dupe what Matt did in edgy.

Changed in linux-source-2.6.19:
status: Fix Committed → Rejected
Ben Collins (ben-collins) wrote :

Added a more robust fix for feisty's initramfs.

Changed in initramfs-tools:
assignee: nobody → ben-collins
status: Confirmed → Fix Committed
Ben Collins (ben-collins) wrote :
Matt Zimmerman (mdz) wrote :

I'm awaiting confirmation from the support team that udev fixed the customer issue before uploading it. initramfs-tools was confirmed to do the right thing (it just needed to get some files into the initramfs)

Matt,

The client have confirmed that the combination kernel + initramfs-tools + udev fixes solve his problem. The FC controller is now recognized at boot, and file systems attached to it mount fine without manual intervention.

Thanks a lot for the prompt response, guys.

Matt Zimmerman (mdz) wrote :

I've uploaded udev to edgy-proposed.

SRU team: please consider these two uploads together as an SRU. I've updated the description accordingly.

description: updated
Matt Zimmerman (mdz) wrote :

Scott, please merge this fix into Feisty udev

Tollef Fog Heen (tfheen) wrote :

udev accepted into edgy-proposed. Please notify the QA team and solicit testing.

Changed in udev:
status: Unconfirmed → Fix Committed
Changed in initramfs-tools:
status: Fix Committed → Fix Released
status: Fix Committed → Fix Released
Simon Law (sfllaw) wrote :

Regression tested to not break Edgy. udev is ready for immediate upload to edgy-updates.

Martin Pitt (pitti) wrote :

Finishing off SRU for mdz:

 udev (093-0ubuntu18.0edgy2) edgy-updates; urgency=low
 .
   * No-change upload to edgy-updates, thanks to Simon Law for QA testing!
 .
 udev (093-0ubuntu18.0edgy1) edgy-proposed; urgency=low
 .
   * Include firmware_helper in initramfs (LP: #74004)

Changed in udev:
status: Fix Committed → Fix Released
Martin Pitt (pitti) wrote :

Closing feisty task for udev, since Ben fixed it in initramfs-tools.

Changed in udev:
status: Confirmed → Rejected
Martin Kreiner (languste) wrote :

Well what about backporting these fixes to 6.06 LTS?

martin

p.s. Need LTS and IBM Blades with Qlogic Fibre Channel Cards won't work in dapper (out of the box)

Changed in initramfs-tools:
assignee: nobody → ben-collins
importance: Undecided → High
status: Unconfirmed → Confirmed
Changed in linux-source-2.6.19:
status: Unconfirmed → Rejected
Changed in udev:
assignee: nobody → ben-collins
importance: Undecided → High
status: Unconfirmed → Confirmed
Martin Pitt (pitti) wrote :

Reassigning to Kyle, as per Ben's request.

Changed in udev:
assignee: ben-collins → kyle
Changed in initramfs-tools:
assignee: ben-collins → kyle
Matt Zimmerman (mdz) wrote :

This hasn't been fixed in Feisty yet: he firmware is included, but the helper program and udev script are not.

Please make sure this is fixed for the next milestone; it is a regression from stable.

Changed in udev:
assignee: nobody → kyle
status: Rejected → Confirmed
Kyle McMartin (kyle) wrote :

Looks like your patch never made it into udev, mdz. Uploaded.

Changed in udev:
status: Confirmed → Fix Released
Martin Pitt (pitti) wrote :

Kyle, can you please give a status update on this? Is it fixed in Feisty? is it still worth considering this for dapper?

Martin Pitt (pitti) wrote :

Kyle confirms that this is an outstanding issue in dapper, which would make it work on more hardware, and no problem to fix.

Changed in initramfs-tools:
importance: High → Wishlist
status: Confirmed → Triaged
Timo Aaltonen (tjaalton) wrote :

Yes, we hit this today when installed dapper on a HP 380G5. Please provide an update.

Changed in udev:
assignee: kyle → ben-collins
status: Confirmed → Fix Committed
Changed in initramfs-tools:
assignee: kyle → ben-collins
status: Triaged → Fix Committed
Martin Pitt (pitti) wrote :

 udev (079-0ubuntu35) dapper-proposed; urgency=low
 .
   * Include firmware_helper in initramfs
     - LP: #74004

 initramfs-tools (0.40ubuntu33) dapper-proposed; urgency=low
 .
   * Add megaraid_sas to initrd:
     - LP: #56854
   * Include qla firmware in initrd
     - LP: #74004

Accepted into dapper-proposed. Can you please test the package in dapper-proposed once it is available and give feedback here? Thank you!

* Martin Pitt

> udev (079-0ubuntu35) dapper-proposed; urgency=low
> .
> * Include firmware_helper in initramfs
> - LP: #74004
>
> initramfs-tools (0.40ubuntu33) dapper-proposed; urgency=low
> .
> * Add megaraid_sas to initrd:
> - LP: #56854
> * Include qla firmware in initrd
> - LP: #74004
>
> Accepted into dapper-proposed. Can you please test the package in
> dapper-proposed once it is available and give feedback here? Thank you!

Just upgraded as follows:

initramfs-tools 0.40ubuntu32 -> 0.40ubuntu34
udev 079-0ubuntu34 -> 079-0ubuntu35

The generated initramfs does contain firmware_helper, but NOT the QLogic
firmware:

root@cuba:~# zcat /boot/initrd.img-2.6.23-rc3lp1 | cpio -t | \
  egrep '(ql2400_fw.bin|firmware_helper)'
lib/udev/firmware_helper
230387 blocks

In fact /lib/firmware isn't present in the initramfs at all.

For reference, my hook-script that has done the job up until now contains:

[....]
# You can do anything you need to from here on.

# the filter-line in lvm.conf is important in case non-RDAC mode
mkdir -p $DESTDIR/etc/lvm
cp /etc/lvm/lvm.conf $DESTDIR/etc/lvm

# Qlogic firmware is needed
copy_exec /lib/udev/firmware_helper /lib/udev
cp -a /lib/firmware $DESTDIR/lib

...which reminded me of the missing lvm.conf and LP #113515. There's a
patch for this available at <http://bugs.debian.org/439761>. Wonder if
that one could be fixed in Dapper too...

Regards
--
Tore Anderson

Matt Zimmerman (mdz) wrote :

On Mon, Oct 22, 2007 at 09:46:57AM -0000, Tore Anderson wrote:
> * Martin Pitt
>
> > udev (079-0ubuntu35) dapper-proposed; urgency=low
> > .
> > * Include firmware_helper in initramfs
> > - LP: #74004
> >
> > initramfs-tools (0.40ubuntu33) dapper-proposed; urgency=low
> > .
> > * Add megaraid_sas to initrd:
> > - LP: #56854
> > * Include qla firmware in initrd
> > - LP: #74004
> >
> > Accepted into dapper-proposed. Can you please test the package in
> > dapper-proposed once it is available and give feedback here? Thank you!
>
> Just upgraded as follows:
>
> initramfs-tools 0.40ubuntu32 -> 0.40ubuntu34
> udev 079-0ubuntu34 -> 079-0ubuntu35
>
> The generated initramfs does contain firmware_helper, but NOT the QLogic
> firmware:
>
> root@cuba:~# zcat /boot/initrd.img-2.6.23-rc3lp1 | cpio -t | \
> egrep '(ql2400_fw.bin|firmware_helper)'
> lib/udev/firmware_helper
> 230387 blocks
>
> In fact /lib/firmware isn't present in the initramfs at all.

It looks like you're running a custom kernel. The firmware provided with
the Ubuntu kernel is in /lib/firmware/<kernel version>, and that's where
initramfs-tools looks for it. Perhaps your firmware is in a different
place?

> For reference, my hook-script that has done the job up until now
> contains:

Your script copies all of /lib/firmware, while the one in the fixed
initramfs-tools copies only the firmware shipped with our kernel, so that
seems likely to be the cause of your problem.

> ...which reminded me of the missing lvm.conf and LP #113515. There's a
> patch for this available at <http://bugs.debian.org/439761>. Wonder if
> that one could be fixed in Dapper too...

Seems worth a look.

--
 - mdz

Tore Anderson (toreanderson) wrote :

* Matt Zimmerman

> It looks like you're running a custom kernel. The firmware provided
> with the Ubuntu kernel is in /lib/firmware/<kernel version>, and
> that's where initramfs-tools looks for it. Perhaps your firmware is
> in a different place?

You're right I run a custom kernel (need it for mptsas and dm-rdac).
Anyway, the QLogic firmware image is kernel version agnostic (and not
shipped in the Ubuntu kernel package either, for that matter), so it's
found in /lib/firmware/ql2400_fw.bin. The firmware_helper looks for it
there out of the box.

> Your script copies all of /lib/firmware, while the one in the fixed
> initramfs-tools copies only the firmware shipped with our kernel, so
> that seems likely to be the cause of your problem.

Maybe you should make it so that initramfs-tools copies in kernel
version agnostic firmware images as well as the kernel version specific
ones? That way it it will work the same during initramfs and after
boot, which is how I'd expect it at least.

I could of course make /lib/firmware/2.6.23-rc3lp1 and put the image
there but I'll surely forget to copy it to /lib/firmware/$new_version
when I upgrade the kernel. :-)

Regards
--
Tore Anderson

Matt Zimmerman (mdz) wrote :

On Mon, Oct 22, 2007 at 10:53:13AM -0000, Tore Anderson wrote:
> * Matt Zimmerman
> > It looks like you're running a custom kernel. The firmware provided
> > with the Ubuntu kernel is in /lib/firmware/<kernel version>, and
> > that's where initramfs-tools looks for it. Perhaps your firmware is
> > in a different place?
>
> You're right I run a custom kernel (need it for mptsas and dm-rdac).
> Anyway, the QLogic firmware image is kernel version agnostic (and not
> shipped in the Ubuntu kernel package either, for that matter), so it's
> found in /lib/firmware/ql2400_fw.bin. The firmware_helper looks for it
> there out of the box.

In this case, that may be true, but firmware is often tied to the version of
the driver, so we always bind them together. It's happened in the past that
firmware changes incompatibly across releases of a driver.

> > Your script copies all of /lib/firmware, while the one in the fixed
> > initramfs-tools copies only the firmware shipped with our kernel, so
> > that seems likely to be the cause of your problem.
>
> Maybe you should make it so that initramfs-tools copies in kernel
> version agnostic firmware images as well as the kernel version specific
> ones? That way it it will work the same during initramfs and after
> boot, which is how I'd expect it at least.

I agree, that seems intuitively correct and it should be straightforward to
add support for this. Changing behaviour like this would be better done in
the development branch, though, rather than in an update to 6.06 LTS.

--
 - mdz

Tore Anderson (toreanderson) wrote :

* Matt Zimmerman

> In this case, that may be true, but firmware is often tied to the
> version of the driver, so we always bind them together. It's
> happened in the past that firmware changes incompatibly across
> releases of a driver.

I know. I don't think that's the case for the QLogic driver, though.

> I agree, that seems intuitively correct and it should be
> straightforward to add support for this. Changing behaviour like
> this would be better done in the development branch, though, rather
> than in an update to 6.06 LTS.

I don't understand this concern. Currently 6.06 LTS does not load
firmware from the initramfs _at all_. If you're going to add support
for that, you might just as well add the support in a completely
«intuitively correct» manner, no?

--
Tore Anderson

Matt Zimmerman (mdz) wrote :

On Mon, Oct 22, 2007 at 11:40:12AM -0000, Tore Anderson wrote:
> * Matt Zimmerman
> > I agree, that seems intuitively correct and it should be
> > straightforward to add support for this. Changing behaviour like
> > this would be better done in the development branch, though, rather
> > than in an update to 6.06 LTS.
>
> I don't understand this concern. Currently 6.06 LTS does not load
> firmware from the initramfs _at all_. If you're going to add support
> for that, you might just as well add the support in a completely
> «intuitively correct» manner, no?

This bug is being fixed by backporting the change which was tested in Ubuntu
7.04 and 7.10.

There's no telling what sort of random garbage folks have lying around in
/lib/firmware. Which one should be preferred, theirs or ours? What happens
if they're using a custom hook to get the firmware they want?

Such a change shouldn't be rolled out without proper testing; the minimal
change used in the update is more appropriate for a stable update.

--
 - mdz

Tore Anderson (toreanderson) wrote :

* Matt Zimmerman

> This bug is being fixed by backporting the change which was tested in
> Ubuntu 7.04 and 7.10.

Okay.

> There's no telling what sort of random garbage folks have lying
> around in /lib/firmware. Which one should be preferred, theirs or
> ours? What happens if they're using a custom hook to get the
> firmware they want?

Copy over both images to the same locations they're found on the root
file system and you'll get the same behaviour in initramfs as when the
system is finished booting. I don't know where firmware_helper looks
first, but as long as the behaviour is consistent, does it really matter?

> Such a change shouldn't be rolled out without proper testing; the
> minimal change used in the update is more appropriate for a stable
> update.

That change won't affect people that upgrade from 6.06 LTS from current
to 6.06 LTS-next-update, nor people that upgrade from 6.06 LTS to
8.whatever LTS as long as it's changed there too.

But it will cause a regression when going from 6.06 to 7.04 though
unless it is changed there too, that's worth a thought I guess.

Regards
--
Tore Anderson

Matt Zimmerman (mdz) wrote :

On Mon, Oct 22, 2007 at 12:49:03PM -0000, Tore Anderson wrote:
> * Matt Zimmerman
> > There's no telling what sort of random garbage folks have lying
> > around in /lib/firmware. Which one should be preferred, theirs or
> > ours? What happens if they're using a custom hook to get the
> > firmware they want?
>
> Copy over both images to the same locations they're found on the root
> file system and you'll get the same behaviour in initramfs as when the
> system is finished booting. I don't know where firmware_helper looks
> first, but as long as the behaviour is consistent, does it really matter?

Good point; this shouldn't be much of a concern. Perhaps the worst that
could happen (barring implementation bugs) is that some amount of irrelevant
data is copied into initramfs.

I'm still inclined to be conservative about this; loading of drivers can be
finicky, and even a change which is demonstrably correct can break working
systems in the field.

--
 - mdz

Timo Aaltonen (tjaalton) wrote :

I've tested the new packages and they work ok. Thanks!

Tore Anderson (toreanderson) wrote :

* Matt Zimmerman

> I'm still inclined to be conservative about this; loading of drivers
> can be finicky, and even a change which is demonstrably correct can
> break working systems in the field.

Well, it's your call. It's not a big deal for me - I can happily live
with my old initramfs-hook for as long as it takes. :-)

Anyway, done testing now. I stuffed the firmware image in the expected
place at /lib/firmware/`uname -r` and it was copied to the initramfs,
and loaded correctly during bootup. So the fix is good.

Regards
--
Tore Anderson

Martin Pitt (pitti) wrote :

udev copied to dapper-updates. initramfs-tools is still stalled for verification of bug 31035.

Changed in udev:
status: Fix Committed → Fix Released
Martin Pitt (pitti) wrote :

Copied to dapper-updates.

Changed in initramfs-tools:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.