bootstrap image is missing some firmware files

Bug #1323354 reported by Vladimir Kuklin on 2014-05-26
40
This bug affects 8 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Critical
Dmitry Burmistrov
4.1.x
High
Registry Administrators
5.0.x
Critical
Dmitry Burmistrov

Bug Description

We are using bootstrap image based on 3.10 kernel which uses so-called kernel-lt-firmware packages which is built not from this source: https://kernel.googlesource.com/pub/scm/linux/kernel/git/firmware/linux-firmware/ but from in-kernel tree. This leads to issues with some hardware, e.g. with some modifications of Dell R410/R610s servers. We need to build firmware packages from this source instead of relying on upstream in-tree files.

Mike Scherbakov (mihgen) on 2014-05-26
tags: added: customer-found
OSCI Robot (oscirobot) wrote :

Package linux-firmware has been built from changeset: http://gerrit.mirantis.com/15911
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-15911/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-15911/centos"

Changed in fuel:
status: Confirmed → In Progress
Meg McRoberts (dreidellhasa) wrote :

Documented in "Known Issues" for 5.0

Miroslav Anashkin (manashkin) wrote :

In several hours after release we already got 2 customers, encountered missing firmware issue.
Raised to Critical for 5.0.1

Miroslav Anashkin (manashkin) wrote :

Attached the list of firmware files, existing in 4.1 bootstrap image , but missing in 5.0's bootstrap.

Frank J. Cameron (fjc) wrote :

I don't see this one mentioned in the attachment "List of missing bootstrap firmware files," but my test server is looking for the file bnx2x/bnx2x-e2-7.8.17.0.fw. It's a Dell R720xd with a BCM57800 (multi 10Gb/1Gb). (It was previously working with 4.1.)

Frank J. Cameron (fjc) wrote :

Just want to confirm for you: I added bnx2x-e2-7.8.17.0.fw to my bootstrap initramfs and the interfaces came up and the system registered with the UI. Thanks.

Fix proposed to branch: master
Review: https://review.openstack.org/97042

Changed in fuel:
assignee: Fuel OSCI Team (fuel-osci) → Dmitry Burmistrov (dburmistrov)
status: Confirmed → In Progress
OSCI Robot (oscirobot) wrote :

Package linux-firmware has been built from changeset: http://gerrit.mirantis.com/15911
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-15911/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-15911/centos"

Michael Petersen (mpetason) wrote :

This extends to Cisco Hardware. Known Hardware with these network cards:

UCSC-BSE-SFF-C200

Andrew (box857+launchpad) wrote :

We are using HP Proliant and have the same issue. Works fine with 4. but not 5.0

Alexander Dibbo (apdibbo) wrote :

We are using Dell R710s. I haven't verified with 4.
The missing file is "bnx2/bnx2-mips-09-6.2.1b.fw"

Mike Scherbakov (mihgen) wrote :

It was documented as known issue for 5.0, and has a workaround for live installs: http://docs.mirantis.com/openstack/fuel/fuel-5.0/release-notes.html#known-issues-in-mirantis-openstack-5-0, see "Bootstrap kernel issues on certain hardware".

Artem Panchenko (apanchenko-8) wrote :

Hi,

I tried to run system tests on that iso (fuel-gerrit-5.1-150-2014-06-17_20-45-12.iso) and got the following errors during CentOS installation (master node):

http://paste.openstack.org/show/84496/

Master node installation wasn't complete, so all tests failed.

OSCI Robot (oscirobot) wrote :

Package linux-firmware has been built from changeset: http://gerrit.mirantis.com/15911
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-15911/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-15911/centos"

Thomas Black (tsblack) wrote :

I suffer with this issue on IBM X3400 hardware. The link for the above ISO doesn't appear to be public. Is there any other link to the test ISO?

Changed in fuel:
assignee: Dmitry Burmistrov (dburmistrov) → Roman Vyalov (r0mikiam)
Dmitry Borodaenko (angdraug) wrote :

Thomas, fuel-gerrit-5.1-172-2014-06-19_21-49-24.iso is an internal ISO image (gerrit in the name indicates that it was built from a gerrit branch and contains code that wasn't mainlined into Fuel yet), and that it's based on 5.1 release which currently is in active development. Given all that, it's probably not a good idea to publish this particular ISO.

Dmitry, can you build an ISO with the same fixes based on 5.0 (current head aka pre-5.0.1, not 5.0 GA)? I think that would be a much better version for Thomas and others to test.

OSCI Robot (oscirobot) wrote :

Package linux-firmware has been built from changeset: http://gerrit.mirantis.com/16800
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.1-stable-16800/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.1-stable-16800/centos"

Public link http://seed-eu01.mirantis.com/fuelweb-iso/fuel-gerrit-5.0.1-219-2014-06-26_22-05-44.iso.torrent
Disclamer: This is NOT a stable build. This build contains code covered with no warranties. Link is temporary and could be deleted anytime without announcements.

OSCI Robot (oscirobot) wrote :

Package linux-firmware has been built from changeset: http://gerrit.mirantis.com/16800
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.1-stable/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.1-stable/centos"

Changed in fuel:
status: In Progress → Fix Committed
OSCI Robot (oscirobot) wrote :

Package linux-firmware has been built from changeset: http://gerrit.mirantis.com/15911
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable/centos"

Reviewed: https://review.openstack.org/97046
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=7a7885f192c4572eaacbe3f6bb4dcd1c48b05ab6
Submitter: Jenkins
Branch: master

commit 7a7885f192c4572eaacbe3f6bb4dcd1c48b05ab6
Author: Dmitry Burmistrov <email address hidden>
Date: Sat May 31 23:43:08 2014 +0400

    Use linux-firmware package instead of kernel-lt-firmware

    Change-Id: Id14c9f10402e273bdc51a2cb68f1b4d53e4666af
    Related-Bug: #1323354

Reviewed: https://review.openstack.org/102813
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=58f11a69bd9ce6ebb99a8c34e8dcb704a710e160
Submitter: Jenkins
Branch: stable/5.0

commit 58f11a69bd9ce6ebb99a8c34e8dcb704a710e160
Author: Dmitry Burmistrov <email address hidden>
Date: Sat May 31 22:16:00 2014 +0400

    Use external kernel firmware package for bootstrap image

    Change-Id: I0da7bddde26fd852b5bf06e263e7f49980b02751
    Closes-Bug: #1323354

Reviewed: https://review.openstack.org/102814
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=bae51e8aa1e211edd3a8354ef6c37bb4fc628f9e
Submitter: Jenkins
Branch: stable/5.0

commit bae51e8aa1e211edd3a8354ef6c37bb4fc628f9e
Author: Dmitry Burmistrov <email address hidden>
Date: Sat May 31 23:43:08 2014 +0400

    Use linux-firmware package instead of kernel-lt-firmware

    Change-Id: Id14c9f10402e273bdc51a2cb68f1b4d53e4666af
    Related-Bug: #1323354

Meg McRoberts (dreidellhasa) wrote :

Can someone please clarify how this should be handled in the 5.0.1 Release Notes? It is marked as Fix Committed for 5.0.1 but the text indicates that this is still a known issue, just that we have provided a better bootstrap image for people who need it. The procedure may need modification -- location for the new file at a minimum.

http://docs.mirantis.com/openstack/fuel/master/release-notes.html#bootstrap-kernel-issues-on-certain-hardware

Dmitry Borodaenko (angdraug) wrote :

Meg, the solution to this problem was to update the bootstrap image in 5.0.1 and 5.1, so as soon as QA confirms that this has indeed been fixed (by updating status to Fix Released), you can move this from Known Issues to Resolved in 5.0.1.

tags: added: release-notes
Artem Panchenko (apanchenko-8) wrote :

Fix released, verified on iso # 122 (5.0.1) and # 317 (5.1)

Arshad Khan (khan-2) wrote :

So how do I upgrade the Fuel Node to 5.0.1? Is there a link for the iso or some procedures to upgrade or is it better to do the workaround with different kernel?

There is an issue which has been marked as duplicate of this one:
https://bugs.launchpad.net/fuel/+bug/1340559

Can somebody please confirm that 1340559 has actually been resolved as part of the resolution for this issue?

Dmitry Pyzhov (dpyzhov) on 2014-08-14
no longer affects: fuel/5.1.x
Changed in fuel:
milestone: 5.0.1 → 5.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers