i915 firmware is not copied to initrd

[Jason Gerard DeRose]

[ Impact ]

On Skylake, skl_dmc firmware is not incorporated into the initramfs. This is because it is a symlink to a versioned blob and the initramfs-tools hook does not dereference symlinks. As the driver is actually loaded from the initramfs, the firmware is missing for the lifetime of the boot, which means that the GPU cannot enter its lowest available power states plus some hotplugging scenarios are broken. This all manifests itself as such in dmesg:

[ 0.728803] i915 0000:00:02.0: Direct firmware load for i915/skl_dmc_ver1.bin failed with error -2
[ 0.728817] [drm:i915_firmware_load_error_print [i915]] *ERROR* failed to load firmware i915/skl_dmc_ver1.bin (0)

[ Test Case ]

After the fix has been applied and the initrd regenerates, the firmware should load successfully after a reboot.

[ Regression Potential ]

Unlikely, as the fix is part of Ubuntu since Wily.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Timo Aaltonen (tjaalton)]

yes, this happens on SKL-S for some reason, probably should upstream a bug about it

[Timo Aaltonen (tjaalton)]

Booting 4.3-rc1 or nightly on SKL-S results in

[ 1.139104] [drm:i915_dump_device_info] i915 device info: gen=9, pciid=0x1912 rev=0x06 flags
=need_gfx_hws,is_skylake,has_fbc,has_hotplug,has_llc,has_ddi,
[ 1.139115] [drm:intel_detect_pch] Found SunrisePoint PCH
[ 1.139419] [drm:intel_csr_ucode_init] Loading i915/skl_dmc_ver1.bin
[ 1.139434] i915 0000:00:02.0: Direct firmware load for i915/skl_dmc_ver1.bin failed with error -2
[ 1.139448] [drm:i915_firmware_load_error_print [i915]] *ERROR* failed to load firmware i915/skl_dmc_ver1.bin (0)

the firmware is there though, and of the latest version.

[Jani-nikula (jani-nikula)]

(In reply to Timo Aaltonen from comment #0)
> Booting 4.3-rc1 or nightly on SKL-S results in
>
> [ 1.139104] [drm:i915_dump_device_info] i915 device info: gen=9,
> pciid=0x1912 rev=0x06 flags
> =need_gfx_hws,is_skylake,has_fbc,has_hotplug,has_llc,has_ddi,
> [ 1.139115] [drm:intel_detect_pch] Found SunrisePoint PCH
> [ 1.139419] [drm:intel_csr_ucode_init] Loading i915/skl_dmc_ver1.bin
> [ 1.139434] i915 0000:00:02.0: Direct firmware load for
> i915/skl_dmc_ver1.bin failed with error -2
> [ 1.139448] [drm:i915_firmware_load_error_print [i915]] *ERROR* failed to
> load firmware i915/skl_dmc_ver1.bin (0)
>
> the firmware is there though, and of the latest version.

i915 loading at 1.139104 seems rather early, maybe you have i915 built-in but no firmware in initrd?

[Timo Aaltonen (tjaalton)]

yeah, turns out i915 is included in initrd in some cases, but not others.. I'll file a distro bug to include the fw too then.

[Timo Aaltonen (tjaalton)]

so i915 is included in initrd, but the firmware is not

[Jason Gerard DeRose (jderose)]

Timo,

I didn't realize that this module is being loaded during the initrd phase before the rootfs is mounted. Mystery solved, I think :D

Thanks!

[Andy Whitcroft (apw)]

Yes the way we copy firmware into the initramfs we simply copy the top level link and do not carry the targets at all. Later we sensibly get rid of hanging symlinks leaving no sign of the firmware.

I suspect we should be iterating over the link copy its components.

[Andy Whitcroft (apw)]

Ok, the PPA below has an update for initramfs-tools for wily (only currently) which I think sorts this out:

     https://launchpad.net/~apw/+archive/ubuntu/initramfs-tools/

If someone who has a platform which has this h/w could check it really loads. Please report any testing back here.

[Jason Gerard DeRose (jderose)]

Andy, I spotted the problem:

Line 120 in /usr/share/initramfs-tools/hook-functions:

    cp -a "$firmware" "$target_dir"

`cp -a` copies the symlink itself rather than the target it points to, so we end up with a broken symlink in initramfs and no actual firmware file.

The fix that comes to mind for me is:

  cp -aL "$firmware" "$target_dir"

But I'm not sure if that would have consequences elsewhere. Are symlinks utilized anywhere in the initramfs?

[Jason Gerard DeRose (jderose)]

Andy: oops, didn't refresh the page and see your latest comments before I posted the above, please ignore.

I just tested the initramfs-tools package in your PPA, confirmed it works:

1) No more error in dmesg about skl_dmc not being loaded

2) `lsinitramfs -l` shows both the "skl_dmc_ver1.bin" symlink and the "skl_dmc_ver1_21.bin firmware file

[Andy Whitcroft (apw)]

Uploaded to wily. Also confirmed that these links appear in linux-firmware backports to both trusty and vivid. Marking those affected.

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.120ubuntu5

---------------
initramfs-tools (0.120ubuntu5) wily; urgency=medium

  * hook-functions: firmware -- copy symlink components into initramfs
    (LP: #1496163)

 -- Andy Whitcroft <email address hidden> Sat, 19 Sep 2015 09:55:13 +0100

[Peter Curtis (pdcurtis)]

I seem to have this bug with a Skylake machine where the firmware load for i915/skl_dmc_ver1.bin failed with error -2.

I am running (or trying to) with the Ubuntu LTS Trusty/14.04 version.

Is there any way I can use the Wily versions of initramfs-tools or do you have an estimate when it will be backported to the LTS version as per #13

[Brad Parker (cbparker)]

Peter: Manually modifying the file per comment #9's fix worked for me on 14.04. But I am still having lots of problems with the GPUs on my laptop. Also had to use xorg-edgers ppa to get hardware acceleration back, otherwise X complained about unknown chipset. Also, the newest kernel versions (like 4.3) are freezing up hard for me, and different older versions only work on certain GPUs. What I mean is like, if I want to use the discrete nvidia GPU, I have to boot 3.19, but if I want integrated graphics, then I have to boot 4.2.4 with i915.preliminary_hw_support=1. Also, running "lspci" on newer kernels freezes the machine hard.

[Nick N. (niktator)]

Any news on this? I have the same bug on 14.04. Manually modifying the file like in comment #9 didn't change anything.

[Brad Parker (cbparker)]

Nick: Did you re-generate the initrd image? I did mine with sudo update-initramfs -u -k all

[Jarkko Korpi (jarkko-korpi-t)]

sudo update-initramfs -u -k all Seems to help. The issue is still valid linux mint 17.3

[Nick Cool (coca-cola1)]

According to a user of easyVDR 3.0 (based on 14.04 LTS) with 4.2.0 it is still unfixed and the fix in #9 does not work.

[Margarita Manterola (marga-9)]

Can we please get the fix in #9 (cp -aL) applied in Trusty? This is currently causing Trusty's initrds to not have the right firmware for skylake machines that are supported by the latest HWE stack.

I tried to use the package from xenial (which has a much more elaborate copy_file function), but it won't install on Trusty because it's incompatible with the console-setup version in trusty (breaking change here: [1]).

The cp -aL fix works fine and we have been using that on our machines, but it's pretty bad that every user has to patch that manually.

[1]: https://anonscm.debian.org/cgit/kernel/initramfs-tools.git/patch?id=0ffc80095319c84e3456156d5c00789b529c3dd3

[Justin King-Lacroix (justinkl)]

Any news on this?

[Justin King-Lacroix (justinkl)]

Hi guys,

This is still biting us. Can we please have this fixed? See LP #1642704 for details. The fix is one character.

Justin

[Justin King-Lacroix (justinkl)]

Hello?

[Philipp Kern (pkern)]

Attached is a debdiff for a SRU for trusty.

[Brian Murray (brian-murray)]

I've uploaded this to Trusty queue for review by the SRU team.

[Steve Langasek (vorlon)]

Hello Jason, or anyone else affected,

Accepted initramfs-tools into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.103ubuntu4.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Justin King-Lacroix (justinkl)]

Yep, that's good. Thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.103ubuntu4.6

---------------
initramfs-tools (0.103ubuntu4.6) trusty; urgency=medium

  [ Andy Whitcroft ]
  * hook-functions: firmware -- copy symlink components into initramfs
    (LP: #1496163)

 -- Philipp Kern <email address hidden> Fri, 16 Dec 2016 09:02:52 -0800

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for initramfs-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.