initramfs unpacking failed

[vmc]

"initramfs unpacking failed: Decoding failed", message appears on boot up.

If I "update-initramfs" using gzip instead of lz, then boot up passes without decoding failed message.

---

However, we currently believe that the decoding error reported in dmesg is actually harmless and has no impact on usability on the system.

Switching from lz4 to gzip compression, simply papers over the warning, without any benefits, and slows down boot.

Kernel should be fixed to correctly parse lz4 compressed initrds, or at least lower the warning, to not be user visible as an error.

[Impact]

 * Decoding failure messages in dmsg with a single lz4 initrd

 * Multiple lz4 compressed initrds cannot be decompressed by kernel, when loaded by grub

 * Multiple lz4 compressed initrds cannot be decompressed by kernel, when there is padding between them

[Test Case]

 * Create empty padding with $ dd if=/dev/zero of=pad4 bs=1 count=4

 * Create an lz4 compressed initrd with a single test-file in it with some content. I.e. echo "second-initrd" > test-file, and then pack that with cpio hewc owned by root & lz4 -l.

 * Create a combined padded initrd of stock initrd, pad4, and the test-marker initrd created above.

 * Boot above with "break=top" kernel command line.

 * With broken kernels, there should be dmesg error message that decoding failed, and one will observe that /test-file does not exist in the shell.

 * With fixed kernel, /test-file in the initrd shell should exist, and should have the expected content "second-initrd".

 * The alignment and padding in the above test case depends on the size of the first initrd => if a given padded initrd does not reproduce the problem, try varying the size of the first initrd or that of the padding between 0..4.

[Where problems could occur]

 * This changes compatible lz4 decompressor in the kernel, which can also be used by other kernel modules such as cryptography, squashfs, zram, f2fs, comprssed kernel image, pstore. For example, previously rejected files with "bogus" length and extra padding may now be accepted, whereas they were previously getting rejected by the decompressor.

 * Ideally kernel should switch to the stable lz4 format which has better specification of end of stream.

[vmc (vmclark)]

This error is for Ubuntu eoan.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1835660/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Steve Langasek (vorlon)]

Just to confirm, you mean that this message is shown on the otherwise-blank console when booting with 'quiet'?

[vmc (vmclark)]

There are two messages that appear, the first is "initramfs unpacking failed: Decoding failed", right afterwards the partitions is clean message appears.

So, yes.

[vmc (vmclark)]

I also noticed on July 3rd thereabouts, was a change for 'lz4', that was approved.
 Not sure if I had this error then. I just started noticing around July 7th.

[Brian Murray (brian-murray)]

I also see this message in my journal but it doesn't seem to affect my system i.e. its booted and working.

Jul 10 09:22:14 speedy kernel: Unpacking initramfs...
Jul 10 09:22:14 speedy kernel: Initramfs unpacking failed: Decoding failed

[vmc (vmclark)]

As of today's ISO (7/11) installed, there is no "initramfs unpacking failed: Decoding failed" message. It appears solved.

[vmc (vmclark)]

"initramfs unpacking failed: Decoding failed" message still fails.
Using:

 eoan ubuntu-budgie, ISO dated 7/18/19

[Gérard Bigot (gerard-bigot)]

I also get this

   Trying to unpack rootfs image as initramfs...
   Initramfs unpacking failed: Decoding failed

on a fully uptodate eoan, just after booting today, in 'journalctl -b'.

[vmc (vmclark)]

as of 10/12/2019 eoan 19.10, Ubuntu, Lubuntu, Xubuntu still have this issue.

What I do to "fix" it is change:

COMPRESS=lz4, to COMPRESS=gzip, @:

/etc/initramfs-tools/initramfs.conf, then

 sudo update-initramfs -u.

[tom (tombuntus)]

Changing to gzip is bad. 19.10 switched over to lz4 because it's way faster decompressing, which makes for faster boots!

[vmc (vmclark)]

Did you read the part: "initramfs unpacking failed: Decoding failed".

  So what's the point if its faster and fails?

If you have splash on, you won't see the error message. use dmesg.

[vmc (vmclark)]

Boot up is actually faster after fix:

dmesg, boot-time b4-fix

$ dmesg|grep Decoding
[ 0.461068] Initramfs unpacking failed: Decoding failed
====
$ systemd-analyze
Startup finished in 14.140s (firmware) + 4.226s (loader) + 6.079s (kernel) + 17.270s (userspace) = 41.717s
graphical.target reached after 17.263s in userspace
====

dmesg, boot-time after-fix

$ dmesg|grep Decoding
====
$ systemd-analyze
Startup finished in 10.258s (firmware) + 3.650s (loader) + 3.262s (kernel) + 16.529s (userspace) = 33.701s
graphical.target reached after 16.507s in userspace

[tom (tombuntus)]

Boot lz4
Startup finished in 6.907s (firmware) + 8.338s (loader) + 2.595s (kernel) + 3.705s (userspace) = 21.547s
graphical.target reached after 3.695s in userspace

Will come back with gzip and see.

[tom (tombuntus)]

Boot lz4
Startup finished in 6.907s (firmware) + 8.338s (loader) + 2.595s (kernel) + 3.705s (userspace) = 21.547s
graphical.target reached after 3.695s in userspace

Boot gzip
Startup finished in 7.799s (firmware) + 8.070s (loader) + 3.562s (kernel) + 10.306s (userspace) = 29.739s
graphical.target reached after 10.162s in userspace

This is why developers switched to lz4!

It's really weird that vmclark is getting faster boot with gzip. It seems like a spinning hard drive and some other weird configurations are to blame.

[tom (tombuntus)]

Is yours failing? Mine is displaying the error message with lz4, but still booting and finding the root filesystem, that's what I thought this bug is about. You must not be failing either, as you have a boot time to measure. In the olden days, the other people who got this error weren't able to boot, and that's what failure to decode error means means.

[tom (tombuntus)]

Switching back to lz4, now I don't get the error! Maybe it's the newest kernel (.19 out today), or maybe it's just tweaking the decoding to gzip and back? Very strange.

Booting lz4 again
Startup finished in 7.775s (firmware) + 7.947s (loader) + 2.136s (kernel) + 9.105s (userspace) = 26.965s
graphical.target reached after 8.947s in userspace

[Nick B. (futurepilot)]

I still see this message on a fully up-to-date Kubuntu 19.10 system. However the system boots fine and everything seems to be working ok. This is apparently a non-fatal error message? It sounds pretty bad but it still seems to have been able to load the initramfs image fine. What does this really mean then?

[Mario (mario-gleirscher)]

Same for me.

I am on Ubuntu 19.10 with EFI Secure Boot and dual boot with Windows 10 (fastboot/hibernation disabled)

uname -a
... 5.3.0-26-generic #28-Ubuntu ...

initramfs-tools are still configured to produce a LZ4 image (the default)

dmesg | grep -i "error\|fail\|warn"
[ 0.504105] Initramfs unpacking failed: Decoding failed
[ 0.589422] i8042: Warning: Keylock active
[ 6.677615] EXT4-fs (nvme0n1p3): re-mounted. Opts: errors=remount-ro

Often booting fails but works after a couple of tries, resulting in this dmesg output.

No clue, it exceeds my knowledge what's going on here.

I fully reinstalled the kernel image and grub boot loader from the Live CD.

[Mario (mario-gleirscher)]

After reinstallation, the issue persists (see dmesg in my last comment) but my machine manages to boot (with EFI Secure Boot!).

[Mario (mario-gleirscher)]

For sake of completeness, dmesg says:

[ 0.476436] Trying to unpack rootfs image as initramfs...
[ 0.559806] Initramfs unpacking failed: Decoding failed
[ 0.565777] Freeing initrd memory: 46704K

Does that mean that the kernel is booting without using initramfs?

I was also looking whether the initrd image needs to be signed in order to be used in a UEFI Secure Boot scenario. But I couldn't find any information on whether Ubuntu's initramfs-tools or other scripts sign initrd, it surely comes with signed kernel images which seem to work fine.

Anyway, booting my machine is still a dice rolling experiment.

[Per-Inge (per-inge-hallin)]

Added Fedora 31 and made a triple boot system with Ubuntu 20.04, Windows 10 and Fedora 31.
Fedora stored the boot record in /dev/sdc1 and I had to change boot disk to sdc in BIOS.
Fedora provides a boot menu with
Fedora
Ubuntu
Windows
When I start Ubuntu via the Fedora boot menu I get this error message.

[Per-Inge (per-inge-hallin)]

I get different behavior depending on how I boot the computer using the BIOS boot menu
- When I select sda with Ubuntu 20.04, I don't get the error message
- When I select sdc with Fedora 31 and select Ubuntu from Fedora's boot menu I get the error message

[Willem Hobers (whobers)]

I am seeing this in my log as well,

Kubuntu 19.10

Linux willem-Aspire-A315-53 5.3.0-29-generic #31-Ubuntu SMP Fri Jan 17 17:27:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Operating System: Kubuntu 19.10
KDE Plasma Version: 5.17.5
KDE Frameworks Version: 5.66.0
Qt Version: 5.12.4
Kernel Version: 5.3.0-29-generic
OS Type: 64-bit
Processors: 8 × Intel® Core™ i5-8250U CPU @ 1.60GHz
Memory: 3,7 GiB

[Dan Andreșan (danyer)]

Possible explanation and fix: in my case it seems the error message appeared since I enabled secure boot (which I think was disabled when I installed Ubuntu). Look below, from my journal:

Dec 05 15:07:52 elite kernel: secureboot: Secure boot could not be determined (mode 0)
Dec 05 15:07:54 elite systemd[1]: secureboot-db.service: Succeeded.
Dec 05 15:14:19 elite kernel: secureboot: Secure boot enabled
Dec 05 15:14:21 elite systemd[1]: secureboot-db.service: Succeeded.

You can see that on December 5th I booted at 15:07 without secureboot and starting with 15:14 with secureboot enabled.

And now, let's see when the error message appeared first:

$ journalctl|grep "unpacking failed"
Dec 05 15:14:19 elite kernel: Initramfs unpacking failed: Decoding failed

Now a possible fix (it was possible for me, it might be possible for you too):

What I did is that I switched compression to gzip, the message dissapeared, then back to lz4, still there is no sign of error (same result as tom in #18). My guess it that there is no need to switch it to gzip and back to lz4, but just regenerate initramfs (which we did after changing compression) with "sudo update-initramfs -u".

Maybe this takes into account that now I have secureboot enabled and generates a "compatible" initramfs.

Someone else should confirm if the fix works for them too, then I guess this bug can be closed.

[Vincenzo Conti (vinc.o)]

Still present in Ubuntu 20.04

[Dan Andreșan (danyer)]

Vincenzo,

can you confirm (or not) that you changed your secure boot settings since Ubuntu was installed? Can you check comment #26?

Thanks!

[vmc (vmclark)]

My Secure Boot is always off, and all three U, X, Kubuntu have the error, as of yesterdays ISO.

[Edward Savage (epssyis)]

Regenerating initramfs as suggested by Dan resolved this issue for me.

I did not attempt to change compression or change any settings in the BIOS.

[vmc (vmclark)]

Xubuntu 20.04 failed:
$ dmesg |grep failed
[ 0.456889] Initramfs unpacking failed: Decoding failed
[ 5.838496] thermal thermal_zone2: failed to read out thermal zone (-61)
[ 6.053231] [drm] failed to retrieve link info, disabling eDP

Then issued the following:

$ sudo update-initramfs -u
 It still fails.

[[e-mail] (sgage)]

I am experiencing this issue as well. I had secure boot turned off - turned it back on, regenerated initramfs, and still have the problem. How does one change the compression method?

[[e-mail] (sgage)]

I added COMPRESS=gzip to /etc/initramfs-tools/initramfs.conf and ran update-initramfs -u, and did NOT see the fail on the ensuing boot.

[Lucas de Eiroz Rodrigues (lucas-de-eiroz)]

I've been experiencing this issue the last couple of days on Ubuntu 20.04 Focal Fossa too. Issuing

$ sudo update-initramfs -u

does nothing for me.
Adding COMPRESS=gzip to /etc/initramfs-tools/initramfs.conf didn't work either.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1835660

[heynnema (heynnema)]

Your shouldn't be using "sudo update-initramfs -u", you should try "sudo update-initramfs -c".

[hasan (hasan-cerit)]

confirmed on eoan (5.3.0-42) after heynnema comments above, that the decoding fail message disappears switching gzip then back to lz4.

editing initramfs.conf and running the command below:

~ $ sudo update-initramfs -c -k all

The result:

~ $ sudo cat /etc/initramfs-tools/initramfs.conf | grep lz4
# COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz ]
COMPRESS=lz4
~ $ dmesg | grep fail
~ $

[hasan (hasan-cerit)]

another note, just upgraded to latest kernel and the error is back...

~ $ uname -r
5.3.0-45-generic
~ $ dmesg | grep fail
[ 0.218191] Initramfs unpacking failed: Decoding failed

[corrado venturini (corradoventu)]

Also on Focal:
corrado@corrado-x5-ff-0329:~$ dmesg | grep fail
[ 0.288574] Initramfs unpacking failed: Decoding failed
corrado@corrado-x5-ff-0329:~$ uname -a
Linux corrado-x5-ff-0329 5.4.0-21-generic #25-Ubuntu SMP Sat Mar 28 13:10:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
corrado@corrado-x5-ff-0329:~$

[Brian Murray (brian-murray)]

@vmclark - Does this affect your ability to install or use the system at all?

[Dimitri John Ledkov (xnox)]

@Siarhei

That looks very valid. And we were close to getting to that, when apw & I timed out looking at this.

lz4 -l Use Legacy format (typically for Linux Kernel compression)
              Note : -l is not compatible with -m (--multiple) nor -r

Which must always be used when creating initrd. Ideally, both userspace output and linux kernel consumption should agree what said legacy format should look like and what it must produce and consume.

Is https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#legacy-frame underspecified, or does lz4 produce non-compliant frames, or kernel misprocesses compliant frames?

Also, can we upgrade lz4 in the kernel tree to 1.9.2?

[Dimitri John Ledkov (xnox)]

Also i guess we (i) need to retest if https://github.com/lz4/lz4/releases/tag/v1.9.3 fixes all the things =/

[Dimitri John Ledkov (xnox)]

So lz4 compressed initrd looks like this with hexdump

568f580 0523 00ac 54bf 4152 4c49 5245 2121 0021
568f590 0001 1cff 0050 0000 0000
568f59a

I do wonder what ram is initialized too, and how those things look when kernel reads initrd from memory as loaded by the bootloader / qemu.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "[PATCH] unlz4: Handle 0-size chunks, discard trailing padding/garbage" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Dimitri John Ledkov (xnox)]

NAK on the linux patch.

I think this is a grub bug.

When loading multiple initrds, grub aligns_up each one of them at 4bytes boundary, and allocates pages for that. And it declares and passes ramdisk_image as the total allocated memory. Rather than the true size of the initrds.

[Dimitri John Ledkov (xnox)]

As discussed at https://github.com/lz4/lz4/issues/956 I really think it is grub2 secureboot patches linuxefi bug. WIP thing is here https://github.com/rhboot/grub2/pull/77/files

[Tim Wetzel (twetzel21)]

Dimitri, I note your comment #106 about secureboot patches upstream. If I recall correctly, a linux security update came down through Ubuntu Updater about the same time or just prior to the LTS 20.04.1 updates in mid September. A number of issues with LTS 20.04.1 appeared following two "batches" of September Ubuntu updates. To the extent that these issues may be inter-related -- at least insofar as being triggered by upstream Linux security change(s) -- please note the bug numbers listed in post #93 above.

If not relevant to this particular bug, then please disregard in deference to the other individual bug numbers; if inter-related, then perhaps this can be helpful.

[Dimitri John Ledkov (xnox)]

@twetzel21 this is not related to those changes at all.

[Dimitri John Ledkov (xnox)]

so what grub is doing is correct.

It pads/aligns every initrd by 4, which is fine, and as per spec.

https://www.kernel.org/doc/html/latest/driver-api/early-userspace/buffer-format.html

initramfs size can be filled with arbitrary amount of "\0" all the way upto initramfs_size.

"In human terms, the initramfs buffer contains a collection of compressed and/or uncompressed cpio archives (in the “newc” or “crc” formats); arbitrary amounts zero bytes (for padding) can be added between members."

Thus it feels to me that decompress_unlz4.c method in the kernel does not correctly stop decompressing and returning consumed output position (posp).

For example, decompress_inflate.c skips over trailer and updates output pos by 8, at the end of successful conversion.

Similarly unlzma also updates posp at the end of conversion.

[Soren ONeill (soren-s)]

I struggled with the same issue on a fresh Kubuntu 20.04 installation (Lenovo Thinkpad P52s).

From the Grub / Advanced options / Recovery mode / root terminal I was able to remount the / as rw (rather than ro)

Changing 'lz4' to 'gzip' as per https://forums.linuxmint.com/viewtopic.php?f=46&t=323152
and running update-initramfs -c -k $(uname -r) I had a new issue: the fstab entry UUID=3e757667-25352354... was not resolved.

To solve this I had to:
manually changed the fstab entry from UUID=... to /dev/nvme0n1p2
change COMPRESS=lz4 to gzim in /etc/initramfs-tools/initramfs.conf
update-initramfs as per above
reboot

[[e-mail] (babazako)]

I am also having this issue, however not a new install. I have been running xubuntu 20.04.1 for two month +-

acer travelmate P215-52
Xubuntu 20.04.1
Kernel: 5.4.0-60 and 5.4.0-58
/ not encrypted
/Home is encrypted

The system worked fine. while booting the system drops to initramfs

with :
- Boot args ( cat /proc/cmdline)
 - Check rootdelay=(did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=***** dose not exist Dropping to shell

in the initramfs, /dev has no nvme drive or crypt drive, /dev/mapper/ has only control in it.

I did not have luck booting in to root via gurb, or getting a shell to edit files on the disk.
Grub shell is showing the root (hd0,gpt2) part /etc , /var... no issues with a cat command of the file on it. (hd0,gpt1) is a efi boot part.

Booting with live usb(xubuntu 20.04.1):
No local block device was found, so it was not possible to mount or a clean reinstall.

[[e-mail] (babazako)]

Fixed by changing the BIOS SATA confg to AHCI.
However I have no idea how this setting was changed, as this was not an action that I have done.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1835660

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[paul-lawrenceville (pkowalzyk)]

I want to sincerely thank you for your information. I assume that you are
referring to my ubuntu based Linux Mint 20.04 problem. Since yesterday, I
have removed and installed Mint 20.1 and find a little error with my
initramfs but I have nothing to worry about. I will file this in my archive
of Mint/Ubuntu remedies. Thank you for your input.

On Wed, Jan 13, 2021, 3:41 PM Ubuntu Kernel Bot <email address hidden>
wrote:

> This bug is missing log files that will aid in diagnosing the problem.
> While running an Ubuntu kernel (not a mainline or third-party kernel)
> please enter the following command in a terminal window:
>
> apport-collect 1835660
>
> and then change the status of the bug to 'Confirmed'.
>
> If, due to the nature of the issue you have encountered, you are unable
> to run this command, please add a comment stating that fact and change
> the bug status to 'Confirmed'.
>
> This change has been made by an automated script, maintained by the
> Ubuntu Kernel Team.
>
> ** Changed in: linux (Ubuntu)
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1870260).
> https://bugs.launchpad.net/bugs/1835660
>
> Title:
> initramfs unpacking failed
>
> Status in OEM Priority Project:
> New
> Status in grub2 package in Ubuntu:
> Invalid
> Status in initramfs-tools package in Ubuntu:
> Invalid
> Status in linux package in Ubuntu:
> Incomplete
>
> Bug description:
> "initramfs unpacking failed: Decoding failed", message appears on
> boot up.
>
> If I "update-initramfs" using gzip instead of lz, then boot up passes
> without decoding failed message.
>
> ---
>
> However, we currently believe that the decoding error reported in
> dmesg is actually harmless and has no impact on usability on the
> system.
>
> Switching from lz4 to gzip compression, simply papers over the
> warning, without any benefits, and slows down boot.
>
> Kernel should be fixed to correctly parse lz4 compressed initrds, or
> at least lower the warning, to not be user visible as an error.
>
> [Impact]
>
> * Decoding failure messages in dmsg with a single lz4 initrd
>
> * Multiple lz4 compressed initrds cannot be decompressed by kernel,
> when loaded by grub
>
> * Multiple lz4 compressed initrds cannot be decompressed by kernel,
> when there is padding between them
>
> [Test Case]
>
> * Create empty padding with $ dd if=/dev/zero of=pad4 bs=1 count=4
>
> * Create an lz4 compressed initrd with a single test-file in it with
> some content. I.e. echo "second-initrd" > test-file, and then pack
> that with cpio hewc owned by root & lz4 -l.
>
> * Create a combined padded initrd of stock initrd, pad4, and the
> test-marker initrd created above.
>
> * Boot above with "break=top" kernel command line.
>
> * With broken kernels, there should be dmesg error message that
> decoding failed, and one will observe that /test-file does not exist
> in the shell.
>
> * With fixed kernel, /test-file in the initrd shell should exist, and
> should have the expected content "second-initrd".
>
> * The alignment and padding in the a...

[Deadly Effect (deadly.effect)]

Hi
I have this issue when I try to boot USB installation pendrive with Ubuntu 20.04.1 live server made from ISO available from the official webpage.
I downloaded ISO two times, I sha256 its content and compared to value stored in the SHA256SUMS file.
I want to install Ubuntu on my PC next to Windows 10 (dual-boot).
My PC is 4790k, 16GB RAM, GTX1080, multiple hard drives, system disk is SSD.

How can I cope with this issue? What can I do to omit it or to resolve it?
The "Initramfs unpacking failed: Decoding failed" message pops up just after the GRUB menu disappears.
Please help, I would really like to run the Ubuntu server on my computer.

Best regards

[Deadly Effect (deadly.effect)]

The same thing happened when I try Ubuntu 20.10 live server... Let me check the 18.04 server. I currently use Ubuntu Desktop 18.04 on my laptop, maybe the server edition will work.

[Deadly Effect (deadly.effect)]

I have successfully resolved the problem. I don't know exactly what helped because I did a few things. Namely: I switched all SATA cables from the second controller to the first (my computer motherboard has two SATA controllers) and disabled the Compatibility Support Module. Unfortunately, CSM turned off the BIOS boot options, only UEFI was left. After changing the settings and rebooting, the message "Initramfs unpacking failed: Decoding failed" was still showing on the screen, but the system continued the boot process and finally saw the Ubuntu Server 20.04 installer

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-groovy' to 'verification-done-groovy'. If the problem still exists, change the tag 'verification-needed-groovy' to 'verification-failed-groovy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[shanghf (checkmode)]

Fix this bug with a patch, please have a try.

[Kelsey Steele (kelsey-steele)]

@vmclarck, @xnox, or anyone who has been dealing with this bug - may you please verify if the kernel in -proposed resolves the problem for you in Focal and/or Groovy? Instructions are listed in comment #120

Thank you!

[vmc (vmclark)]

This bug failure is still present as of current ISO dated 01/25/2021 using Kubuntu:
[code]vmc@01-25-21:~$ sudo dmesg|grep failed
[ 0.538632] Initramfs unpacking failed: Decoding failed[/code]

[vmc (vmclark)]

This bug is still failing per current ISO dated 01/25/2021 booting from Kubuntu 21.04:

$ sudo dmesg|grep failed
[ 0.538632] Initramfs unpacking failed: Decoding failed

[vmc (vmclark)]

The corrected ISO date is this:
 hirsute-desktop-amd64.iso 2021-01-26 05:51
from Kubuntu daily-live current. It still fails.

[Dimitri John Ledkov (xnox)]

@vmc

this bugfix is not yet available on any .iso for any series yet. It can only be tested from installed systems and enabling proposed.

[Dimitri John Ledkov (xnox)]

1) checking that reproducer still fails

sudo qemu-system-x86_64 -kernel ./vmlinuz-5.4.0-64-generic -initrd ./initrd-pad4-5.4.0-64-generic -append 'break=top quiet console=ttyS0' -m 2G -nograt

[ 3.581106] Initramfs unpacking failed: Decoding failed
Spawning shell within the initramfs

BusyBox v1.30.1 (Ubuntu 1:1.30.1-4ubuntu6.3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs) cat /test-file
cat: can't open '/test-file': No such file or directory
(initramfs) uname -a
Linux (none) 5.4.0-64-generic #72-Ubuntu SMP Fri Jan 15 10:27:54 UTC 2021 x86_64 GNU/Linux
(initramfs)

As expected, error message shown and file from second initrd is not accessible

2) checking that #65 ABI build of v5.4 is good

Verifying v5.4 using a custom test case:

sudo qemu-system-x86_64 -kernel ./vmlinuz-5.4.0-65-generic -initrd ./initrd-pad4-5.4.0-65-generic -append 'break=top quiet console=ttyS0' -m 2G -nograt

Spawning shell within the initramfs

BusyBox v1.30.1 (Ubuntu 1:1.30.1-4ubuntu6.3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs) cat /test-file
second
(initramfs) uname -a
Linux (none) 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 GNU/Linux
(initramfs) QEMU 5.0.0 monitor - type 'help' for more information
(qemu) quit

All is good.

[Dimitri John Ledkov (xnox)]

Verifying v5.8 build

$ sudo qemu-system-x86_64 -kernel ./vmlinuz-5.8.0-41-generic -initrd ./initrd-pad4-5.8.0-41-generic -append 'break=top quiet console=ttyS0' -m 2G -nograt

Spawning shell within the initramfs

BusyBox v1.30.1 (Ubuntu 1:1.30.1-4ubuntu6.3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs) uname -a
Linux (none) 5.8.0-41-generic #46~20.04.1-Ubuntu SMP Mon Jan 18 17:52:23 UTC 2021 x86_64 GNU/Linux
(initramfs) cat /test-file
second
(initramfs)

[Dimitri John Ledkov (xnox)]

$ sudo qemu-system-x86_64 -kernel ./vmlinuz-5.8.0-41-generic -initrd ./initrd-pad4-5.8.0-41-generic -append 'break=top quiet console=ttyS0' -m 2G -nographic -no-reboot

Spawning shell within the initramfs

BusyBox v1.30.1 (Ubuntu 1:1.30.1-4ubuntu6.3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs) cat /test-file
hello
(initramfs) uname -a
Linux (none) 5.8.0-41-generic #46-Ubuntu SMP Mon Jan 18 16:48:44 UTC 2021 x86_64 GNU/Linux
(initramfs)

Groovy verified too now.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.4.0-65.73

---------------
linux (5.4.0-65.73) focal; urgency=medium

  * focal/linux: 5.4.0-65.73 -proposed tracker (LP: #1912220)

  * initramfs unpacking failed (LP: #1835660)
    - SAUCE: lib/decompress_unlz4.c: correctly handle zero-padding around initrds.

  * overlay: permission regression in 5.4.0-51.56 due to patches related to
    CVE-2020-16120 (LP: #1900141)
    - ovl: do not fail because of O_NOATIME

  * Focal update: v5.4.79 upstream stable release (LP: #1907151)
    - net/mlx5: Use async EQ setup cleanup helpers for multiple EQs
    - net/mlx5: poll cmd EQ in case of command timeout
    - net/mlx5: Fix a race when moving command interface to events mode
    - net/mlx5: Add retry mechanism to the command entry index allocation

  * Kernel 5.4.0-56 Wi-Fi does not connect (LP: #1906770)
    - mt76: fix fix ampdu locking

  * [Ubuntu 21.04 FEAT] mpt3sas: Request to include the patch set which supports
    topology where zoning is enabled in expander (LP: #1899802)
    - scsi: mpt3sas: Define hba_port structure
    - scsi: mpt3sas: Allocate memory for hba_port objects
    - scsi: mpt3sas: Rearrange _scsih_mark_responding_sas_device()
    - scsi: mpt3sas: Update hba_port's sas_address & phy_mask
    - scsi: mpt3sas: Get device objects using sas_address & portID
    - scsi: mpt3sas: Rename transport_del_phy_from_an_existing_port()
    - scsi: mpt3sas: Get sas_device objects using device's rphy
    - scsi: mpt3sas: Update hba_port objects after host reset
    - scsi: mpt3sas: Set valid PhysicalPort in SMPPassThrough
    - scsi: mpt3sas: Handling HBA vSES device
    - scsi: mpt3sas: Add bypass_dirty_port_flag parameter
    - scsi: mpt3sas: Handle vSES vphy object during HBA reset
    - scsi: mpt3sas: Add module parameter multipath_on_hba
    - scsi: mpt3sas: Bump driver version to 35.101.00.00

 -- Kleber Sacilotto de Souza <email address hidden> Mon, 18 Jan 2021 17:31:23 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.8.0-41.46

---------------
linux (5.8.0-41.46) groovy; urgency=medium

  * groovy/linux: 5.8.0-41.46 -proposed tracker (LP: #1912219)

  * Groovy update: upstream stable patchset 2020-12-17 (LP: #1908555) // nvme
    drive fails after some time (LP: #1910866)
    - Revert "nvme-pci: remove last_sq_tail"

  * initramfs unpacking failed (LP: #1835660)
    - SAUCE: lib/decompress_unlz4.c: correctly handle zero-padding around initrds.

  * overlay: permission regression in 5.4.0-51.56 due to patches related to
    CVE-2020-16120 (LP: #1900141)
    - ovl: do not fail because of O_NOATIME

 -- Kleber Sacilotto de Souza <email address hidden> Mon, 18 Jan 2021 17:01:08 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.10.0-14.15

---------------
linux (5.10.0-14.15) hirsute; urgency=medium

  * hirsute/linux: 5.10.0-14.15 -proposed tracker (LP: #1913724)

  * Restore palm ejection on multi-input devices (LP: #1913520)
    - HID: multitouch: Apply MT_QUIRK_CONFIDENCE quirk for multi-input devices

  * intel-hid is not loaded on new Intel platform (LP: #1907160)
    - platform/x86: intel-hid: add Rocket Lake ACPI device ID

  * Hirsute update: v5.10.11 upstream stable release (LP: #1913430)
    - scsi: target: tcmu: Fix use-after-free of se_cmd->priv
    - mtd: rawnand: gpmi: fix dst bit offset when extracting raw payload
    - mtd: rawnand: nandsim: Fix the logic when selecting Hamming soft ECC engine
    - i2c: tegra: Wait for config load atomically while in ISR
    - i2c: bpmp-tegra: Ignore unknown I2C_M flags
    - platform/x86: ideapad-laptop: Disable touchpad_switch for ELAN0634
    - ALSA: seq: oss: Fix missing error check in snd_seq_oss_synth_make_info()
    - ALSA: hda/realtek - Limit int mic boost on Acer Aspire E5-575T
    - ALSA: hda/via: Add minimum mute flag
    - crypto: xor - Fix divide error in do_xor_speed()
    - dm crypt: fix copy and paste bug in crypt_alloc_req_aead
    - ACPI: scan: Make acpi_bus_get_device() clear return pointer on error
    - btrfs: don't get an EINTR during drop_snapshot for reloc
    - btrfs: do not double free backref nodes on error
    - btrfs: fix lockdep splat in btrfs_recover_relocation
    - btrfs: don't clear ret in btrfs_start_dirty_block_groups
    - btrfs: send: fix invalid clone operations when cloning from the same file
      and root
    - fs: fix lazytime expiration handling in __writeback_single_inode()
    - pinctrl: ingenic: Fix JZ4760 support
    - mmc: core: don't initialize block size from ext_csd if not present
    - mmc: sdhci-of-dwcmshc: fix rpmb access
    - mmc: sdhci-xenon: fix 1.8v regulator stabilization
    - mmc: sdhci-brcmstb: Fix mmc timeout errors on S5 suspend
    - dm: avoid filesystem lookup in dm_get_dev_t()
    - dm integrity: fix a crash if "recalculate" used without "internal_hash"
    - dm integrity: conditionally disable "recalculate" feature
    - drm/atomic: put state on error path
    - drm/syncobj: Fix use-after-free
    - drm/amdgpu: remove gpu info firmware of green sardine
    - drm/amd/display: DCN2X Find Secondary Pipe properly in MPO + ODM Case
    - drm/i915/gt: Prevent use of engine->wa_ctx after error
    - drm/i915: Check for rq->hwsp validity after acquiring RCU lock
    - ASoC: Intel: haswell: Add missing pm_ops
    - ASoC: rt711: mutex between calibration and power state changes
    - SUNRPC: Handle TCP socket sends with kernel_sendpage() again
    - HID: sony: select CONFIG_CRC32
    - dm integrity: select CRYPTO_SKCIPHER
    - x86/hyperv: Fix kexec panic/hang issues
    - scsi: ufs: Relax the condition of UFSHCI_QUIRK_SKIP_MANUAL_WB_FLUSH_CTRL
    - scsi: ufs: Correct the LUN used in eh_device_reset_handler() callback
    - scsi: qedi: Correct max length of CHAP secret
    - scsi: scsi_debug: Fix memleak in scsi_debug_init()
    - scsi: sd: Suppress spurious errors when WRITE SAME is being disabled
    - riscv: ...

[Fred (eldmannen+launchpad)]

I have this issue on Hirsuite.
Is this only fixed in 5.8 and 5.10 but not 5.11?

$ uname -a
Linux example 5.11.8-051108-generic #202103200636 SMP Sat Mar 20 11:17:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

I got the 5.11.8 kernel from https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.11.8/
because 5.11.0 caused GPU hangs for me on Intel 4770K Haswell.

[Dimitri John Ledkov (xnox)]

@Fred eldmannen+launchpad

This issue is only fixed in the Ubuntu patchset for the Linux Kernel. Although I have submitted this fix upstream, it has not been picked up yet by kernel.org vanilla kernels. See https://lkml.org/lkml/2021/1/14/1091

The mainline builds you point to, do not contain Ubuntu patchset, and thus are as vanilla as possible. Meaning that yes, those builds are affected by this issue.

There should be point release updates for Hirsute soon.

[Armandoobnaiala (armando1986)]

# An empty value - compcache isn't used, or added to the initramfs at all.
# An integer and K (e.g. 65536 K) - use a number of kilobytes.
# An integer and M (e.g. 256 M) - use a number of megabytes.
# An integer and G (e.g. 1 G) - use a number of gigabytes.
# An integer and % (e.g. 50 %) - use a percentage of the amount of RAM.
#
# You can optionally install the compcache package to configure this setting
# via debconf and have userspace scripts to load and unload compcache.
#

COMPCACHE_SIZE=""

#
# COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz ]
#

stuck here

[Armandoobnaiala (armando1986)]

update-initramfs: Generating /boot/initrd.img-5.8.0-45-generic
/usr/sbin/mkinitramfs: 177: COMPRESS: parameter not set
update-initramfs: failed for /boot/initrd.img-5.8.0-45-generic with 2.

[paul-lawrenceville (pkowalzyk)]

Works well, thank you for the update.

On Fri, Mar 26, 2021, 7:31 AM Armandoobnaiala <email address hidden>
wrote:

> update-initramfs: Generating /boot/initrd.img-5.8.0-45-generic
> /usr/sbin/mkinitramfs: 177: COMPRESS: parameter not set
> update-initramfs: failed for /boot/initrd.img-5.8.0-45-generic with 2.
>
>
> ** Changed in: initramfs-tools (Ubuntu Focal)
> Assignee: (unassigned) => Armandoobnaiala (armando1986)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1835660
>
> Title:
> initramfs unpacking failed
>
> Status in OEM Priority Project:
> New
> Status in grub2 package in Ubuntu:
> Invalid
> Status in initramfs-tools package in Ubuntu:
> Invalid
> Status in linux package in Ubuntu:
> Fix Released
> Status in grub2 source package in Focal:
> Invalid
> Status in initramfs-tools source package in Focal:
> Invalid
> Status in linux source package in Focal:
> Fix Released
> Status in grub2 source package in Groovy:
> Invalid
> Status in initramfs-tools source package in Groovy:
> Invalid
> Status in linux source package in Groovy:
> Fix Released
> Status in grub2 source package in Hirsute:
> Invalid
> Status in initramfs-tools source package in Hirsute:
> Invalid
> Status in linux source package in Hirsute:
> Fix Released
>
> Bug description:
> "initramfs unpacking failed: Decoding failed", message appears on
> boot up.
>
> If I "update-initramfs" using gzip instead of lz, then boot up passes
> without decoding failed message.
>
> ---
>
> However, we currently believe that the decoding error reported in
> dmesg is actually harmless and has no impact on usability on the
> system.
>
> Switching from lz4 to gzip compression, simply papers over the
> warning, without any benefits, and slows down boot.
>
> Kernel should be fixed to correctly parse lz4 compressed initrds, or
> at least lower the warning, to not be user visible as an error.
>
> [Impact]
>
> * Decoding failure messages in dmsg with a single lz4 initrd
>
> * Multiple lz4 compressed initrds cannot be decompressed by kernel,
> when loaded by grub
>
> * Multiple lz4 compressed initrds cannot be decompressed by kernel,
> when there is padding between them
>
> [Test Case]
>
> * Create empty padding with $ dd if=/dev/zero of=pad4 bs=1 count=4
>
> * Create an lz4 compressed initrd with a single test-file in it with
> some content. I.e. echo "second-initrd" > test-file, and then pack
> that with cpio hewc owned by root & lz4 -l.
>
> * Create a combined padded initrd of stock initrd, pad4, and the
> test-marker initrd created above.
>
> * Boot above with "break=top" kernel command line.
>
> * With broken kernels, there should be dmesg error message that
> decoding failed, and one will observe that /test-file does not exist
> in the shell.
>
> * With fixed kernel, /test-file in the initrd shell should exist, and
> should have the expected content "second-initrd".
>
> * The alignment and padding in the above test case depends on the
> size of the first initrd => if a given padded initrd does not
> ...

[Mario Limonciello (superm1)]

#134: @xnox, have you re-sent it? It looks like https://lkml.org/lkml/2021/1/14/1091 got no feedback.

[Dimitri John Ledkov (xnox)]

@superm1 yes I have, it is now pulled into v5.14 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=2c484419efc09e7234c667aa72698cb79ba8d8ed

I will request it to be included in linux-stable series.

Note that in Impish we have now switched to zstd compression for the initramfs, as it has even better bootspeed performance.

[Ken Takusagawa (ken-takusagawa-2)]

I've discovered that this behavior can also occur with the installer (similar to bug 1900130) when the USB stick to which the installer ISO was written is bad. (In this case, the error message is saying exactly what it should be saying.)