Kernel panic when booting with initramfs-tools version 0.120ubuntu1 generated initramfs

[Daniel Winzen]

After updating initramfs-tools to version 0.120ubuntu1 the kernel stops with a kernel panic during boot.

…
Begin Running /scripts/local-bottom...
Begin Running /scripts/init-bottom...
done.
/init: exec: line 372: switch_root: not found
Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000200
…

For the rest of the backtrace, see the attached image.

[Andy Whitcroft (apw)]

@Daniel -- could you attach the broken initramfs from /boot please.

This seems very odd to my eye as we should only use switch_root if we have found switch_root present as a command on the previous line in /init.

  if [ -z "$drop_caps" ] && command -v switch_root >/dev/null 2>&1; then
        exec switch_root ${rootmnt} ${init} "$@" ${recovery:+--startup-event=recovery} <${rootmnt}/dev/console >${rootmnt}/dev/console 2>&1
  elif command -v run-init >/dev/null 2>&1; then
        exec run-init ${drop_caps} ${rootmnt} ${init} "$@" ${recovery:+--startup-event=recovery} <${rootmnt}/dev/console >${rootmnt}/dev/console 2>&1
  fi

[Daniel Winzen (q-d-deactivatedaccount)]

Sure, here it is

[Andy Whitcroft (apw)]

Do you have the intel-microcode package installed on this system by any chance?

[Daniel Winzen (q-d-deactivatedaccount)]

Yes, it is installed.

[Daniel Winzen (q-d-deactivatedaccount)]

I just removed intel-microcode and tested again. Still the same issue as soon as I upgrade initramfs-tools and reboot.

[Harry (harry33)]

Same here, kernel panic with v. 0.120ubuntu1
I also use intel-microcode; remiving does not solve anything.
However, intramfs.tools_0.103ubuntu16 works well.
I had to chroot to downgrade to that one.

[Thomas Karl Pietrowski (thopiekar)]

Also happens here. Every newly installed linux versions fail here. Using amd-microcode here, so IMO it happens during upgrade of initramfs-tools.
Is there a workaround?

[Harry (harry33)]

The correct workaround or fix for the time being is to downgrade the version 0.120ubuntu1 to the working version 0.103ubuntu16.

You will need to chroot the root partition to do this.
I believe package intel-microcode has nothing to with this.

[Andy Whitcroft (apw)]

@Daniel -- ok could I have the broken /boot/initrd file from that it built without intel-microcode. Also I think I am going to need a transcript of the upgrade process as well, so as you install 120 and it eats your initrd. I cannot for the life of me tickle this in my installs.

[Andy Whitcroft (apw)]

@Daniel -- could you also run the update-initramfs -v -u -k `uname -r` after the upgrade of initramfs-tools and before you reboot. The output with -v should tell us what exactly it did when rebuilding. Attach what it produces and the initrd as well please. Thanks.

[Daniel Winzen (q-d-deactivatedaccount)]

Ok, I attached an archive with both, the old and the new initrd and update-initrd logs for comparison.
To minimize the size, I uninstalled the linux-image-extra-* and linux-firmware packages, but the issue is still the same.

[Andy Whitcroft (apw)]

@Daniel -- ok your log implies that somehow you do not have busybox-static installed, which is supprising me somewhat. Could you also append a dpkg -l listing please.

[Daniel Winzen (q-d-deactivatedaccount)]

@Andy -- Well, I thought that's what the busybox-initramfs package was for. I didn't have that package installed for years and never had any problems. But I just tested the upgrade again after installing busybox-static and it solves this issue.
Maybe busybox-static should be added as a dependency?

[Andy Whitcroft (apw)]

Ahh enlightenment dawns, actually you shoudn't need busybox-static and this is a hooks issue in busybox. Will poke that.

[Andy Whitcroft (apw)]

Ok, I think we have this one licked. The packages in the PPA below should sort this issue out. @Daniel -- if you could test them out and confirm that would be helpful, but I am fairly confident this is your issue (it is cirtainly _an_ issue). Please report any testing back here. Thanks:

    https://launchpad.net/~apw/+archive/ubuntu/initramfs-tools/

[Daniel Winzen (q-d-deactivatedaccount)]

@Andy -- Yes, the PPA version resolves the issue, booting fine. Thanks!

[Andy Whitcroft (apw)]

@Daniel -- thanks for the quick feedback.

[Harry (harry33)]

I can confirm too, the PPA solved the issue.
I also have only busybox-initramfs installed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package busybox - 1:1.22.0-9ubuntu2

---------------
busybox (1:1.22.0-9ubuntu2) wily; urgency=medium

  * Move zz-busybox to busybox-initramfs to ensure we get links to all
    the tools we need, stop shipping it anywhere else. (LP: #1481733)
  * Prefer busybox commands over klibc commands where there is duplication.

 -- Andy Whitcroft <email address hidden> Fri, 07 Aug 2015 13:03:51 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.120ubuntu2

---------------
initramfs-tools (0.120ubuntu2) wily; urgency=medium

    * update-initramfs: switch dpkg --compare-versions from > to >=
      to avoid warnings from dpkg. (LP: #1480340)
    * debian/control: update Vcs-* links for Ubuntu.
    * scripts/functions: use :> rather than touch to create fsck
      stamp. (LP: #1481789)
    * hooks/busybox: revert local changes, these now need to occur
      in the new late zz-busybox-initramfs hook. (LP: #1481733)

 -- Andy Whitcroft <email address hidden> Wed, 05 Aug 2015 13:02:34 +0100