fails to boot with "Attempted to kill init" in VMWare, absolute /sbin/init symlink does not work

[Michael Bienia]

My Ubuntu 15.04 VM (running under VMware) doesn't boot anymore after permanently switching to systemd.
I've switched to systemd with "apt install systemd-sysv --purge" (it only purged the upstart package) and with the default "Ubuntu" entry the boot fails after about 2.5 sec with a kernel panic ("Attempted to kill init"). But it works when edit the grub entry (in the grub shell) and add "init=/lib/systemd/systemd".

Attached is the serial output (with "debug" but no "init=" parameter).

[Martin Pitt (pitti)]

Pretty much the entire log is from the initramfs, the root fs' /sbin/init indeed doesnt' seem to get called.

Can you please check ls -l /sbin/init in your VM? It should be a symlink to /lib/systemd/systemd.

Also, could you boot with "break=bottom", and check ls -l /root/sbin/init ? Does that exist, or was it removed/modified in some funky way?

[Michael Bienia (geser)]

From the booted system:
$ ls -l /sbin/init
lrwxrwxrwx 1 root root 20 Feb 10 10:59 /sbin/init -> /lib/systemd/systemd
$ ls -l /lib/systemd/systemd
-rwxr-xr-x 1 root root 1396464 Feb 10 11:00 /lib/systemd/systemd

For the output from the initramfs see the attached picture.

[Martin Pitt (pitti)]

Michael confirmed on IRC that it does work with a relative symlink, so this sounds like a bug in initramfs-tools. But it works fine with an absolute link on all my machines, VMs, LXCs etc, so I wonder if initramfs-tools does something special on VMWare?

[Michael Bienia (geser)]

I managed to some more output to the serial log (booted with "console=ttyS1,115200" only, no "console=tty0"):

starting version 219
[ 1.851984] sd 2:0:0:0: [sda] Assuming drive cache: write through
/init: line 307: chroot: not found
Target filesystem doesn't have requested /sbin/init.
/init: line 325: chroot: not found
/bin/sh: 0: Can't open splash
[ 2.909334] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00

[Martin Pitt (pitti)]

For the record, the chroot is done here:

        # Work around absolute symlinks
        if [ -d "${rootmnt}" ] && [ -h "${rootmnt}${checktarget}" ]; then
                case $(readlink "${rootmnt}${checktarget}") in /*)
                        checktarget="$(chroot ${rootmnt} readlink ${checktarget})"
                        ;;
                esac
        fi

So if chroot doesn't work, we would indeed fail if /sbin/init is an absolute symlink. However, /sbin/chroot is supposed to exist in the initramfs:

$ zcat /initrd.img |cpio -tv | grep chroot
-rwxr-xr-x 141 root root 0 Oct 27 09:40 sbin/chroot

How does that look for you? Perhaps you have a different initramfs config which skips that?

[Martin Pitt (pitti)]

Michael confirmed on IRC that with an explicit ln -s busybox ${DESTDIR}/bin/chroot in /usr/share/initramfs-tools/hooks/busybox it works. It's indeed a bit curious where the existing hardlink comes from..

[Martin Pitt (pitti)]

FTR, that happens if busybox-static is *not* installed; that's in ubuntu-standard, which explains why most people have it.

[Dimitri John Ledkov (xnox)]

Correct, ubuntu defaults to pulling in busybox-static into initramfs if available. In practice, we pretty much require it for all our initramfs and rarely don't have it. Maybe it should become a hard dependency on ubuntu?

However, given that we support booting with either systemd or upstart, shouldn't our initramfs search for /sbin/upstart or /lib/systemd/systemd direct rather than /sbin/init?

[Michael Bienia (geser)]

initramfs-tools already depends busybox-initramfs. What's missing is the hook to symlink bin/chroot to busybox in the initrd.
Supporting /sbin/init being a symlink was not fixed for all cases in bug #1351295. There was only a readlink symlink added but the chroot call in the following line was missed. By coincidence the busybox-static hook installed a chroot symlink so it worked for most cases (busybox-static is a dependency of ubuntu-standard).

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.103ubuntu13

---------------
initramfs-tools (0.103ubuntu13) vivid; urgency=medium

  * hooks/busybox: Create /bin/chroot symlink, to fix absolute symlinks when
    busybox-static is not installed. (LP: #1421117)
 -- Martin Pitt <email address hidden> Tue, 24 Feb 2015 11:56:53 +0100

[Will Rouesnel (w-rouesnel)]

So can this be backported to utopic? It's kind of a serious bug since it renders the system unbootable.