kernel panic during MAAS fastpath install of Vivid images

[Brad Figg]

[Impact]
overlayfs support doesn't work with kernels > 3.18 which will, for example, break MAAS commissioning/deploying using trusty+hwe-v (when it exists).

[Test Case]
Create custom trusty image using the vivid kernel and attempt to commission a node with MAAS using that image.

[Regression Potential]
If the kernel detection code is buggy, it could regress functionality on older kernels (e.g. 3.13/3.16).

[Andy Whitcroft (apw)]

From the image we can see that the maas installer is using overlayroot to allow customisation of the image. It is not handling the new workdir= parameter which prevents successful mounting of root, and aborts the boot.

[Scott Moser (smoser)]

fix uploaded.
just for some more information, can fairly easily reproduce this bug using a cloud image:

img_url="http://cloud-images.ubuntu.com/daily/server/vivid/current/vivid-server-cloudimg-amd64-disk1.img"
img_dl="${img_url##*/}"
wget -O "$img_dl" "$img_url"
qemu-img convert -O qcow2 "$img_dl" "disk.img.dist"
qemu-img create -f qcow2 -b disk.img.dist disk.img

# pull out kernel/initramfs
sudo mount-image-callback disk.img --read-only -- \
   sh -xc 'cp $MOUNTPOINT/boot/vmlinu?-* kernel &&
           cp $MOUNTPOINT/boot/initrd.* initrd &&
           chmod ugo+r kernel initrd'

# boot qemu, should boot to bash prompt, currently on vivid (3.18 + 0.25ubuntu1) will kernel panic
qemu-system-x86_64 -enable-kvm -m 1024 \
   -drive if=virtio,file=disk.img \
   -kernel kernel -initrd initrd \
   -append "root=LABEL=cloudimg-rootfs overlayroot=tmpfs:debug=1 console=ttyS0 init=/bin/bash" \
   -nographic -echr 0x5

# patchin a new deb
sudo mount-image-callback disk.img -- \
   sh -exc '
     deb=$1; shift;
     cp $deb $MOUNTPOINT/tmp/fix.deb
     chroot $MOUNTPOINT dpkg -i /tmp/fix.deb
     rm $MOUNTPOINT/tmp/fix.deb' -- overlayroot_0.26ubuntu1_all.deb

# then re-copy out the kernel and boot again to see fix.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-initramfs-tools - 0.26ubuntu1

---------------
cloud-initramfs-tools (0.26ubuntu1) vivid; urgency=medium

  * cloud-initramfs-dyn-netconf: support BOOTIF= that contains
    upper case a-f.
  * overlayroot: use required 'workdir' option on overlayfs mount for
    kernels >= 3.18 (LP: #1411294)
 -- Scott Moser <email address hidden> Fri, 16 Jan 2015 12:01:06 -0500

[Brad Figg (brad-figg)]

I can verify that with today's daily image I was able to successfully install Vivid.

[dann frazier (dannf)]

I'm working on MAAS enablement for a system that will require a >= 3.18 kernel and I've hit this issue in trusty. I'm obviously using custom images since trusty doesn't include a newer kernel - but I wonder if this won't be a problem with hwe-v images starting in 14.04.3.

[jpo (jpo-9)]

The Fix in 0.26ubuntu1 does not address the fact that the module was renamed from overlayfs to overlay.

[dann frazier (dannf)]

On Wed, May 27, 2015 at 2:59 AM, jpo <email address hidden> wrote:
> The Fix in 0.26ubuntu1 does not address the fact that the module was
> renamed from overlayfs to overlay.

Note that the Ubuntu 15.04 kernel contains patches to provide
backwards compatibility:
  http://kernel.ubuntu.com/git/ubuntu/ubuntu-vivid.git/commit/fs/overlayfs?id=2b561327c40709b6f2145fe3c4c3159289653e7b

I do, however, agree that overlayroot should also support kernels w/o
these patches (e.g. upstream). I'd personally recommend filing that as
a separate bug though, as this one has been fixed and has reached a
terminal state.

  -dann

> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1411294
>
> Title:
> kernel panic during MAAS fastpath install of Vivid images
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/cloud-initramfs-tools/+bug/1411294/+subscriptions

[Scott Moser (smoser)]

jpo, please feel free to open a bug on cloud-initramfs-tools for proper 'overlay' support without using overlayfs path.

[Chris J Arges (arges)]

Hello Brad, or anyone else affected,

Accepted cloud-initramfs-tools into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-initramfs-tools/0.25ubuntu1.14.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Brad, or anyone else affected,

Accepted cloud-initramfs-tools into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-initramfs-tools/0.25ubuntu1.14.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Newell Jensen (newell-jensen)]

I tested https://launchpad.net/ubuntu/+archive/primary/+files/overlayroot_0.25ubuntu1.14.04.1_all.deb and the issue was resolved.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-initramfs-tools - 0.25ubuntu1.14.04.1

---------------
cloud-initramfs-tools (0.25ubuntu1.14.04.1) trusty-proposed; urgency=medium

  * overlayroot: use required 'workdir' option on overlayfs mount for
    kernels >= 3.18 (LP: #1411294)

 -- dann frazier <email address hidden> Tue, 19 May 2015 11:31:29 -0600

[Chris J Arges (arges)]

The verification of the Stable Release Update for cloud-initramfs-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-initramfs-tools - 0.25ubuntu1.14.10.1

---------------
cloud-initramfs-tools (0.25ubuntu1.14.10.1) utopic-proposed; urgency=medium

  * overlayroot: use required 'workdir' option on overlayfs mount for
    kernels >= 3.18 (LP: #1411294)

 -- dann frazier <email address hidden> Tue, 19 May 2015 11:26:46 -0600

[Scott Moser (smoser)]

Just a comment, as it is right now, attempt to install trusty with hwe-v will fail due to this bug.
In order to make that work, we need a released (or daily) image and specifically the related boot-initrd > 20150609.

That should happen in due course, the cloud-image daily builds had not succeeded since 20150609 and maas ephemeral are derived from those.

[Ben Howard (darkmuggle-deactivatedaccount)]

I am manually hand-holding a build through for the latest kernel CVE. Image builds and promotion are in the works. ETA tomorrow morning.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Scott Moser (smoser)]

Marking this fix-committed in maas-images just to mark that.
The fix is available in the daily stream of images . If you have a image version >= 20150611 you should should not see this.

See https://bugs.launchpad.net/maas/+bug/1465071/comments/7 for how to read /var/lib/maas/boot-resources/current/maas.meta and show what versions you have.

I guess we'll mark this fix-released when we have an image in the release stream.