kernel panic during MAAS fastpath install of Vivid images

Bug #1411294 reported by Brad Figg on 2015-01-15
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
maas-images
High
Scott Moser
cloud-initramfs-tools (Ubuntu)
High
Scott Moser
Trusty
Undecided
dann frazier
Utopic
Undecided
dann frazier
linux (Ubuntu)
Trusty
Undecided
Unassigned
Utopic
Undecided
Unassigned

Bug Description

[Impact]
overlayfs support doesn't work with kernels > 3.18 which will, for example, break MAAS commissioning/deploying using trusty+hwe-v (when it exists).

[Test Case]
Create custom trusty image using the vivid kernel and attempt to commission a node with MAAS using that image.

[Regression Potential]
If the kernel detection code is buggy, it could regress functionality on older kernels (e.g. 3.13/3.16).

Related branches

Brad Figg (brad-figg) wrote :
description: updated
description: updated
Changed in linux (Ubuntu):
importance: Undecided → High
tags: added: kernel-da-key vivid
Andy Whitcroft (apw) on 2015-01-15
Changed in linux (Ubuntu):
milestone: none → ubuntu-15.01
Andy Whitcroft (apw) on 2015-01-15
Changed in cloud-initramfs-tools (Ubuntu):
status: New → Triaged
importance: Undecided → High
assignee: nobody → Andy Whitcroft (apw)
Andy Whitcroft (apw) wrote :

From the image we can see that the maas installer is using overlayroot to allow customisation of the image. It is not handling the new workdir= parameter which prevents successful mounting of root, and aborts the boot.

Changed in cloud-initramfs-tools (Ubuntu):
assignee: Andy Whitcroft (apw) → nobody
Scott Moser (smoser) on 2015-01-16
Changed in cloud-initramfs-tools (Ubuntu):
assignee: nobody → Scott Moser (smoser)
status: Triaged → In Progress
Scott Moser (smoser) wrote :

fix uploaded.
just for some more information, can fairly easily reproduce this bug using a cloud image:

img_url="http://cloud-images.ubuntu.com/daily/server/vivid/current/vivid-server-cloudimg-amd64-disk1.img"
img_dl="${img_url##*/}"
wget -O "$img_dl" "$img_url"
qemu-img convert -O qcow2 "$img_dl" "disk.img.dist"
qemu-img create -f qcow2 -b disk.img.dist disk.img

# pull out kernel/initramfs
sudo mount-image-callback disk.img --read-only -- \
   sh -xc 'cp $MOUNTPOINT/boot/vmlinu?-* kernel &&
           cp $MOUNTPOINT/boot/initrd.* initrd &&
           chmod ugo+r kernel initrd'

# boot qemu, should boot to bash prompt, currently on vivid (3.18 + 0.25ubuntu1) will kernel panic
qemu-system-x86_64 -enable-kvm -m 1024 \
   -drive if=virtio,file=disk.img \
   -kernel kernel -initrd initrd \
   -append "root=LABEL=cloudimg-rootfs overlayroot=tmpfs:debug=1 console=ttyS0 init=/bin/bash" \
   -nographic -echr 0x5

# patchin a new deb
sudo mount-image-callback disk.img -- \
   sh -exc '
     deb=$1; shift;
     cp $deb $MOUNTPOINT/tmp/fix.deb
     chroot $MOUNTPOINT dpkg -i /tmp/fix.deb
     rm $MOUNTPOINT/tmp/fix.deb' -- overlayroot_0.26ubuntu1_all.deb

# then re-copy out the kernel and boot again to see fix.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-initramfs-tools - 0.26ubuntu1

---------------
cloud-initramfs-tools (0.26ubuntu1) vivid; urgency=medium

  * cloud-initramfs-dyn-netconf: support BOOTIF= that contains
    upper case a-f.
  * overlayroot: use required 'workdir' option on overlayfs mount for
    kernels >= 3.18 (LP: #1411294)
 -- Scott Moser <email address hidden> Fri, 16 Jan 2015 12:01:06 -0500

Changed in cloud-initramfs-tools (Ubuntu):
status: In Progress → Fix Released
Andy Whitcroft (apw) on 2015-01-21
Changed in linux (Ubuntu):
status: Triaged → Invalid
Brad Figg (brad-figg) wrote :

I can verify that with today's daily image I was able to successfully install Vivid.

dann frazier (dannf) wrote :

I'm working on MAAS enablement for a system that will require a >= 3.18 kernel and I've hit this issue in trusty. I'm obviously using custom images since trusty doesn't include a newer kernel - but I wonder if this won't be a problem with hwe-v images starting in 14.04.3.

Changed in linux (Ubuntu Trusty):
status: New → Invalid
Changed in cloud-initramfs-tools (Ubuntu Trusty):
status: New → Confirmed
dann frazier (dannf) on 2015-05-20
Changed in cloud-initramfs-tools (Ubuntu Trusty):
assignee: nobody → dann frazier (dannf)
Changed in cloud-initramfs-tools (Ubuntu Utopic):
assignee: nobody → dann frazier (dannf)
Changed in cloud-initramfs-tools (Ubuntu Trusty):
status: Confirmed → In Progress
Changed in cloud-initramfs-tools (Ubuntu Utopic):
status: New → In Progress
dann frazier (dannf) on 2015-05-20
description: updated
jpo (jpo-9) wrote :

The Fix in 0.26ubuntu1 does not address the fact that the module was renamed from overlayfs to overlay.

On Wed, May 27, 2015 at 2:59 AM, jpo <email address hidden> wrote:
> The Fix in 0.26ubuntu1 does not address the fact that the module was
> renamed from overlayfs to overlay.

Note that the Ubuntu 15.04 kernel contains patches to provide
backwards compatibility:
  http://kernel.ubuntu.com/git/ubuntu/ubuntu-vivid.git/commit/fs/overlayfs?id=2b561327c40709b6f2145fe3c4c3159289653e7b

I do, however, agree that overlayroot should also support kernels w/o
these patches (e.g. upstream). I'd personally recommend filing that as
a separate bug though, as this one has been fixed and has reached a
terminal state.

  -dann

> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1411294
>
> Title:
> kernel panic during MAAS fastpath install of Vivid images
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/cloud-initramfs-tools/+bug/1411294/+subscriptions

Scott Moser (smoser) wrote :

jpo, please feel free to open a bug on cloud-initramfs-tools for proper 'overlay' support without using overlayfs path.

Hello Brad, or anyone else affected,

Accepted cloud-initramfs-tools into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-initramfs-tools/0.25ubuntu1.14.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-initramfs-tools (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Chris J Arges (arges) wrote :

Hello Brad, or anyone else affected,

Accepted cloud-initramfs-tools into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-initramfs-tools/0.25ubuntu1.14.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-initramfs-tools (Ubuntu Utopic):
status: In Progress → Fix Committed
Newell Jensen (newell-jensen) wrote :
tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-initramfs-tools - 0.25ubuntu1.14.04.1

---------------
cloud-initramfs-tools (0.25ubuntu1.14.04.1) trusty-proposed; urgency=medium

  * overlayroot: use required 'workdir' option on overlayfs mount for
    kernels >= 3.18 (LP: #1411294)

 -- dann frazier <email address hidden> Tue, 19 May 2015 11:31:29 -0600

Changed in cloud-initramfs-tools (Ubuntu Trusty):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for cloud-initramfs-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-initramfs-tools - 0.25ubuntu1.14.10.1

---------------
cloud-initramfs-tools (0.25ubuntu1.14.10.1) utopic-proposed; urgency=medium

  * overlayroot: use required 'workdir' option on overlayfs mount for
    kernels >= 3.18 (LP: #1411294)

 -- dann frazier <email address hidden> Tue, 19 May 2015 11:26:46 -0600

Changed in cloud-initramfs-tools (Ubuntu Utopic):
status: Fix Committed → Fix Released
Scott Moser (smoser) wrote :

Just a comment, as it is right now, attempt to install trusty with hwe-v will fail due to this bug.
In order to make that work, we need a released (or daily) image and specifically the related boot-initrd > 20150609.

That should happen in due course, the cloud-image daily builds had not succeeded since 20150609 and maas ephemeral are derived from those.

I am manually hand-holding a build through for the latest kernel CVE. Image builds and promotion are in the works. ETA tomorrow morning.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux (Ubuntu Utopic):
status: New → Confirmed
Scott Moser (smoser) wrote :

Marking this fix-committed in maas-images just to mark that.
The fix is available in the daily stream of images . If you have a image version >= 20150611 you should should not see this.

See https://bugs.launchpad.net/maas/+bug/1465071/comments/7 for how to read /var/lib/maas/boot-resources/current/maas.meta and show what versions you have.

I guess we'll mark this fix-released when we have an image in the release stream.

Changed in linux (Ubuntu Utopic):
status: Confirmed → Invalid
Changed in maas-images:
assignee: nobody → Scott Moser (smoser)
importance: Undecided → High
status: New → Fix Committed
Scott Moser (smoser) on 2015-10-29
no longer affects: linux (Ubuntu)
Scott Moser (smoser) on 2016-08-23
Changed in maas-images:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers