mount-image-callback may not mount overlay when it could

[Scott Moser]

=== Begin SRU Template ===
[Impact]
The 16.10 kernel dropped a legacy kernel module alias that allowed usage of
the 'overlay' filesystem via name 'overlayfs'. This broke overlayroot as
it explicitly tried to to use 'overlayfs' by name in loading of modules and
also in entry in /etc/fstab.

Without this fix, overlayroot will simply not work on any upstream kernel
or Ubuntu kernel of 16.10 (yakkety) or later.

[Test Case]
Note, not applying proposed as shown in step 3 below will recreate failure.

1.) Start an instance of a cloud image.

2.) get a suitable 4.8 kernel
    On 16.10 or later, this is already done. On 16.04, we currently need to
    install the kernel team's PPA to get one.

    $ sudo apt-add-repository -y ppa:canonical-kernel-team/ppa
    $ sudo apt update -q && sudo apt install -y linux-virtual-hwe-16.04-edge </dev/null

3.) Enable proposed and install overlayroot to show fix.
    $ rel=$(lsb_release -sc)
    $ echo "deb http://archive.ubuntu.com/ubuntu $rel-proposed main" |
    $ sudo tee /etc/apt/sources.list.d/proposed.list
    $ sudo apt update -qy && sudo apt install -qy overlayroot </dev/null
    $ dpkg-query --show overlayroot
    overlayroot 0.27ubuntu1.3

4.) Enable overlayroot and reboot
    # remove the cloud-init written mount options for /dev/vdb
    # if we do not do this, then /mnt ends up not mounted due to ordering.
    $ sudo sed -i.dist s/,x-systemd.requires=cloud-init.service,comment=cloudconfig// /etc/fstab
    $ echo "overlayroot=tmpfs" | sudo tee /etc/overlayroot.local.conf
    $ sudo reboot

5.) log back in and look around.
   a.) check that 'overlayroot' is in /proc/mounts
       $ awk '$1 == "overlayroot" { print $0 }' /proc/mounts
       overlayroot / overlay rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0

   b.) check /run/initramfs/overlayroot.log
       $ grep success /run/initramfs/overlayroot.log
       [success]: configured root with 'tmpfs' using overlay per /dev/vda1/etc/overlayroot.local.conf

6.) try with recurse disabled

    Assuming you're on the same system and in an overlayroot, to change the
    file necessary, we use overlayroot-chroot.

    $ echo overlayroot=tmpfs:recurse=0 | sudo overlayroot-chroot tee /etc/overlayroot.local.conf
    $ sudo reboot

7.) log back in and look around.
    This time the /mnt should not have overlay on it.

    $ grep vdb /proc/mounts
    /dev/vdb /mnt ext4 rw,relatime,data=ordered 0 0
    $ grep overlay /proc/mounts
    overlayroot / overlay rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0
    $ cat /etc/overlayroot.local.conf
    overlayroot=tmpfs:recurse=0

[Regression Potential]
The most likely regression is just in failure for overlayroot to work.
That was the case 100% of the time on any kernel without 'overlayfs'
filesystem, so this can't really make things worse from that perspective.

Some of the code change was related to fixing another issue, with 'recurse'.
Testing recurse (where not just / is mounted as an overlayroot) is done
above
echo overlayroot=tmpfs:recurse=1 | sudo overlayroot-chroot tee /etc/overlayroot.local.conf

[Other Info]
The full overlayroot/init-bottom/overlayroot can be seen at [1]. The
specific change that fixed the issue is in revision 115 at [2].

Related bugs:
 * bug 1630274 adjusted the mechanism for determining if overlay was supported.
   The change to do so is included in the xenial backport for this bug.

[1] http://bazaar.launchpad.net/~cloud-initramfs-tools/cloud-initramfs-tools/trunk/view/head:/overlayroot/scripts/init-bottom/overlayroot
[2] http://bazaar.launchpad.net/~cloud-initramfs-tools/cloud-initramfs-tools/trunk/revision/115

=== End SRU Template ===

=== Original Description ===
As mentioned in LP: #1411294, it's now called 'overlay' instead of 'overlayfs'.
Ubuntu had patched the kernel for compatibility.

The Ubuntu kernels as of 4.8 (16.10 kernel) and possibly a bit before no longer have a overlayfs module either. Thus, this is now affecting yakkety.

(The original reporter is @~gpo-9.)

The changes recently added to mount-image-callback under bug 1493188 that were targeted at supporting 'overlay' filesystem type while still supporting 'overlayfs' may result in exiting with "unsupported" when a mount would work.

The changes basically said:
  if overlay in /proc/filesystems:
     return GOOD
  modprobe --quiet --use-blacklist overlay
     if overlay in /proc/filesystems:
        return GOOD
  return BAD

the path that fails now that previously succeeded is when 'modprobe' would not work, but 'mount -t overlay' *would*. This occurs in build systems where the running kernel is of one version and code is running in a chroot that does not have access to the kernel modules in /lib/modules/.

'mount -t overlay' works in this scenario because the mount abi call passes the filesystem type 'overlay', and the kernel then does:
   fs/filesystems.c: if (!fs && (request_module("fs-%.*s", len, name) == 0))

The kernel's scope is not chrooted, so the module gets loaded and everything is happy.

The new changes are to instead basically do:
  for fstype in overlay overlayfs; do
     mount -t $fstype <options> && return GOOD
     if /proc/filesystems is readable and $fstype in /proc/filesystems; then
         return FAIL
     fi
  done

[Scott Moser (smoser)]

marked this as critical, as it is blocking maas image builds for 16.10.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-utils - 0.29-0ubuntu5

---------------
cloud-utils (0.29-0ubuntu5) yakkety; urgency=medium

  * debian/control: update to Standards-Version 3.9.8
  * debian/copyright: fix lintian file-without-copyright-information.
  * sync to trunk at revno 306
    * mount-image-callback: try mounting overlay rather than checking first
      (LP: #1630274)

 -- Scott Moser <email address hidden> Tue, 04 Oct 2016 12:36:21 -0400

[Brian Murray (brian-murray)]

Hello Scott, or anyone else affected,

Accepted cloud-utils into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-utils/0.27-0ubuntu25.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted cloud-utils (0.27-0ubuntu25.2) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

lxc/2.0.11-0ubuntu1~16.04.3 (ppc64el, amd64, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#cloud-utils

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!