mount-image-callback may not mount overlay when it could

[Scott Moser]

=== Begin SRU Template ===
[Impact]
The 16.10 kernel dropped a legacy kernel module alias that allowed usage of
the 'overlay' filesystem via name 'overlayfs'. This broke overlayroot as
it explicitly tried to to use 'overlayfs' by name in loading of modules and
also in entry in /etc/fstab.

Without this fix, overlayroot will simply not work on any upstream kernel
or Ubuntu kernel of 16.10 (yakkety) or later.

[Test Case]
Note, not applying proposed as shown in step 3 below will recreate failure.

1.) Start an instance of a cloud image.

2.) get a suitable 4.8 kernel
    On 16.10 or later, this is already done. On 16.04, we currently need to
    install the kernel team's PPA to get one.

    $ sudo apt-add-repository -y ppa:canonical-kernel-team/ppa
    $ sudo apt update -q && sudo apt install -y linux-virtual-hwe-16.04-edge </dev/null

3.) Enable proposed and install overlayroot to show fix.
    $ rel=$(lsb_release -sc)
    $ echo "deb http://archive.ubuntu.com/ubuntu $rel-proposed main" |
    $ sudo tee /etc/apt/sources.list.d/proposed.list
    $ sudo apt update -qy && sudo apt install -qy overlayroot </dev/null
    $ dpkg-query --show overlayroot
    overlayroot 0.27ubuntu1.3

4.) Enable overlayroot and reboot
    # remove the cloud-init written mount options for /dev/vdb
    # if we do not do this, then /mnt ends up not mounted due to ordering.
    $ sudo sed -i.dist s/,x-systemd.requires=cloud-init.service,comment=cloudconfig// /etc/fstab
    $ echo "overlayroot=tmpfs" | sudo tee /etc/overlayroot.local.conf
    $ sudo reboot

5.) log back in and look around.
   a.) check that 'overlayroot' is in /proc/mounts
       $ awk '$1 == "overlayroot" { print $0 }' /proc/mounts
       overlayroot / overlay rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0

   b.) check /run/initramfs/overlayroot.log
       $ grep success /run/initramfs/overlayroot.log
       [success]: configured root with 'tmpfs' using overlay per /dev/vda1/etc/overlayroot.local.conf

6.) try with recurse disabled

    Assuming you're on the same system and in an overlayroot, to change the
    file necessary, we use overlayroot-chroot.

    $ echo overlayroot=tmpfs:recurse=0 | sudo overlayroot-chroot tee /etc/overlayroot.local.conf
    $ sudo reboot

7.) log back in and look around.
    This time the /mnt should not have overlay on it.

    $ grep vdb /proc/mounts
    /dev/vdb /mnt ext4 rw,relatime,data=ordered 0 0
    $ grep overlay /proc/mounts
    overlayroot / overlay rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0
    $ cat /etc/overlayroot.local.conf
    overlayroot=tmpfs:recurse=0

[Regression Potential]
The most likely regression is just in failure for overlayroot to work.
That was the case 100% of the time on any kernel without 'overlayfs'
filesystem, so this can't really make things worse from that perspective.

Some of the code change was related to fixing another issue, with 'recurse'.
Testing recurse (where not just / is mounted as an overlayroot) is done
above
echo overlayroot=tmpfs:recurse=1 | sudo overlayroot-chroot tee /etc/overlayroot.local.conf

[Other Info]
The full overlayroot/init-bottom/overlayroot can be seen at [1]. The
specific change that fixed the issue is in revision 115 at [2].

Related bugs:
 * bug 1630274 adjusted the mechanism for determining if overlay was supported.
   The change to do so is included in the xenial backport for this bug.

[1] http://bazaar.launchpad.net/~cloud-initramfs-tools/cloud-initramfs-tools/trunk/view/head:/overlayroot/scripts/init-bottom/overlayroot
[2] http://bazaar.launchpad.net/~cloud-initramfs-tools/cloud-initramfs-tools/trunk/revision/115

=== End SRU Template ===

=== Original Description ===
As mentioned in LP: #1411294, it's now called 'overlay' instead of 'overlayfs'.
Ubuntu had patched the kernel for compatibility.

The Ubuntu kernels as of 4.8 (16.10 kernel) and possibly a bit before no longer have a overlayfs module either. Thus, this is now affecting yakkety.

(The original reporter is @~gpo-9.)

The changes recently added to mount-image-callback under bug 1493188 that were targeted at supporting 'overlay' filesystem type while still supporting 'overlayfs' may result in exiting with "unsupported" when a mount would work.

The changes basically said:
  if overlay in /proc/filesystems:
     return GOOD
  modprobe --quiet --use-blacklist overlay
     if overlay in /proc/filesystems:
        return GOOD
  return BAD

the path that fails now that previously succeeded is when 'modprobe' would not work, but 'mount -t overlay' *would*. This occurs in build systems where the running kernel is of one version and code is running in a chroot that does not have access to the kernel modules in /lib/modules/.

'mount -t overlay' works in this scenario because the mount abi call passes the filesystem type 'overlay', and the kernel then does:
   fs/filesystems.c: if (!fs && (request_module("fs-%.*s", len, name) == 0))

The kernel's scope is not chrooted, so the module gets loaded and everything is happy.

The new changes are to instead basically do:
  for fstype in overlay overlayfs; do
     mount -t $fstype <options> && return GOOD
     if /proc/filesystems is readable and $fstype in /proc/filesystems; then
         return FAIL
     fi
  done

[Scott Moser (smoser)]

marked this as critical, as it is blocking maas image builds for 16.10.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-utils - 0.29-0ubuntu5

---------------
cloud-utils (0.29-0ubuntu5) yakkety; urgency=medium

  * debian/control: update to Standards-Version 3.9.8
  * debian/copyright: fix lintian file-without-copyright-information.
  * sync to trunk at revno 306
    * mount-image-callback: try mounting overlay rather than checking first
      (LP: #1630274)

 -- Scott Moser <email address hidden> Tue, 04 Oct 2016 12:36:21 -0400

[Brian Murray (brian-murray)]

Hello Scott, or anyone else affected,

Accepted cloud-utils into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-utils/0.27-0ubuntu25.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted cloud-utils (0.27-0ubuntu25.2) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

lxc/2.0.11-0ubuntu1~16.04.3 (ppc64el, amd64, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#cloud-utils

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Chad Smith (chad.smith)]

While the test case run on xenial did show overlay properly mounted across reboot. The generated deb doesn't contain the applied patches to mount-image-callback per the upload changelog. Needs investigation before claiming victory here.

[Chad Smith (chad.smith)]

I extended the SRU verification test case to validate that expected
changes are present in mount-image-callback which loop over "fstype in overlay overlayfs".

I can confirm both that proper bits are uploaded and "overlay" fs is functional using cloud-utils (0.27-0ubuntu25.2).

#### Updated test script

#!/bin/bash

set -x

cat > install_hwe_kernel_and_overlay_root.sh <<EOF
#!/bin/bash

echo "3.) Enable proposed, install HWE kernel and install overlayroot to show fix."
mirror=http://archive.ubuntu.com/ubuntu
echo deb \$mirror xenial-proposed main | tee /etc/apt/sources.list.d/proposed.list
apt-get update
apt-get install -qy cloud-utils overlayroot linux-generic-hwe-16.04 | egrep 'linux-image|cloud-utils'
EOF

expect_overlayfs_mounts(){
  expected_mounts=$1
  overlay_mounts=$(multipass exec $VM mount | grep overlay -c)
  [[ "$expected_mounts" = "$overlay_mounts" ]] && echo "SUCCESS: found $overlay_mounts overlay mounts" || echo "FAILURE: expected $expected_mounts overlay mounts found $overlay_mounts"
}

VM=sru-1493188
RELEASE=xenial
multipass launch daily:$RELEASE -n $VM
multipass exec $VM -- cloud-init status --wait --long > /dev/null
echo --- No overlayfs mounts on initial boot
expect_overlayfs_mounts 0
multipass transfer install_hwe_kernel_and_overlay_root.sh $VM:.
multipass exec $VM -- sudo bash ./install_hwe_kernel_and_overlay_root.sh
echo "--- confirm overlay overlayfs patch present in mount-image-callback"
fs_types=$(multipass exec $VM -- grep -c "for fstype in overlay overlayfs" /usr/bin/mount-image-callback)
[[ "1" = "$fs_types" ]] || echo "FAILURE: didn't find 'for fstype in overlay overlayfs'"
multipass exec $VM -- sudo sh -c "echo overlayroot=tmpfs:recurse=0 | sudo tee -a /etc/overlayroot.local.conf"
multipass restart $VM
echo "--- expect 1 overlayfs mounts because using HWE kernel"
multipass exec $VM -- cloud-init status --wait --long
expect_overlayfs_mounts 1
multipass exec $VM -- sudo sh -c "echo overlayroot=tmpfs:recurse=1 | sudo overlayroot-chroot tee /etc/overlayroot.local.conf"
multipass restart $VM
expect_overlayfs_mounts 1
multipass exec $VM -- grep overlay /etc/fstab || echo "FAIL: didn't find overlayroot"
multipass exec $VM -- grep success /run/initramfs/overlayroot.log || echo "FAIL: didn't find overlayroot"

### Xenial verification results
+ cat
+ VM=sru-1493188
+ RELEASE=xenial
+ multipass launch daily:xenial -n sru-1493188
Launched: sru-1493188
+ multipass exec sru-1493188 -- cloud-init status --wait --long
+ echo --- No overlayfs mounts on initial boot
--- No overlayfs mounts on initial boot
+ expect_overlayfs_mounts 0
+ expected_mounts=0
++ multipass exec sru-1493188 mount
++ grep overlay -c
+ overlay_mounts=0
+ [[ 0 = \0 ]]
+ echo 'SUCCESS: found 0 overlay mounts'
SUCCESS: found 0 overlay mounts
+ multipass transfer install_hwe_kernel_and_overlay_root.sh sru-1493188:.
+ multipass exec sru-1493188 -- sudo bash ./install_hwe_kernel_and_overlay_root.sh
3.) Enable proposed, install HWE kernel and install overlayroot to show fix.
...

[Joshua Powers (powersj)]

Updating the tags so this turns green on the SRU board.

Was `verificiaton-complete` but it should be `verification-done` :)

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-utils - 0.27-0ubuntu25.2

---------------
cloud-utils (0.27-0ubuntu25.2) xenial; urgency=medium

  * debian/patches/lp-1493188-support-overlay-filesystem:
    mount-image-callback: support 'overlay' filesystem type rather
    than ubuntu specific 'overlayfs' (LP: #1493188)
  * debian/patches/lp-1630274-mount-overlay-first:
    mount-image-callback: try mounting overlay rather than checking first
    (LP: #1630274)

 -- Chad Smith <email address hidden> Fri, 04 Sep 2020 08:47:08 -0600

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for cloud-utils has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.