"overlayfs" no longer exists

[宋文武]

=== Begin SRU Template ===
[Impact]
The 16.10 kernel dropped a legacy kernel module alias that allowed usage of
the 'overlay' filesystem via name 'overlayfs'. This broke overlayroot as
it explicitly tried to to use 'overlayfs' by name in loading of modules and
also in entry in /etc/fstab.

Without this fix, overlayroot will simply not work on any upstream kernel
or Ubuntu kernel of 16.10 (yakkety) or later.

[Test Case]
Note, not applying proposed as shown in step 3 below will recreate failure.

1.) Start an instance of a cloud image.

2.) get a suitable 4.8 kernel
    On 16.10 or later, this is already done. On 16.04, we currently need to
    install the kernel team's PPA to get one.

    $ sudo apt-add-repository -y ppa:canonical-kernel-team/ppa
    $ sudo apt update -q && sudo apt install -y linux-virtual-hwe-16.04-edge </dev/null

3.) Enable proposed and install overlayroot to show fix.
    $ rel=$(lsb_release -sc)
    $ echo "deb http://archive.ubuntu.com/ubuntu $rel-proposed main" |
    $ sudo tee /etc/apt/sources.list.d/proposed.list
    $ sudo apt update -qy && sudo apt install -qy overlayroot </dev/null
    $ dpkg-query --show overlayroot
    overlayroot 0.27ubuntu1.3

4.) Enable overlayroot and reboot
    # remove the cloud-init written mount options for /dev/vdb
    # if we do not do this, then /mnt ends up not mounted due to ordering.
    $ sudo sed -i.dist s/,x-systemd.requires=cloud-init.service,comment=cloudconfig// /etc/fstab
    $ echo "overlayroot=tmpfs" | sudo tee /etc/overlayroot.local.conf
    $ sudo reboot

5.) log back in and look around.
   a.) check that 'overlayroot' is in /proc/mounts
       $ awk '$1 == "overlayroot" { print $0 }' /proc/mounts
       overlayroot / overlay rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0

   b.) check /run/initramfs/overlayroot.log
       $ grep success /run/initramfs/overlayroot.log
       [success]: configured root with 'tmpfs' using overlay per /dev/vda1/etc/overlayroot.local.conf

6.) try with recurse disabled

    Assuming you're on the same system and in an overlayroot, to change the
    file necessary, we use overlayroot-chroot.

    $ echo overlayroot=tmpfs:recurse=0 | sudo overlayroot-chroot tee /etc/overlayroot.local.conf
    $ sudo reboot

7.) log back in and look around.
    This time the /mnt should not have overlay on it.

    $ grep vdb /proc/mounts
    /dev/vdb /mnt ext4 rw,relatime,data=ordered 0 0
    $ grep overlay /proc/mounts
    overlayroot / overlay rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0
    $ cat /etc/overlayroot.local.conf
    overlayroot=tmpfs:recurse=0

[Regression Potential]
The most likely regression is just in failure for overlayroot to work.
That was the case 100% of the time on any kernel without 'overlayfs'
filesystem, so this can't really make things worse from that perspective.

Some of the code change was related to fixing another issue, with 'recurse'.
Testing recurse (where not just / is mounted as an overlayroot) is done
above
echo overlayroot=tmpfs:recurse=1 | sudo overlayroot-chroot tee /etc/overlayroot.local.conf

[Other Info]
The full overlayroot/init-bottom/overlayroot can be seen at [1]. The
specific change that fixed the issue is in revision 115 at [2].

Related bugs:
 * bug 1630274 adjusted the mechanism for determining if overlay was supported.
   The change to do so is included in the xenial backport for this bug.

[1] http://bazaar.launchpad.net/~cloud-initramfs-tools/cloud-initramfs-tools/trunk/view/head:/overlayroot/scripts/init-bottom/overlayroot
[2] http://bazaar.launchpad.net/~cloud-initramfs-tools/cloud-initramfs-tools/trunk/revision/115

=== End SRU Template ===

As mentioned in LP: #1411294, it's now called 'overlay' instead of 'overlayfs'.
Ubuntu had patched the kernel for compatibility.

The Ubuntu kernels as of 4.8 (16.10 kernel) and possibly a bit before no longer have a overlayfs module either. Thus, this is now affecting yakkety.

(The original reporter is @~gpo-9.)

[Marc Deslauriers (mdeslaur)]

Thanks for taking the time to report this bug and helping to make Ubuntu better. We appreciate the difficulties you are facing, but this appears to be a "regular" (non-security) bug. I have unmarked it as a security issue since this bug does not show evidence of allowing attackers to cross privilege boundaries nor directly cause loss of data/privacy. Please feel free to report any other bugs you may find.

[宋文武 (iyzsong)]

Oh, sorry, I don't mean to set it as a 'security' bug.
Thanks!

[Scott Moser (smoser)]

I've marked cloud-utils as relevant here tto fix mount-image-callback as well.

[Scott Moser (smoser)]

for some specific information on different ubuntu kernels:

-- latest kernel before 4.8 entered yakkety --
$ uname -r
4.4.0-9136-generic

$ grep overlay /lib/modules/4.4.0-9136-generic/modules.alias
alias overlayfs overlay
alias fs-overlayfs overlay
alias fs-overlay overlay

$ sudo modprobe -v overlayfs
insmod /lib/modules/4.4.0-9136-generic/kernel/fs/overlayfs/overlay.ko

$ lsmod | grep overlay
overlay

$ grep overlay /proc/filesystems
nodev overlayfs
nodev overlay

-- 4.8 kernel in yakkety --
$ uname -r
4.8.0-15-generic

$ grep overlay /lib/modules/4.8.0-15-generic/modules.alias
alias fs-overlay overlay

$ modprobe -v overlayfs
modprobe: FATAL: Module overlayfs not found in directory /lib/modules/4.8.0-15-generic

$ modprobe -v overlay
insmod /lib/modules/4.8.0-15-generic/kernel/fs/overlayfs/overlay.ko

$ grep overlay /proc/filesystems
nodev overlay

-- 3.13 kernel (trusty GA) --
$ uname -r
3.13.0-96-generic

$ grep overlay /lib/modules/3.13.0-96-generic/modules.alias
alias fs-overlayfs overlayfs

$ sudo modprobe -v overlay
modprobe: FATAL: Module overlay not found.

$ sudo modprobe -v overlayfs
insmod /lib/modules/3.13.0-96-generic/kernel/fs/overlayfs/overlayfs.ko

$ grep overlay /proc/filesystems
nodev overlayfs

[Steve Langasek (vorlon)]

Hello 宋文武, or anyone else affected,

Accepted cloud-initramfs-tools into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-initramfs-tools/0.29ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Shuhao (shuhao)]

I can test this too if we build a copy for xenial for arm (raspi3).

[Scott Moser (smoser)]

I've verified functional in yakkety in uvt-kvm.
Below 'ussh' is a wrapper arond uvt-kvm
  https://gist.github.com/smoser/88a5a77ab0debf268b945d46314ea447
for reader purposes, it can be assumed that it operates like 'lxc exec'. ie,
   ussh name command arguments
executed command and arguments on 'name'.

# make a new system.
name=sm-y1
uvt-kvm create $name release=yakkety

# enable proposed
$ ussh $name sudo sh -c 'echo "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-proposed main" >/etc/apt/sources.list.d/proposed.list && apt-get update -qy'
# install a package
ussh $name sudo apt-get install -qy overlayroot

# show.
$ ussh $name dpkg-query --show overlayroot
overlayroot 0.29ubuntu1

# enable overlayroot without recurse
$ ussh "$name" sh -c 'echo overlayroot=tmpfs:recurse=0 | sudo tee /etc/overlayroot.local.conf'

# reboot
$ ussh $name reboot

# show
$ ussh $name grep overlay /proc/mounts
overlayroot / overlay rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0

# turn on recurse
$ ussh "$name" sh -c 'echo overlayroot=tmpfs:recurse=1 | sudo overlayroot-chroot tee /etc/overlayroot.local.conf'

# reboot
$ ussh $name reboot

# show /proc/mounts
$ ussh "$name" grep overlay /proc/mounts
overlayroot / overlay rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0
/media/root-ro/mnt /mnt overlay rw,relatime,lowerdir=/media/root-ro/mnt,upperdir=/media/root-rw/overlay/mnt,workdir=/media/root-rw/overlay-workdir/mnt 0 0

$ ussh $name grep overlay /etc/fstab
# This fstab is in an overlay. The real one can be found at
# sudo overlayroot-chroot
/media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ 0 0
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
/media/root-ro/mnt /mnt overlay lowerdir=/media/root-ro/mnt,upperdir=/media/root-rw/overlay/mnt,workdir=/media/root-rw/overlay-workdir/mnt 0 0

# show kernel version
$ ussh $name uname -r
4.8.0-17-generic

# boot into normal root
$ ussh sm-y1 sudo sh -c 'overlayroot-chroot rm /etc/overlayroot.local.conf && reboot'

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-initramfs-tools - 0.29ubuntu1

---------------
cloud-initramfs-tools (0.29ubuntu1) yakkety; urgency=medium

  * overlayroot: support 'overlay' filesystem explicitly rather than
    relying on ubuntu specific kernel module 'overlayfs'. (LP: #1493188)

 -- Scott Moser <email address hidden> Fri, 23 Sep 2016 17:00:37 -0400

[Scott Moser (smoser)]

fix committed in cloud-utils trunk at revno 303.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-utils - 0.29-0ubuntu4

---------------
cloud-utils (0.29-0ubuntu4) yakkety; urgency=medium

  * sync to trunk at revno 303
    * mount-image-callback: support 'overlay' filesystem type rather
      than ubuntu specific 'overlayfs' (LP: #1493188)

 -- Scott Moser <email address hidden> Tue, 27 Sep 2016 21:20:58 -0400

[Chris Halse Rogers (raof)]

Hello 宋文武, or anyone else affected,

Accepted cloud-initramfs-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-initramfs-tools/0.27ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

I've verified this on xenial using the steps provided in the test case above.
$ uname -r
4.8.0-28-generic
$ dpkg -S /boot/vmlinuz-$(uname -r)
linux-image-4.8.0-28-generic: /boot/vmlinuz-4.8.0-28-generic
$ dpkg-query --show overlayroot
overlayroot 0.27ubuntu1.3

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-initramfs-tools - 0.27ubuntu1.3

---------------
cloud-initramfs-tools (0.27ubuntu1.3) xenial; urgency=medium

  * sync with upstream at 0.32ubuntu1 (revno 129)
    * overlayroot: fix overlayroot=crypt with newer initramfs-tools
      (LP: #1634310)
    * overlayroot: support random seed from systemd systemd-random-seed.service
    * cloud-initramfs-dyn-netconf: Further adjustments to support
      IPV6 and the new DEVICE6 and net6-DEVICE.conf files. (LP: #1621615)
    * overlayroot: support 'overlay' filesystem explicitly rather than
      relying on ubuntu specific kernel module 'overlayfs'. (LP: #1493188)
    * overlayroot: write debug to /run/initramfs not /dev/.initramfs but
      support writing to /dev/.initramfs if that is all there is. (LP: #1485752)
    * overlayroot: fix overlayroot if recurse=0 was not provided.
      This fuctionality was lost since workdir support was added. (LP: #1619459)
    * whitespace cleanup.

 -- Scott Moser <email address hidden> Mon, 28 Nov 2016 20:12:51 -0500

[Robie Basak (racb)]

The verification of the Stable Release Update for cloud-initramfs-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Scott Moser (smoser)]

this fix is present in cloud-utils 0.30

[Robert C Jennings (rcj)]

With the move to Bionic host kernels on the builders, the Xenial MAAS images fail to build. This is because cloud-utils mount-image-callback remains unfixed in Xenial while the Bionic kernel does not have the overlayfs kernel module. The following patch at minimum is required in Xenial:

https://bazaar.launchpad.net/~smoser/cloud-utils/trunk.lp1493188/revision/303

Could we have this fix SRUed to Xenial please?

[Chad Smith (chad.smith)]

Proposed branch is up for xenial handling overlayfs issues.

https://code.launchpad.net/~chad.smith/ubuntu/+source/cloud-utils/+git/cloud-utils/+ref/upload-xenial-handle-missing-overlayfs

 The review has been approved and a test ppa is setup with cloud-utils version 0.27-0ubuntu25.2 on Xenial which should resolve this issue.

If there is an opportunity to this this to confirm the fix, that would be great:

Verification procedure:
sudo add-apt-repository ppa:chad.smith/cloud-utils-uploads -y
sudo apt-get update
sudo apt-get install cloud-utils=0.27-0ubuntu25.2

[Robert C Jennings (rcj)]

@chad.smith I have used the cloud-utils in your ppa to build Xenial maas-images on a bionic host to confirm that things are working now.

[Brian Murray (brian-murray)]

Hello 宋文武, or anyone else affected,

Accepted cloud-utils into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-utils/0.27-0ubuntu25.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted cloud-utils (0.27-0ubuntu25.2) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

lxc/2.0.11-0ubuntu1~16.04.3 (ppc64el, amd64, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#cloud-utils

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Chad Smith (chad.smith)]

While the test case run on xenial did show overlay properly mounted across reboot. The generated deb doesn't contain the applied patches to mount-image-callback per the upload changelog. Needs investigation before claiming victory here.

[Chad Smith (chad.smith)]

Extended the SRU verification test case to validate that expected
changes are present in mount-image-callback which loop over "fstype in overlay overlayfs".

I can confirm both that proper bits are uploaded in any overlay fs is functional using cloud-utils (0.27-0ubuntu25.2).

#### Updated test script

#!/bin/bash

set -x

cat > install_hwe_kernel_and_overlay_root.sh <<EOF
#!/bin/bash

echo "3.) Enable proposed, install HWE kernel and install overlayroot to show fix."
mirror=http://archive.ubuntu.com/ubuntu
echo deb \$mirror xenial-proposed main | tee /etc/apt/sources.list.d/proposed.list
apt-get update
apt-get install -qy cloud-utils overlayroot linux-generic-hwe-16.04 | egrep 'linux-image|cloud-utils'
EOF

expect_overlayfs_mounts(){
  expected_mounts=$1
  overlay_mounts=$(multipass exec $VM mount | grep overlay -c)
  [[ "$expected_mounts" = "$overlay_mounts" ]] && echo "SUCCESS: found $overlay_mounts overlay mounts" || echo "FAILURE: expected $expected_mounts overlay mounts found $overlay_mounts"
}

VM=sru-1493188
RELEASE=xenial
multipass launch daily:$RELEASE -n $VM
multipass exec $VM -- cloud-init status --wait --long > /dev/null
echo --- No overlayfs mounts on initial boot
expect_overlayfs_mounts 0
multipass transfer install_hwe_kernel_and_overlay_root.sh $VM:.
multipass exec $VM -- sudo bash ./install_hwe_kernel_and_overlay_root.sh
echo "--- confirm overlay overlayfs patch present in mount-image-callback"
fs_types=$(multipass exec $VM -- grep -c "for fstype in overlay overlayfs" /usr/bin/mount-image-callback)
[[ "1" = "$fs_types" ]] || echo "FAILURE: didn't find 'for fstype in overlay overlayfs'"
multipass exec $VM -- sudo sh -c "echo overlayroot=tmpfs:recurse=0 | sudo tee -a /etc/overlayroot.local.conf"
multipass restart $VM
echo "--- expect 1 overlayfs mounts because using HWE kernel"
multipass exec $VM -- cloud-init status --wait --long
expect_overlayfs_mounts 1
multipass exec $VM -- sudo sh -c "echo overlayroot=tmpfs:recurse=1 | sudo overlayroot-chroot tee /etc/overlayroot.local.conf"
multipass restart $VM
expect_overlayfs_mounts 1
multipass exec $VM -- grep overlay /etc/fstab || echo "FAIL: didn't find overlayroot"
multipass exec $VM -- grep success /run/initramfs/overlayroot.log || echo "FAIL: didn't find overlayroot"

### Xenial verification results
+ cat
+ VM=sru-1493188
+ RELEASE=xenial
+ multipass launch daily:xenial -n sru-1493188
Launched: sru-1493188
+ multipass exec sru-1493188 -- cloud-init status --wait --long
+ echo --- No overlayfs mounts on initial boot
--- No overlayfs mounts on initial boot
+ expect_overlayfs_mounts 0
+ expected_mounts=0
++ multipass exec sru-1493188 mount
++ grep overlay -c
+ overlay_mounts=0
+ [[ 0 = \0 ]]
+ echo 'SUCCESS: found 0 overlay mounts'
SUCCESS: found 0 overlay mounts
+ multipass transfer install_hwe_kernel_and_overlay_root.sh sru-1493188:.
+ multipass exec sru-1493188 -- sudo bash ./install_hwe_kernel_and_overlay_root.sh
3.) Enable proposed, install HWE kernel and install overlayroot to show fix.
d...

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-utils - 0.27-0ubuntu25.2

---------------
cloud-utils (0.27-0ubuntu25.2) xenial; urgency=medium

  * debian/patches/lp-1493188-support-overlay-filesystem:
    mount-image-callback: support 'overlay' filesystem type rather
    than ubuntu specific 'overlayfs' (LP: #1493188)
  * debian/patches/lp-1630274-mount-overlay-first:
    mount-image-callback: try mounting overlay rather than checking first
    (LP: #1630274)

 -- Chad Smith <email address hidden> Fri, 04 Sep 2020 08:47:08 -0600