"overlayfs" no longer exists
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| cloud-initramfs-tools |
Medium
|
Scott Moser | ||
| cloud-utils |
Medium
|
Unassigned | ||
| cloud-initramfs-tools (Ubuntu) |
Medium
|
Unassigned | ||
| Xenial |
Medium
|
Unassigned | ||
| Yakkety |
Medium
|
Unassigned | ||
| cloud-utils (Ubuntu) |
Medium
|
Unassigned | ||
| Xenial |
Medium
|
Chad Smith |
Bug Description
=== Begin SRU Template ===
[Impact]
The 16.10 kernel dropped a legacy kernel module alias that allowed usage of
the 'overlay' filesystem via name 'overlayfs'. This broke overlayroot as
it explicitly tried to to use 'overlayfs' by name in loading of modules and
also in entry in /etc/fstab.
Without this fix, overlayroot will simply not work on any upstream kernel
or Ubuntu kernel of 16.10 (yakkety) or later.
[Test Case]
Note, not applying proposed as shown in step 3 below will recreate failure.
1.) Start an instance of a cloud image.
2.) get a suitable 4.8 kernel
On 16.10 or later, this is already done. On 16.04, we currently need to
install the kernel team's PPA to get one.
$ sudo apt-add-repository -y ppa:canonical-
$ sudo apt update -q && sudo apt install -y linux-virtual-
3.) Enable proposed and install overlayroot to show fix.
$ rel=$(lsb_release -sc)
$ echo "deb http://
$ sudo tee /etc/apt/
$ sudo apt update -qy && sudo apt install -qy overlayroot </dev/null
$ dpkg-query --show overlayroot
overlayroot 0.27ubuntu1.3
4.) Enable overlayroot and reboot
# remove the cloud-init written mount options for /dev/vdb
# if we do not do this, then /mnt ends up not mounted due to ordering.
$ sudo sed -i.dist s/,x-systemd.
$ echo "overlayroot=tmpfs" | sudo tee /etc/overlayroo
$ sudo reboot
5.) log back in and look around.
a.) check that 'overlayroot' is in /proc/mounts
$ awk '$1 == "overlayroot" { print $0 }' /proc/mounts
overlayroot / overlay rw,relatime,
b.) check /run/initramfs/
$ grep success /run/initramfs/
[success]: configured root with 'tmpfs' using overlay per /dev/vda1/
6.) try with recurse disabled
Assuming you're on the same system and in an overlayroot, to change the
file necessary, we use overlayroot-chroot.
$ echo overlayroot=
$ sudo reboot
7.) log back in and look around.
This time the /mnt should not have overlay on it.
$ grep vdb /proc/mounts
/dev/vdb /mnt ext4 rw,relatime,
$ grep overlay /proc/mounts
overlayroot / overlay rw,relatime,
$ cat /etc/overlayroo
overlayroot
[Regression Potential]
The most likely regression is just in failure for overlayroot to work.
That was the case 100% of the time on any kernel without 'overlayfs'
filesystem, so this can't really make things worse from that perspective.
Some of the code change was related to fixing another issue, with 'recurse'.
Testing recurse (where not just / is mounted as an overlayroot) is done
above
echo overlayroot=
[Other Info]
The full overlayroot/
specific change that fixed the issue is in revision 115 at [2].
Related bugs:
* bug 1630274 adjusted the mechanism for determining if overlay was supported.
The change to do so is included in the xenial backport for this bug.
[1] http://
[2] http://
=== End SRU Template ===
As mentioned in LP: #1411294, it's now called 'overlay' instead of 'overlayfs'.
Ubuntu had patched the kernel for compatibility.
The Ubuntu kernels as of 4.8 (16.10 kernel) and possibly a bit before no longer have a overlayfs module either. Thus, this is now affecting yakkety.
(The original reporter is @~gpo-9.)
Related branches
- Paride Legovini (community): Approve on 2020-09-14
- Robert C Jennings (community): Approve on 2020-09-11
- Scott Moser (community): Approve on 2020-09-04
- Andreas Hasenack: Pending requested 2020-09-04
-
Diff: 404 lines (+376/-0)4 files modifieddebian/changelog (+11/-0)
debian/patches/lp-1493188-support-overlay-filesystem (+240/-0)
debian/patches/lp-1630274-mount-overlay-first (+123/-0)
debian/patches/series (+2/-0)
information type: | Public → Public Security |
information type: | Public Security → Public |
Oh, sorry, I don't mean to set it as a 'security' bug.
Thanks!
Changed in cloud-initramfs-tools (Ubuntu): | |
status: | New → Confirmed |
importance: | Undecided → Medium |
Scott Moser (smoser) wrote : | #3 |
I've marked cloud-utils as relevant here tto fix mount-image-
Changed in cloud-initramfs-tools: | |
status: | New → Confirmed |
Changed in cloud-utils: | |
status: | New → Confirmed |
Changed in cloud-utils (Ubuntu): | |
status: | New → Confirmed |
Changed in cloud-initramfs-tools: | |
importance: | Undecided → Medium |
Changed in cloud-utils: | |
importance: | Undecided → Medium |
Changed in cloud-utils (Ubuntu): | |
importance: | Undecided → Medium |
summary: |
- overlayroot doesn't work with vanilla kernel + "overlayfs" no longer exists |
description: | updated |
Changed in cloud-initramfs-tools: | |
assignee: | nobody → Scott Moser (smoser) |
status: | Confirmed → In Progress |
Scott Moser (smoser) wrote : | #4 |
for some specific information on different ubuntu kernels:
-- latest kernel before 4.8 entered yakkety --
$ uname -r
4.4.0-9136-generic
$ grep overlay /lib/modules/
alias overlayfs overlay
alias fs-overlayfs overlay
alias fs-overlay overlay
$ sudo modprobe -v overlayfs
insmod /lib/modules/
$ lsmod | grep overlay
overlay
$ grep overlay /proc/filesystems
nodev overlayfs
nodev overlay
-- 4.8 kernel in yakkety --
$ uname -r
4.8.0-15-generic
$ grep overlay /lib/modules/
alias fs-overlay overlay
$ modprobe -v overlayfs
modprobe: FATAL: Module overlayfs not found in directory /lib/modules/
$ modprobe -v overlay
insmod /lib/modules/
$ grep overlay /proc/filesystems
nodev overlay
-- 3.13 kernel (trusty GA) --
$ uname -r
3.13.0-96-generic
$ grep overlay /lib/modules/
alias fs-overlayfs overlayfs
$ sudo modprobe -v overlay
modprobe: FATAL: Module overlay not found.
$ sudo modprobe -v overlayfs
insmod /lib/modules/
$ grep overlay /proc/filesystems
nodev overlayfs
Changed in cloud-initramfs-tools (Ubuntu Xenial): | |
status: | New → Confirmed |
importance: | Undecided → Medium |
Changed in cloud-utils (Ubuntu Xenial): | |
status: | New → Confirmed |
importance: | Undecided → Medium |
Changed in cloud-initramfs-tools: | |
status: | In Progress → Fix Committed |
Hello 宋文武, or anyone else affected,
Accepted cloud-initramfs
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-
Further information regarding the verification process can be found at https:/
Changed in cloud-initramfs-tools (Ubuntu Yakkety): | |
status: | Confirmed → Fix Committed |
tags: | added: verification-needed |
Shuhao (shuhao) wrote : | #6 |
I can test this too if we build a copy for xenial for arm (raspi3).
Scott Moser (smoser) wrote : | #7 |
I've verified functional in yakkety in uvt-kvm.
Below 'ussh' is a wrapper arond uvt-kvm
https:/
for reader purposes, it can be assumed that it operates like 'lxc exec'. ie,
ussh name command arguments
executed command and arguments on 'name'.
# make a new system.
name=sm-y1
uvt-kvm create $name release=yakkety
# enable proposed
$ ussh $name sudo sh -c 'echo "deb http://
# install a package
ussh $name sudo apt-get install -qy overlayroot
# show.
$ ussh $name dpkg-query --show overlayroot
overlayroot 0.29ubuntu1
# enable overlayroot without recurse
$ ussh "$name" sh -c 'echo overlayroot=
# reboot
$ ussh $name reboot
# show
$ ussh $name grep overlay /proc/mounts
overlayroot / overlay rw,relatime,
# turn on recurse
$ ussh "$name" sh -c 'echo overlayroot=
# reboot
$ ussh $name reboot
# show /proc/mounts
$ ussh "$name" grep overlay /proc/mounts
overlayroot / overlay rw,relatime,
/media/root-ro/mnt /mnt overlay rw,relatime,
$ ussh $name grep overlay /etc/fstab
# This fstab is in an overlay. The real one can be found at
# sudo overlayroot-chroot
/media/root-ro/ / overlay lowerdir=
LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:
/media/root-ro/mnt /mnt overlay lowerdir=
# show kernel version
$ ussh $name uname -r
4.8.0-17-generic
# boot into normal root
$ ussh sm-y1 sudo sh -c 'overlayroot-chroot rm /etc/overlayroo
tags: |
added: verification-done removed: verification-needed |
Launchpad Janitor (janitor) wrote : | #8 |
This bug was fixed in the package cloud-initramfs
---------------
cloud-initramfs
* overlayroot: support 'overlay' filesystem explicitly rather than
relying on ubuntu specific kernel module 'overlayfs'. (LP: #1493188)
-- Scott Moser <email address hidden> Fri, 23 Sep 2016 17:00:37 -0400
Changed in cloud-initramfs-tools (Ubuntu Yakkety): | |
status: | Fix Committed → Fix Released |
Scott Moser (smoser) wrote : | #9 |
fix committed in cloud-utils trunk at revno 303.
Changed in cloud-utils: | |
status: | Confirmed → Fix Committed |
Launchpad Janitor (janitor) wrote : | #10 |
This bug was fixed in the package cloud-utils - 0.29-0ubuntu4
---------------
cloud-utils (0.29-0ubuntu4) yakkety; urgency=medium
* sync to trunk at revno 303
* mount-image-
than ubuntu specific 'overlayfs' (LP: #1493188)
-- Scott Moser <email address hidden> Tue, 27 Sep 2016 21:20:58 -0400
Changed in cloud-utils (Ubuntu): | |
status: | Confirmed → Fix Released |
Changed in cloud-initramfs-tools: | |
status: | Fix Committed → Fix Released |
Changed in cloud-initramfs-tools (Ubuntu Xenial): | |
status: | Confirmed → In Progress |
description: | updated |
Chris Halse Rogers (raof) wrote : | #11 |
Hello 宋文武, or anyone else affected,
Accepted cloud-initramfs
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-
Further information regarding the verification process can be found at https:/
Changed in cloud-initramfs-tools (Ubuntu Xenial): | |
status: | In Progress → Fix Committed |
tags: | removed: verification-done |
tags: | added: verification-needed |
Scott Moser (smoser) wrote : | #12 |
I've verified this on xenial using the steps provided in the test case above.
$ uname -r
4.8.0-28-generic
$ dpkg -S /boot/vmlinuz-
linux-image-
$ dpkg-query --show overlayroot
overlayroot 0.27ubuntu1.3
description: | updated |
tags: |
added: verification-done removed: verification-needed |
Launchpad Janitor (janitor) wrote : | #13 |
This bug was fixed in the package cloud-initramfs
---------------
cloud-initramfs
* sync with upstream at 0.32ubuntu1 (revno 129)
* overlayroot: fix overlayroot=crypt with newer initramfs-tools
(LP: #1634310)
* overlayroot: support random seed from systemd systemd-
* cloud-initramfs
IPV6 and the new DEVICE6 and net6-DEVICE.conf files. (LP: #1621615)
* overlayroot: support 'overlay' filesystem explicitly rather than
relying on ubuntu specific kernel module 'overlayfs'. (LP: #1493188)
* overlayroot: write debug to /run/initramfs not /dev/.initramfs but
support writing to /dev/.initramfs if that is all there is. (LP: #1485752)
* overlayroot: fix overlayroot if recurse=0 was not provided.
This fuctionality was lost since workdir support was added. (LP: #1619459)
* whitespace cleanup.
-- Scott Moser <email address hidden> Mon, 28 Nov 2016 20:12:51 -0500
Changed in cloud-initramfs-tools (Ubuntu Xenial): | |
status: | Fix Committed → Fix Released |
Robie Basak (racb) wrote : Update Released | #14 |
The verification of the Stable Release Update for cloud-initramfs
Scott Moser (smoser) wrote : | #15 |
this fix is present in cloud-utils 0.30
Changed in cloud-utils: | |
status: | Fix Committed → Fix Released |
Robert C Jennings (rcj) wrote : | #16 |
With the move to Bionic host kernels on the builders, the Xenial MAAS images fail to build. This is because cloud-utils mount-image-
https:/
Could we have this fix SRUed to Xenial please?
Chad Smith (chad.smith) wrote : | #17 |
Proposed branch is up for xenial handling overlayfs issues.
The review has been approved and a test ppa is setup with cloud-utils version 0.27-0ubuntu25.2 on Xenial which should resolve this issue.
If there is an opportunity to this this to confirm the fix, that would be great:
Verification procedure:
sudo add-apt-repository ppa:chad.
sudo apt-get update
sudo apt-get install cloud-utils=
Robert C Jennings (rcj) wrote : | #18 |
@chad.smith I have used the cloud-utils in your ppa to build Xenial maas-images on a bionic host to confirm that things are working now.
description: | updated |
Changed in cloud-initramfs-tools (Ubuntu): | |
status: | Fix Released → Fix Committed |
assignee: | nobody → Chad Smith (chad.smith) |
status: | Fix Committed → Fix Released |
assignee: | Chad Smith (chad.smith) → nobody |
Changed in cloud-utils (Ubuntu Xenial): | |
assignee: | nobody → Chad Smith (chad.smith) |
status: | Confirmed → Fix Committed |
Hello 宋文武, or anyone else affected,
Accepted cloud-utils into xenial-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
tags: |
added: verification-needed verification-needed-xenial removed: verification-done |
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (cloud-utils/0.27-0ubuntu25.2) | #20 |
All autopkgtests for the newly accepted cloud-utils (0.27-0ubuntu25.2) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:
lxc/2.0.
Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUp
[1] https:/
Thank you!
tags: |
added: verification-done verification-done-xenial removed: verification-needed verification-needed-xenial |
Chad Smith (chad.smith) wrote : | #22 |
While the test case run on xenial did show overlay properly mounted across reboot. The generated deb doesn't contain the applied patches to mount-image-
tags: |
added: verification-needed verification-needed-xenial removed: verification-done verification-done-xenial |
Chad Smith (chad.smith) wrote : | #23 |
Extended the SRU verification test case to validate that expected
changes are present in mount-image-
I can confirm both that proper bits are uploaded in any overlay fs is functional using cloud-utils (0.27-0ubuntu25.2).
#### Updated test script
#!/bin/bash
set -x
cat > install_
#!/bin/bash
echo "3.) Enable proposed, install HWE kernel and install overlayroot to show fix."
mirror=http://
echo deb \$mirror xenial-proposed main | tee /etc/apt/
apt-get update
apt-get install -qy cloud-utils overlayroot linux-generic-
EOF
expect_
expected_
overlay_
[[ "$expected_mounts" = "$overlay_mounts" ]] && echo "SUCCESS: found $overlay_mounts overlay mounts" || echo "FAILURE: expected $expected_mounts overlay mounts found $overlay_mounts"
}
VM=sru-1493188
RELEASE=xenial
multipass launch daily:$RELEASE -n $VM
multipass exec $VM -- cloud-init status --wait --long > /dev/null
echo --- No overlayfs mounts on initial boot
expect_
multipass transfer install_
multipass exec $VM -- sudo bash ./install_
echo "--- confirm overlay overlayfs patch present in mount-image-
fs_types=
[[ "1" = "$fs_types" ]] || echo "FAILURE: didn't find 'for fstype in overlay overlayfs'"
multipass exec $VM -- sudo sh -c "echo overlayroot=
multipass restart $VM
echo "--- expect 1 overlayfs mounts because using HWE kernel"
multipass exec $VM -- cloud-init status --wait --long
expect_
multipass exec $VM -- sudo sh -c "echo overlayroot=
multipass restart $VM
expect_
multipass exec $VM -- grep overlay /etc/fstab || echo "FAIL: didn't find overlayroot"
multipass exec $VM -- grep success /run/initramfs/
### Xenial verification results
+ cat
+ VM=sru-1493188
+ RELEASE=xenial
+ multipass launch daily:xenial -n sru-1493188
Launched: sru-1493188
+ multipass exec sru-1493188 -- cloud-init status --wait --long
+ echo --- No overlayfs mounts on initial boot
--- No overlayfs mounts on initial boot
+ expect_
+ expected_mounts=0
++ multipass exec sru-1493188 mount
++ grep overlay -c
+ overlay_mounts=0
+ [[ 0 = \0 ]]
+ echo 'SUCCESS: found 0 overlay mounts'
SUCCESS: found 0 overlay mounts
+ multipass transfer install_
+ multipass exec sru-1493188 -- sudo bash ./install_
3.) Enable proposed, install HWE kernel and install overlayroot to show fix.
d...
tags: |
added: verification-done verification-done-xenial removed: verification-needed verification-needed-xenial |
Launchpad Janitor (janitor) wrote : | #24 |
This bug was fixed in the package cloud-utils - 0.27-0ubuntu25.2
---------------
cloud-utils (0.27-0ubuntu25.2) xenial; urgency=medium
* debian/
mount-
than ubuntu specific 'overlayfs' (LP: #1493188)
* debian/
mount-
(LP: #1630274)
-- Chad Smith <email address hidden> Fri, 04 Sep 2020 08:47:08 -0600
Changed in cloud-utils (Ubuntu Xenial): | |
status: | Fix Committed → Fix Released |
Thanks for taking the time to report this bug and helping to make Ubuntu better. We appreciate the difficulties you are facing, but this appears to be a "regular" (non-security) bug. I have unmarked it as a security issue since this bug does not show evidence of allowing attackers to cross privilege boundaries nor directly cause loss of data/privacy. Please feel free to report any other bugs you may find.