mount-image-callback may not mount overlay when it could
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| cloud-utils |
High
|
Unassigned | ||
| cloud-utils (Ubuntu) |
Critical
|
Scott Moser | ||
| Xenial |
Undecided
|
Unassigned |
Bug Description
=== Begin SRU Template ===
[Impact]
The 16.10 kernel dropped a legacy kernel module alias that allowed usage of
the 'overlay' filesystem via name 'overlayfs'. This broke overlayroot as
it explicitly tried to to use 'overlayfs' by name in loading of modules and
also in entry in /etc/fstab.
Without this fix, overlayroot will simply not work on any upstream kernel
or Ubuntu kernel of 16.10 (yakkety) or later.
[Test Case]
Note, not applying proposed as shown in step 3 below will recreate failure.
1.) Start an instance of a cloud image.
2.) get a suitable 4.8 kernel
On 16.10 or later, this is already done. On 16.04, we currently need to
install the kernel team's PPA to get one.
$ sudo apt-add-repository -y ppa:canonical-
$ sudo apt update -q && sudo apt install -y linux-virtual-
3.) Enable proposed and install overlayroot to show fix.
$ rel=$(lsb_release -sc)
$ echo "deb http://
$ sudo tee /etc/apt/
$ sudo apt update -qy && sudo apt install -qy overlayroot </dev/null
$ dpkg-query --show overlayroot
overlayroot 0.27ubuntu1.3
4.) Enable overlayroot and reboot
# remove the cloud-init written mount options for /dev/vdb
# if we do not do this, then /mnt ends up not mounted due to ordering.
$ sudo sed -i.dist s/,x-systemd.
$ echo "overlayroot=tmpfs" | sudo tee /etc/overlayroo
$ sudo reboot
5.) log back in and look around.
a.) check that 'overlayroot' is in /proc/mounts
$ awk '$1 == "overlayroot" { print $0 }' /proc/mounts
overlayroot / overlay rw,relatime,
b.) check /run/initramfs/
$ grep success /run/initramfs/
[success]: configured root with 'tmpfs' using overlay per /dev/vda1/
6.) try with recurse disabled
Assuming you're on the same system and in an overlayroot, to change the
file necessary, we use overlayroot-chroot.
$ echo overlayroot=
$ sudo reboot
7.) log back in and look around.
This time the /mnt should not have overlay on it.
$ grep vdb /proc/mounts
/dev/vdb /mnt ext4 rw,relatime,
$ grep overlay /proc/mounts
overlayroot / overlay rw,relatime,
$ cat /etc/overlayroo
overlayroot
[Regression Potential]
The most likely regression is just in failure for overlayroot to work.
That was the case 100% of the time on any kernel without 'overlayfs'
filesystem, so this can't really make things worse from that perspective.
Some of the code change was related to fixing another issue, with 'recurse'.
Testing recurse (where not just / is mounted as an overlayroot) is done
above
echo overlayroot=
[Other Info]
The full overlayroot/
specific change that fixed the issue is in revision 115 at [2].
Related bugs:
* bug 1630274 adjusted the mechanism for determining if overlay was supported.
The change to do so is included in the xenial backport for this bug.
[1] http://
[2] http://
=== End SRU Template ===
=== Original Description ===
As mentioned in LP: #1411294, it's now called 'overlay' instead of 'overlayfs'.
Ubuntu had patched the kernel for compatibility.
The Ubuntu kernels as of 4.8 (16.10 kernel) and possibly a bit before no longer have a overlayfs module either. Thus, this is now affecting yakkety.
(The original reporter is @~gpo-9.)
The changes recently added to mount-image-
The changes basically said:
if overlay in /proc/filesystems:
return GOOD
modprobe --quiet --use-blacklist overlay
if overlay in /proc/filesystems:
return GOOD
return BAD
the path that fails now that previously succeeded is when 'modprobe' would not work, but 'mount -t overlay' *would*. This occurs in build systems where the running kernel is of one version and code is running in a chroot that does not have access to the kernel modules in /lib/modules/.
'mount -t overlay' works in this scenario because the mount abi call passes the filesystem type 'overlay', and the kernel then does:
fs/filesyste
The kernel's scope is not chrooted, so the module gets loaded and everything is happy.
The new changes are to instead basically do:
for fstype in overlay overlayfs; do
mount -t $fstype <options> && return GOOD
if /proc/filesystems is readable and $fstype in /proc/filesystems; then
return FAIL
fi
done
Related branches
- Registry Administrators: Pending requested 2016-10-04
-
Diff: 107 lines (+37/-34)1 file modifiedbin/mount-image-callback (+37/-34)
- Paride Legovini (community): Approve on 2020-09-14
- Robert C Jennings (community): Approve on 2020-09-11
- Scott Moser (community): Approve on 2020-09-04
- Andreas Hasenack: Pending requested 2020-09-04
-
Diff: 404 lines (+376/-0)4 files modifieddebian/changelog (+11/-0)
debian/patches/lp-1493188-support-overlay-filesystem (+240/-0)
debian/patches/lp-1630274-mount-overlay-first (+123/-0)
debian/patches/series (+2/-0)
Changed in cloud-utils (Ubuntu): | |
assignee: | nobody → Scott Moser (smoser) |
importance: | Undecided → High |
status: | New → Confirmed |
Changed in cloud-utils: | |
status: | New → Confirmed |
importance: | Undecided → High |
status: | Confirmed → In Progress |
Changed in cloud-utils (Ubuntu): | |
status: | Confirmed → In Progress |
Changed in cloud-utils: | |
status: | In Progress → Fix Committed |
description: | updated |
Launchpad Janitor (janitor) wrote : | #2 |
This bug was fixed in the package cloud-utils - 0.29-0ubuntu5
---------------
cloud-utils (0.29-0ubuntu5) yakkety; urgency=medium
* debian/control: update to Standards-Version 3.9.8
* debian/copyright: fix lintian file-without-
* sync to trunk at revno 306
* mount-image-
(LP: #1630274)
-- Scott Moser <email address hidden> Tue, 04 Oct 2016 12:36:21 -0400
Changed in cloud-utils (Ubuntu): | |
status: | In Progress → Fix Released |
Changed in cloud-utils: | |
status: | Fix Committed → Fix Released |
Changed in cloud-utils (Ubuntu Xenial): | |
status: | New → In Progress |
description: | updated |
Hello Scott, or anyone else affected,
Accepted cloud-utils into xenial-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
Changed in cloud-utils (Ubuntu Xenial): | |
status: | In Progress → Fix Committed |
tags: | added: verification-needed verification-needed-xenial |
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (cloud-utils/0.27-0ubuntu25.2) | #4 |
All autopkgtests for the newly accepted cloud-utils (0.27-0ubuntu25.2) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:
lxc/2.0.
Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUp
[1] https:/
Thank you!
tags: |
added: verification-complete verification-complete-xenial removed: verification-needed verification-needed-xenial |
Chad Smith (chad.smith) wrote : | #6 |
While the test case run on xenial did show overlay properly mounted across reboot. The generated deb doesn't contain the applied patches to mount-image-
tags: |
added: verification-needed verification-needed-xenial removed: verification-complete verification-complete-xenial |
Chad Smith (chad.smith) wrote : | #7 |
I extended the SRU verification test case to validate that expected
changes are present in mount-image-
I can confirm both that proper bits are uploaded and "overlay" fs is functional using cloud-utils (0.27-0ubuntu25.2).
#### Updated test script
#!/bin/bash
set -x
cat > install_
#!/bin/bash
echo "3.) Enable proposed, install HWE kernel and install overlayroot to show fix."
mirror=http://
echo deb \$mirror xenial-proposed main | tee /etc/apt/
apt-get update
apt-get install -qy cloud-utils overlayroot linux-generic-
EOF
expect_
expected_
overlay_
[[ "$expected_mounts" = "$overlay_mounts" ]] && echo "SUCCESS: found $overlay_mounts overlay mounts" || echo "FAILURE: expected $expected_mounts overlay mounts found $overlay_mounts"
}
VM=sru-1493188
RELEASE=xenial
multipass launch daily:$RELEASE -n $VM
multipass exec $VM -- cloud-init status --wait --long > /dev/null
echo --- No overlayfs mounts on initial boot
expect_
multipass transfer install_
multipass exec $VM -- sudo bash ./install_
echo "--- confirm overlay overlayfs patch present in mount-image-
fs_types=
[[ "1" = "$fs_types" ]] || echo "FAILURE: didn't find 'for fstype in overlay overlayfs'"
multipass exec $VM -- sudo sh -c "echo overlayroot=
multipass restart $VM
echo "--- expect 1 overlayfs mounts because using HWE kernel"
multipass exec $VM -- cloud-init status --wait --long
expect_
multipass exec $VM -- sudo sh -c "echo overlayroot=
multipass restart $VM
expect_
multipass exec $VM -- grep overlay /etc/fstab || echo "FAIL: didn't find overlayroot"
multipass exec $VM -- grep success /run/initramfs/
### Xenial verification results
+ cat
+ VM=sru-1493188
+ RELEASE=xenial
+ multipass launch daily:xenial -n sru-1493188
Launched: sru-1493188
+ multipass exec sru-1493188 -- cloud-init status --wait --long
+ echo --- No overlayfs mounts on initial boot
--- No overlayfs mounts on initial boot
+ expect_
+ expected_mounts=0
++ multipass exec sru-1493188 mount
++ grep overlay -c
+ overlay_mounts=0
+ [[ 0 = \0 ]]
+ echo 'SUCCESS: found 0 overlay mounts'
SUCCESS: found 0 overlay mounts
+ multipass transfer install_
+ multipass exec sru-1493188 -- sudo bash ./install_
3.) Enable proposed, install HWE kernel and install overlayroot to show fix.
...
tags: |
added: verification-complete verification-complete-xenial removed: verification-needed verification-needed-xenial |
Joshua Powers (powersj) wrote : | #8 |
Updating the tags so this turns green on the SRU board.
Was `verificiaton-
tags: |
added: verification-done verification-done-xenial removed: verification-complete verification-complete-xenial |
Launchpad Janitor (janitor) wrote : | #10 |
This bug was fixed in the package cloud-utils - 0.27-0ubuntu25.2
---------------
cloud-utils (0.27-0ubuntu25.2) xenial; urgency=medium
* debian/
mount-
than ubuntu specific 'overlayfs' (LP: #1493188)
* debian/
mount-
(LP: #1630274)
-- Chad Smith <email address hidden> Fri, 04 Sep 2020 08:47:08 -0600
Changed in cloud-utils (Ubuntu Xenial): | |
status: | Fix Committed → Fix Released |
The verification of the Stable Release Update for cloud-utils has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.
marked this as critical, as it is blocking maas image builds for 16.10.