using ecryptfs, creating frameworks fail to bind mount issues

Bug #1427264 reported by Sebastien Bacher
134
This bug affects 25 people
Affects Status Importance Assigned to Milestone
schroot (Debian)
Fix Released
Unknown
schroot (Ubuntu)
Fix Released
High
Tyler Hicks
Trusty
Won't Fix
Undecided
Unassigned
Vivid
Won't Fix
Undecided
Unassigned

Bug Description

Using vivid creating framework fails for ecryptfs users, the issue is similar to bug #769595

The userdir is mounted in a way which makes unmounts fail

"E: 10mount: umount: /var/lib/schroot/mount/click-ubuntu-sdk-14.10-armhf-ec6aaf62-31e0-47e9-b2f8-73f0b038fb4d/home/user: target is busy E: 10mount: (In some cases useful info about processes that E: 10mount: use the device is found by lsof(8) or fuser(1).) "

changing the fstab line to be "/home/user /home/user none rw,bind 0 0" workarounds the issue

Revision history for this message
Sebastien Bacher (seb128) wrote :
Changed in click (Ubuntu):
importance: Undecided → High
Revision history for this message
Sebastien Bacher (seb128) wrote :

the fstab to change is /etc/schroot/click/fstab

Revision history for this message
Tyler Hicks (tyhicks) wrote :

This issue is not specific to eCryptfs. It is specific to a user's home directory being a mount point. For example, I can reproduce this issue with ext4:

foo@sec-vivid-amd64:~$ grep home\/foo /proc/mounts
/dev/loop0 /home/foo ext4 rw,relatime,data=ordered 0 0
foo@sec-vivid-amd64:~$ schroot -c click-amd64 -- true
E: 10mount: umount: /var/lib/schroot/mount/click-amd64-63d79a95-7c79-4f59-97ea-c9089e9c0c9f/home/foo: target is busy
E: 10mount: (In some cases useful info about processes that
E: click-amd64-63d79a95-7c79-4f59-97ea-c9089e9c0c9f: Chroot setup failed: stage=setup-stop
foo@sec-vivid-amd64:~$ grep home\/foo /proc/mounts
/dev/loop0 /home/foo ext4 rw,relatime,data=ordered 0 0
/dev/loop0 /var/lib/schroot/mount/click-amd64-63d79a95-7c79-4f59-97ea-c9089e9c0c9f/home/foo ext4 rw,relatime,data=ordered 0 0

Revision history for this message
Tyler Hicks (tyhicks) wrote :

This bug seems to be caused by the way recursive bind mounting and unmounting works:

foo@sec-vivid-amd64:~$ grep home\/foo /proc/mounts
/dev/loop0 /home/foo ext4 rw,relatime,data=ordered 0 0
foo@sec-vivid-amd64:~$ sudo mount -R /home /tmp/home
foo@sec-vivid-amd64:~$ grep home\/foo /proc/mounts
/dev/loop0 /home/foo ext4 rw,relatime,data=ordered 0 0
/dev/loop0 /tmp/home/foo ext4 rw,relatime,data=ordered 0 0
foo@sec-vivid-amd64:~$ sudo umount -R /tmp/home
umount: /tmp/home/foo: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)
foo@sec-vivid-amd64:~$ grep home\/foo /proc/mounts
/dev/loop0 /home/foo ext4 rw,relatime,data=ordered 0 0
/dev/loop0 /tmp/home/foo ext4 rw,relatime,data=ordered 0 0

Revision history for this message
Tyler Hicks (tyhicks) wrote :

The issue is around mount propagation. The /home/ directory is being recursively bound into the schroot tree. When the schroot tree is recursively unmounted and SCHROOT_DIR/home/foo is unmounted, that unmount event is being propagated to the original /home/foo mount point. To prevent that from happening, the recursively bound SCHROOT_DIR/home mount point needs to be recursively private. Using the example in comment #4, with eCryptfs instead of ext4, that would look like this:

foo@sec-vivid-amd64:~$ grep home/foo /proc/mounts
/home/foo/.Private /home/foo ecryptfs rw,nosuid,nodev,relatime,ecryptfs_fnek_sig=176339c8321afc83,ecryptfs_sig=35e81f7d08221a22,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_unlink_sigs 0 0
foo@sec-vivid-amd64:~$ sudo mount -R /home /tmp/home
foo@sec-vivid-amd64:~$ sudo mount --make-rslave /tmp/home
foo@sec-vivid-amd64:~$ grep home/foo /proc/mounts
/home/foo/.Private /home/foo ecryptfs rw,nosuid,nodev,relatime,ecryptfs_fnek_sig=176339c8321afc83,ecryptfs_sig=35e81f7d08221a22,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_unlink_sigs 0 0
/home/foo/.Private /tmp/home/foo ecryptfs rw,nosuid,nodev,relatime,ecryptfs_fnek_sig=176339c8321afc83,ecryptfs_sig=35e81f7d08221a22,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_unlink_sigs 0 0
foo@sec-vivid-amd64:~$ sudo umount -R /tmp/home
foo@sec-vivid-amd64:~$ grep home/foo /proc/mounts
/home/foo/.Private /home/foo ecryptfs rw,nosuid,nodev,relatime,ecryptfs_fnek_sig=176339c8321afc83,ecryptfs_sig=35e81f7d08221a22,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_unlink_sigs 0 0

Note that two separate mount commands are required. The -R and --make-rslave options cannot be specified in the same mount command. The mount(8) program will turn this into two mount(2) syscalls and strip the MS_REC flag off of the first mount(2) syscall, resulting in a non-recursive bind mount:

mount("/home", "/tmp/home", 0x1880f30, MS_MGC_VAL|MS_BIND, NULL) = 0
mount("none", "/tmp/home", NULL, MS_REC|MS_SLAVE, NULL) = 0

This means that we can add a new line to the click/fstab file, specifying that the schroot's /home/ recursive bind mount point should be made recursively private. While I haven't had a chance to do extensive testing, the following change seems to fix this bug:

$ diff -Nurp /etc/schroot/click/fstab.orig /etc/schroot/click/fstab
--- /etc/schroot/click/fstab.orig 2015-03-12 15:23:37.631296891 -0500
+++ /etc/schroot/click/fstab 2015-03-12 15:24:09.183296891 -0500
@@ -8,5 +8,6 @@
 /dev /dev none rw,bind 0 0
 /dev/pts /dev/pts none rw,bind 0 0
 /home /home none rw,rbind 0 0
+none /home none rslave 0 0
 /tmp /tmp none rw,bind 0 0
 /run/shm /run/shm none rw,bind 0 0

Changed in click (Ubuntu):
status: New → Triaged
Revision history for this message
Tyler Hicks (tyhicks) wrote :

Serge - I've subscribed you to this bug in hopes that you could tell us if you expect any problems with making the change suggested at the bottom of comment #5 since I know you have a lot of experience in mount namespaces.

To summarize the problem, schroot doesn't handle /home/$USER being a mount point very well. If /home is recursively bound (rbind) into the schroot, then the recursive umount unmounts the /home/$USER outside of the schroot when tearing down the schroot session. I think recursively making /home a slave (rslave) fixes the problems but don't feel comfortable rolling out that change without getting another opinion.

Revision history for this message
Tyler Hicks (tyhicks) wrote :

I just noticed a mistake that I made a couple times in comment #5. I said that the /home/ mount point inside the schroot needs to be made recursively private (rprivate). However, what I meant to say is that the /home/ mount point inside the schroot needs to recursively be made a slave (rslave). The diff at the bottom of the comment is correct.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Hi Tyler,

yes since 15.04 switches to systemd which mounts everything as MS_SHARED by default, I expect a few more such bugs. What you recommend sounds right.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Note you use slave and private somewhat interchangeably above. Which you want to use will depend on the behavior you want to see if the uesr tries to umount something under /home. If there might be devices or remotes which need to be freed, then you'll wan to use slave so that the chroot can't pin the source. If you actually want the mount to stick around in the chroot then you'll want to use private.

Revision history for this message
Michał Sawicz (saviq) wrote :

Seems related to this, after using schroot with overlay on /dev/shm, I end up with multiple /dev/shm mounts on top of one another.

I'll try and collect more data and will comment back here or file a new bug.

Revision history for this message
Michał Sawicz (saviq) wrote :

Hmm I take that back, a reboot solved my issues...

Revision history for this message
Scott Moser (smoser) wrote :

I've recreated this in a cloud image.
See the steps to do so here in this attachment.

Scott Moser (smoser)
Changed in schroot (Ubuntu):
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Tyler Hicks (tyhicks) wrote :

Hi Scott - The bug that you're experiencing is different than this bug. I've opened bug #1438942 to track your issue. Please confirm what I placed in the description and set the bug to confirmed. Thanks!

Revision history for this message
Kyle Fazzari (kyrofa) wrote :

Tyler, I got bit by this issue this morning (vivid). The rbind addition outlined in #5 seems to have resolved it for me as well.

Revision history for this message
Cris Dywan (kalikiana) wrote :

Adding 'none /home none rslave 0 0' to /etc/schroot/click/fstab allowed me to successfully create a new chroot.

Revision history for this message
Timo Jyrinki (timo-jyrinki) wrote :

Confirming the workaround in comment #15 works.

Revision history for this message
Kyle Fazzari (kyrofa) wrote :

Does this also bite users who simply have their /home on a separate partition? Disregarding encryption?

Revision history for this message
Michał Sawicz (saviq) wrote : Re: [Bug 1427264] Re: using ecryptfs, creating frameworks fail to bind mount issues

W dniu 13.05.2015 o 14:11, Kyle Fazzari pisze:
> Does this also bite users who simply have their /home on a separate
> partition? Disregarding encryption?

I'm on full-disk-encrypted btrfs with a separate @home and this works
fine AFAICT. Maybe it's mounting /home/$USER itself that's the issue?

Revision history for this message
Tyler Hicks (tyhicks) wrote :

On 2015-05-13 12:49:45, Michał Sawicz wrote:
> W dniu 13.05.2015 o 14:11, Kyle Fazzari pisze:
> > Does this also bite users who simply have their /home on a separate
> > partition? Disregarding encryption?
>
> I'm on full-disk-encrypted btrfs with a separate @home and this works
> fine AFAICT. Maybe it's mounting /home/$USER itself that's the issue?

Correct. Any filesystem mounted at /home/$USER will hit this bug (not
just eCryptfs).

Tyler Hicks (tyhicks)
Changed in schroot (Ubuntu):
assignee: nobody → Tyler Hicks (tyhicks)
status: Confirmed → In Progress
Revision history for this message
Mitchell (curious-mitchell) wrote :

Wow - this experience has been painful. Have just spent an hour or so debugging my machine to work out why it froze each time I logged into my main user account. No terminals worked, nothing, nada. Finally logged into another user (not with admin rights), used the password from my main account to give it admin rights, then needed to DELETE the line I added from comment #15 above to be able to log back in as my main user again. This is nasty!!!

Looking forward to seeing a workable solution soon.

Revision history for this message
Didier Roche-Tolomelli (didrocks) wrote :

Confirming that I got the same mount loop issue than Mitchell with the line #15, (thousands of mounts). The worst is that as the sessions are not cleaned up on a forced shutdown, the next login after reboot will restart the schroots, and so the mount hell.

Has to do that either with another admin account or with a live…

Revision history for this message
Tyler Hicks (tyhicks) wrote :

I've submitted a proposed fix for this bug to schroot upstream/Debian and linked the bug to this one.

Changed in schroot (Debian):
status: Unknown → Confirmed
Revision history for this message
Cris Dywan (kalikiana) wrote :

Once again wasted an hour for nothing downloading tons of packages, as click very happily downloads the whole package set only to delete it and fail afterall:

Processing triggers for ca-certificates (20141019ubuntu0.15.04.1) ...
Updating certificates in /etc/ssl/certs...
173 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d....
done.
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
E: 10mount: umount: /var/lib/schroot/mount/click-ubuntu-sdk-15.04-armhf-78d6d7ed-d487-40f3-827e-3cfe9cba6089/home/maple: target is busy E: 10mount: (In some cases useful info about processes that E: 10mount: use the device is found by lsof(8) or fuser(1).)
E: click-ubuntu-sdk-15.04-armhf-78d6d7ed-d487-40f3-827e-3cfe9cba6089: Chroot setup failed: stage=setup-stop
umount: /var/lib/schroot/mount/click-ubuntu-sdk-15.04-armhf-78d6d7ed-d487-40f3-827e-3cfe9cba6089/home/maple: target is busy (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1).)
[...]
Unmount failed... trying to force Unmounting /var/lib/schroot/mount/click-ubuntu-sdk-15.04-armhf-78d6d7ed-d487-40f3-827e-3cfe9cba6089/home/maple failed Deleting /etc/schroot/chroot.d/click-ubuntu-sdk-15.04-armhf Deleting /var/lib/schroot/chroots/click-ubuntu-sdk-15.04-armhf click target was removed successfully
umount: /var/lib/schroot/mount/click-ubuntu-sdk-15.04-armhf-78d6d7ed-d487-40f3-827e-3cfe9cba6089/home/maple: mountpoint not found umount: /var/lib/schroot/mount/click-ubuntu-sdk-15.04-armhf-78d6d7ed-d487-40f3-827e-3cfe9cba6089/home/maple: mountpoint not found
---Click exited with errors, please check the output---

Revision history for this message
Dmitry Ershov (edvice) wrote :

I have same error on Ubuntu 14.04 with ecryptfs. Change or comment /home in /etc/schroot/click/fstab does not take effect.
Switching from kernel 3.19.0-29-generic to 3.16.0-50-generic or 3.13.0-64-generic solved the problem.

Revision history for this message
Dmitry Ershov (edvice) wrote :

For me this bug fixed in schroot package version 1.6.8-1ubuntu1.1 by https://bugs.launchpad.net/ubuntu/+source/schroot/+bug/1398523.

Can anyone else confirm that bug fixed?

tags: added: verification-needed
Revision history for this message
Nicolas DERIVE (kalon33) wrote :

@edvice: If you look at duplicates, which are recent, you will see that this problem is not fixed, at least on Wily.

Revision history for this message
Nathan Osman (george-edison55) wrote :

I am still seeing this on Wily.

    $ dpkg -s schroot | grep Version
    Version: 1.6.10-1ubuntu2

Revision history for this message
António Lima (amrlima) wrote :

Not fixed on wily.

Revision history for this message
Felipe De la Puente (fdelapuente) wrote :

Hi,

I have /home and /home/${USER}/Data on different partitions which are mounted based on /etc/fstab specs.
When I had automatic updates of my kits in ubuntu sdk, every time it attempted to update it umounted my two home related partitions. This problem was solved by de-activating auto update. Instead I update my kits manually with apt-get.

Now I'm trying to create a new kit for amd64 and have the same problem described here. I'm running 15.04 up-to-date.

So:

1. Please confirm if the current WA would be to do #15 before attempting to modify kits, then once finished revert the WA to avoid #20.

2. Any progress on the fix for this showstopper? Any target release?

Thanks a lot!

Changed in schroot (Debian):
status: Confirmed → Fix Released
Mathew Hodson (mhodson)
no longer affects: click (Ubuntu)
Revision history for this message
Mathew Hodson (mhodson) wrote :

This was fixed in package schroot 1.6.10-3 available in Ubuntu Artful and later.

Changed in schroot (Ubuntu Trusty):
status: New → Won't Fix
Changed in schroot (Ubuntu Vivid):
status: New → Won't Fix
Changed in schroot (Ubuntu):
status: In Progress → Fix Released
tags: removed: verification-needed
Revision history for this message
Mathew Hodson (mhodson) wrote :

schroot (1.6.10-3) unstable; urgency=medium

  * By default mark all mounts done by schroot-mount as "private"
    to avoid bad interactions caused by systemd's default of "shared"
    that resulted in failure to unmount them.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.