using ecryptfs, creating frameworks fail to bind mount issues

[Sebastien Bacher]

Using vivid creating framework fails for ecryptfs users, the issue is similar to bug #769595

The userdir is mounted in a way which makes unmounts fail

"E: 10mount: umount: /var/lib/schroot/mount/click-ubuntu-sdk-14.10-armhf-ec6aaf62-31e0-47e9-b2f8-73f0b038fb4d/home/user: target is busy E: 10mount: (In some cases useful info about processes that E: 10mount: use the device is found by lsof(8) or fuser(1).) "

changing the fstab line to be "/home/user /home/user none rw,bind 0 0" workarounds the issue

[Sebastien Bacher (seb128)]

https://code.launchpad.net/~mvo/click/lp1319790-chroot-fstab might address the issue?

[Sebastien Bacher (seb128)]

the fstab to change is /etc/schroot/click/fstab

[Tyler Hicks (tyhicks)]

This issue is not specific to eCryptfs. It is specific to a user's home directory being a mount point. For example, I can reproduce this issue with ext4:

foo@sec-vivid-amd64:~$ grep home\/foo /proc/mounts
/dev/loop0 /home/foo ext4 rw,relatime,data=ordered 0 0
foo@sec-vivid-amd64:~$ schroot -c click-amd64 -- true
E: 10mount: umount: /var/lib/schroot/mount/click-amd64-63d79a95-7c79-4f59-97ea-c9089e9c0c9f/home/foo: target is busy
E: 10mount: (In some cases useful info about processes that
E: click-amd64-63d79a95-7c79-4f59-97ea-c9089e9c0c9f: Chroot setup failed: stage=setup-stop
foo@sec-vivid-amd64:~$ grep home\/foo /proc/mounts
/dev/loop0 /home/foo ext4 rw,relatime,data=ordered 0 0
/dev/loop0 /var/lib/schroot/mount/click-amd64-63d79a95-7c79-4f59-97ea-c9089e9c0c9f/home/foo ext4 rw,relatime,data=ordered 0 0

[Tyler Hicks (tyhicks)]

This bug seems to be caused by the way recursive bind mounting and unmounting works:

foo@sec-vivid-amd64:~$ grep home\/foo /proc/mounts
/dev/loop0 /home/foo ext4 rw,relatime,data=ordered 0 0
foo@sec-vivid-amd64:~$ sudo mount -R /home /tmp/home
foo@sec-vivid-amd64:~$ grep home\/foo /proc/mounts
/dev/loop0 /home/foo ext4 rw,relatime,data=ordered 0 0
/dev/loop0 /tmp/home/foo ext4 rw,relatime,data=ordered 0 0
foo@sec-vivid-amd64:~$ sudo umount -R /tmp/home
umount: /tmp/home/foo: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)
foo@sec-vivid-amd64:~$ grep home\/foo /proc/mounts
/dev/loop0 /home/foo ext4 rw,relatime,data=ordered 0 0
/dev/loop0 /tmp/home/foo ext4 rw,relatime,data=ordered 0 0

[Tyler Hicks (tyhicks)]

The issue is around mount propagation. The /home/ directory is being recursively bound into the schroot tree. When the schroot tree is recursively unmounted and SCHROOT_DIR/home/foo is unmounted, that unmount event is being propagated to the original /home/foo mount point. To prevent that from happening, the recursively bound SCHROOT_DIR/home mount point needs to be recursively private. Using the example in comment #4, with eCryptfs instead of ext4, that would look like this:

foo@sec-vivid-amd64:~$ grep home/foo /proc/mounts
/home/foo/.Private /home/foo ecryptfs rw,nosuid,nodev,relatime,ecryptfs_fnek_sig=176339c8321afc83,ecryptfs_sig=35e81f7d08221a22,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_unlink_sigs 0 0
foo@sec-vivid-amd64:~$ sudo mount -R /home /tmp/home
foo@sec-vivid-amd64:~$ sudo mount --make-rslave /tmp/home
foo@sec-vivid-amd64:~$ grep home/foo /proc/mounts
/home/foo/.Private /home/foo ecryptfs rw,nosuid,nodev,relatime,ecryptfs_fnek_sig=176339c8321afc83,ecryptfs_sig=35e81f7d08221a22,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_unlink_sigs 0 0
/home/foo/.Private /tmp/home/foo ecryptfs rw,nosuid,nodev,relatime,ecryptfs_fnek_sig=176339c8321afc83,ecryptfs_sig=35e81f7d08221a22,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_unlink_sigs 0 0
foo@sec-vivid-amd64:~$ sudo umount -R /tmp/home
foo@sec-vivid-amd64:~$ grep home/foo /proc/mounts
/home/foo/.Private /home/foo ecryptfs rw,nosuid,nodev,relatime,ecryptfs_fnek_sig=176339c8321afc83,ecryptfs_sig=35e81f7d08221a22,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_unlink_sigs 0 0

Note that two separate mount commands are required. The -R and --make-rslave options cannot be specified in the same mount command. The mount(8) program will turn this into two mount(2) syscalls and strip the MS_REC flag off of the first mount(2) syscall, resulting in a non-recursive bind mount:

mount("/home", "/tmp/home", 0x1880f30, MS_MGC_VAL|MS_BIND, NULL) = 0
mount("none", "/tmp/home", NULL, MS_REC|MS_SLAVE, NULL) = 0

This means that we can add a new line to the click/fstab file, specifying that the schroot's /home/ recursive bind mount point should be made recursively private. While I haven't had a chance to do extensive testing, the following change seems to fix this bug:

$ diff -Nurp /etc/schroot/click/fstab.orig /etc/schroot/click/fstab
--- /etc/schroot/click/fstab.orig 2015-03-12 15:23:37.631296891 -0500
+++ /etc/schroot/click/fstab 2015-03-12 15:24:09.183296891 -0500
@@ -8,5 +8,6 @@
 /dev /dev none rw,bind 0 0
 /dev/pts /dev/pts none rw,bind 0 0
 /home /home none rw,rbind 0 0
+none /home none rslave 0 0
 /tmp /tmp none rw,bind 0 0
 /run/shm /run/shm none rw,bind 0 0

[Tyler Hicks (tyhicks)]

Serge - I've subscribed you to this bug in hopes that you could tell us if you expect any problems with making the change suggested at the bottom of comment #5 since I know you have a lot of experience in mount namespaces.

To summarize the problem, schroot doesn't handle /home/$USER being a mount point very well. If /home is recursively bound (rbind) into the schroot, then the recursive umount unmounts the /home/$USER outside of the schroot when tearing down the schroot session. I think recursively making /home a slave (rslave) fixes the problems but don't feel comfortable rolling out that change without getting another opinion.

[Tyler Hicks (tyhicks)]

I just noticed a mistake that I made a couple times in comment #5. I said that the /home/ mount point inside the schroot needs to be made recursively private (rprivate). However, what I meant to say is that the /home/ mount point inside the schroot needs to recursively be made a slave (rslave). The diff at the bottom of the comment is correct.

[Serge Hallyn (serge-hallyn)]

Hi Tyler,

yes since 15.04 switches to systemd which mounts everything as MS_SHARED by default, I expect a few more such bugs. What you recommend sounds right.

[Serge Hallyn (serge-hallyn)]

Note you use slave and private somewhat interchangeably above. Which you want to use will depend on the behavior you want to see if the uesr tries to umount something under /home. If there might be devices or remotes which need to be freed, then you'll wan to use slave so that the chroot can't pin the source. If you actually want the mount to stick around in the chroot then you'll want to use private.

[Michał Sawicz (saviq)]

Seems related to this, after using schroot with overlay on /dev/shm, I end up with multiple /dev/shm mounts on top of one another.

I'll try and collect more data and will comment back here or file a new bug.

[Michał Sawicz (saviq)]

Hmm I take that back, a reboot solved my issues...

[Scott Moser (smoser)]

I've recreated this in a cloud image.
See the steps to do so here in this attachment.

[Tyler Hicks (tyhicks)]

Hi Scott - The bug that you're experiencing is different than this bug. I've opened bug #1438942 to track your issue. Please confirm what I placed in the description and set the bug to confirmed. Thanks!

[Kyle Fazzari (kyrofa)]

Tyler, I got bit by this issue this morning (vivid). The rbind addition outlined in #5 seems to have resolved it for me as well.

[Christian Dywan (kalikiana)]

Adding 'none /home none rslave 0 0' to /etc/schroot/click/fstab allowed me to successfully create a new chroot.

[Timo Jyrinki (timo-jyrinki)]

Confirming the workaround in comment #15 works.

[Kyle Fazzari (kyrofa)]

Does this also bite users who simply have their /home on a separate partition? Disregarding encryption?

[Michał Sawicz (saviq)]

W dniu 13.05.2015 o 14:11, Kyle Fazzari pisze:
> Does this also bite users who simply have their /home on a separate
> partition? Disregarding encryption?

I'm on full-disk-encrypted btrfs with a separate @home and this works
fine AFAICT. Maybe it's mounting /home/$USER itself that's the issue?

[Tyler Hicks (tyhicks)]

On 2015-05-13 12:49:45, Michał Sawicz wrote:
> W dniu 13.05.2015 o 14:11, Kyle Fazzari pisze:
> > Does this also bite users who simply have their /home on a separate
> > partition? Disregarding encryption?
>
> I'm on full-disk-encrypted btrfs with a separate @home and this works
> fine AFAICT. Maybe it's mounting /home/$USER itself that's the issue?

Correct. Any filesystem mounted at /home/$USER will hit this bug (not
just eCryptfs).

[Mitchell (curious-mitchell)]

Wow - this experience has been painful. Have just spent an hour or so debugging my machine to work out why it froze each time I logged into my main user account. No terminals worked, nothing, nada. Finally logged into another user (not with admin rights), used the password from my main account to give it admin rights, then needed to DELETE the line I added from comment #15 above to be able to log back in as my main user again. This is nasty!!!

Looking forward to seeing a workable solution soon.

[Didier Roche (didrocks)]

Confirming that I got the same mount loop issue than Mitchell with the line #15, (thousands of mounts). The worst is that as the sessions are not cleaned up on a forced shutdown, the next login after reboot will restart the schroots, and so the mount hell.

Has to do that either with another admin account or with a live…

[Tyler Hicks (tyhicks)]

I've submitted a proposed fix for this bug to schroot upstream/Debian and linked the bug to this one.

[Christian Dywan (kalikiana)]

Once again wasted an hour for nothing downloading tons of packages, as click very happily downloads the whole package set only to delete it and fail afterall:

Processing triggers for ca-certificates (20141019ubuntu0.15.04.1) ...
Updating certificates in /etc/ssl/certs...
173 added, 0 removed; done. Running hooks in /etc/ca-certificates/update.d....
done.
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
E: 10mount: umount: /var/lib/schroot/mount/click-ubuntu-sdk-15.04-armhf-78d6d7ed-d487-40f3-827e-3cfe9cba6089/home/maple: target is busy E: 10mount: (In some cases useful info about processes that E: 10mount: use the device is found by lsof(8) or fuser(1).)
E: click-ubuntu-sdk-15.04-armhf-78d6d7ed-d487-40f3-827e-3cfe9cba6089: Chroot setup failed: stage=setup-stop
umount: /var/lib/schroot/mount/click-ubuntu-sdk-15.04-armhf-78d6d7ed-d487-40f3-827e-3cfe9cba6089/home/maple: target is busy (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1).)
[...]
Unmount failed... trying to force Unmounting /var/lib/schroot/mount/click-ubuntu-sdk-15.04-armhf-78d6d7ed-d487-40f3-827e-3cfe9cba6089/home/maple failed Deleting /etc/schroot/chroot.d/click-ubuntu-sdk-15.04-armhf Deleting /var/lib/schroot/chroots/click-ubuntu-sdk-15.04-armhf click target was removed successfully
umount: /var/lib/schroot/mount/click-ubuntu-sdk-15.04-armhf-78d6d7ed-d487-40f3-827e-3cfe9cba6089/home/maple: mountpoint not found umount: /var/lib/schroot/mount/click-ubuntu-sdk-15.04-armhf-78d6d7ed-d487-40f3-827e-3cfe9cba6089/home/maple: mountpoint not found
---Click exited with errors, please check the output---

[Dmitry Ershov (edvice)]

I have same error on Ubuntu 14.04 with ecryptfs. Change or comment /home in /etc/schroot/click/fstab does not take effect.
Switching from kernel 3.19.0-29-generic to 3.16.0-50-generic or 3.13.0-64-generic solved the problem.

[Dmitry Ershov (edvice)]

For me this bug fixed in schroot package version 1.6.8-1ubuntu1.1 by https://bugs.launchpad.net/ubuntu/+source/schroot/+bug/1398523.

Can anyone else confirm that bug fixed?

[Nicolas DERIVE (kalon33)]

@edvice: If you look at duplicates, which are recent, you will see that this problem is not fixed, at least on Wily.

[Nathan Osman (george-edison55)]

I am still seeing this on Wily.

    $ dpkg -s schroot | grep Version
    Version: 1.6.10-1ubuntu2

[António Lima (amrlima)]

Not fixed on wily.

[Felipe De la Puente (fdelapuente)]

Hi,

I have /home and /home/${USER}/Data on different partitions which are mounted based on /etc/fstab specs.
When I had automatic updates of my kits in ubuntu sdk, every time it attempted to update it umounted my two home related partitions. This problem was solved by de-activating auto update. Instead I update my kits manually with apt-get.

Now I'm trying to create a new kit for amd64 and have the same problem described here. I'm running 15.04 up-to-date.

So:

1. Please confirm if the current WA would be to do #15 before attempting to modify kits, then once finished revert the WA to avoid #20.

2. Any progress on the fix for this showstopper? Any target release?

Thanks a lot!