when /home is somewhere else, snaps don't work

Bug #1620771 reported by Marcin on 2016-09-06
218
This bug affects 44 people
Affects Status Importance Assigned to Milestone
Snappy
Undecided
Unassigned

Bug Description

Problem:
$ hello-world
cannot bind mount /home to /tmp/snap.rootfs_vAgIND/home. errmsg: Permission denied

When /home is a symlink snaps don't work.
When /home is a real directory snaps work, see output below

Output:
marcin@ubuntu:~$ snap list
Name Version Rev Developer Notes
hello-test 0.01 1 suncheul-kim -
hello-world 6.3 27 canonical -
ubuntu-core 16.04.1 352 canonical -
marcin@ubuntu:~$ hello-world
cannot bind mount /home to /tmp/snap.rootfs_eILRgd/home. errmsg: Permission denied
marcin@ubuntu:~$ hello-test.hello
cannot bind mount /home to /tmp/snap.rootfs_A9iYc5/home. errmsg: Permission denied
marcin@ubuntu:~$ sudo rm -R /home && sudo mkdir -p /home/${whoami} && sudo chmod ugo+rwx /home/$whoami
marcin@ubuntu:~$ hello-test.hello
Hello, world!
marcin@ubuntu:~$ hello-world
Hello World!
marcin@ubuntu:~$ sudo rm -R /home && sudo ln -s /media/Dane/.home/ /home
marcin@ubuntu:~$ hello-world
cannot bind mount /home to /tmp/snap.rootfs_vAgIND/home. errmsg: Permission denied
marcin@ubuntu:~$

My configuration:

$ ll /home
lrwxrwxrwx 1 root root 18 Sep 6 20:02 /home -> /media/Dane/.home//

As /media/Dane is encrypted with LUKS+ext4 fs. I tried to use hardlink but they are not allowed for directories:(

Marcin (mwoz123) wrote :

It also doesn't work when user home is a symlink (and /home "normal" directory) :

marcin@ubuntu:~$ sudo rm -R /home && sudo mkdir -p /home/
marcin@ubuntu:~$ sudo ln -s /media/Dane/.home/marcin/ /home/marcin
marcin@ubuntu:~$ hello-world
failed to create user data directory. errmsg: Too many levels of symbolic links
marcin@ubuntu:~$

Changed in snapcraft:
status: New → Invalid
Marcin (mwoz123) wrote :

@Sergio any reasons/explanations why you changed bug status to "invalid"?
Do you need more information?

Gustavo Niemeyer (niemeyer) wrote :

Hi Marcin,

Sergio just moved over the issue from snapcraft to snapd.

That said, I'm not sure there's much we can do about this. We have strict permissions enforced via apparmor which depend on things being on the correct place.

Have you tried to bind-mount your home instead of symlinking it?

Sergio Schvezov (sergiusens) wrote :

Right, I marked it as Invalid from the point of view of snapcraft; I did not dismiss the problem out of hand as I added the snappy project as well.

Zygmunt Krynicki (zyga) wrote :

Current design of snap-confine makes using symbolic links as the home directory problematic. Snap-confine itself is heavily confined to prevent abuse (as it is a seguid-root executable) and snap-confine carefully avoids traversing symbolic links implicitly. Even when we do the apparmor profile for snap-confine prevents it from performing certain operations, this includes arbitrary source and destination of various mount operations it performs internally.

As a quick workaround, I suspect that you can bind mount custom home directory back to your regular directory (see mount --bind /source /destination) and this should allow you to have the desired data layout and the ability to execute snaps.

Changed in snap-confine:
status: New → Triaged
Zygmunt Krynicki (zyga) on 2016-09-07
Changed in snap-confine:
importance: Undecided → Wishlist
Jamie Strandboge (jdstrand) wrote :

Actually you don't have to do bind mounts. One of the reasons we use apparmor variables for HOME is so that an administrator can adjust what HOME expands to for situations like this.

Feel free to do one of:
$ sudo dpkg-reconfigure apparmor

Or drop a file into /etc/apparmor.d/tunables/home.d that has:
@{HOMEDIRS}+=/media/Dane/.home/

Jamie Strandboge (jdstrand) wrote :

I forgot to mention you'll need to reload the policy after doing this. So after adjust the HOMEDIRS apparmor variable as stated in comment #6, I suggest:

$ sudo rm -f /etc/apparmor.d/cache/* /var/cache/apparmor/snap.*
$ sudo reboot

(there are other ways to do this without a reboot, but this should work regardless of what apparmor policy you have already generated).

Marcin (mwoz123) wrote :

Solved (or rather workaround) :
by using as @zyga suggested mount --bind option.
I added to /etc/fstab entry:
/media/Dane/.home/ /home none bind

summary: - /home symlink, snaps don't work
+ when /home is somewhere else, snaps don't work

I seem to have encountered a slight variant of this issue. snaps don't work when the homedir is a real directory, but not in /home.

Scenario:

A user's home is defined in /etc/passwd as "/nonstandard/home/rebel".

$> ECHO $HOME

/nonstandard/home/rebel

$>hello-world

cannot create user data directory: /nonstandard/home/rebel/snap/hello-world/27: Read-only file system

That's interesting, because that directory is in fact created when running hello-world for the first time:

$tree ./snap

snap
└── hello-world
    ├── 27
    └── common

On the same system, hello-world works fine for a user under /home:

$> ECHO $HOME

/home/wellbehaved

$> hello-world

Hello World!

I will try the workaround in comments #6-#7.

Update: Putting this into /etc/apparmor.d/tunables/home.d/nonstandard_homedirs:

@{HOMEDIRS}+=/nonstandard/home/

Did not work. hello-world still fails claiming read-only filesystem (after writing to it successfully).

Exploring other workarounds. Can't bind mount to /home, because valid homedirs exist there already.

Jamie Strandboge (jdstrand) wrote :

@Terry,

Did you reload the profiles? In particular, you need to reload /etc/apparmor.d/usr.lib.snapd.snap-confine and /var/lib/snap/apparmor/profiles. Eg:

sudo rm -f /etc/apparmor.d/cache/usr.lib.snapd.snap-confine /var/cache/apparmor/snap.*
sudo apparmor_parser -r /etc/apparmor.d/usr.lib.snapd.snap-confine /var/lib/snapd/apparmor/profiles/*

(or reboot after removing the cache files).

@Jamie,

Yes. The reboot was the reason for the break between the "I will try ..." and "Update:" follow up message. Just to be sure. :-)

I tried reloading the profiles first, then rebooted to drive the point home in case the computer was being stubborn. Same result both times.

I did a full experiment to eliminate the possibility that there was something in my existing accounts that was causing the difference.

$ cat /etc/apparmor.d/tunables/home.d/snaptest

# The following is a space-separated list of where additional user home
# directories are stored, each must have a trailing '/'. Directories added
# here are appended to @{HOMEDIRS}. See tunables/home for details.
@{HOMEDIRS}+=/nonstandard/home/

At this point I did both the profile reload you suggested, and also tried deleting the cache files + reboot. In either case the following was the result. (Cleaned up in between, I created the test accounts from scratch after in-place profile reload and after delete+reboot).

** General info **

Test system is KDE Neon. Created by doing a basic install from standard Ubuntu 16.04 install media and then adding the Neon repos afterward. From all outward appearances, this system behaves the same as all my Ubuntu 16.04 systems, which include several Ubuntu Server, Ubuntu MATE and Neon installs. The underlying base system seems to be identical on all of them - they differ only in the desktop components.

** Non standard home location **
/nonstandard is a separate LVM partition on my system, for what it's worth.

mount shows:

/dev/mapper/VG-NONSTANDARD on /nonstandard type ext4 (rw,relatime,data=ordered)

ls -l /nonstandard

drwxr-xr-x 6 root root 4096 Feb 21 14:34 home
drwx------ 2 root users 16384 Apr 4 2004 lost+found

** The test **

$ sudo adduser snaptest-goodfella

* standard adduser output here *

$ sudo adduser snaptest-rebel --home /nonstandard/home/snaptest-rebel

* standard adduser output here *

$ su -l snaptest-goodfella

$ pwd
/home/snaptest-goodfella

$ hello-world
Hello World!

$ tree
.
└── snap
    └── hello-world
        ├── 27
        └── common

$ exit

$ su -l snaptest-rebel

$ pwd
/nonstandard/home/snaptest-rebel

$ hello-world
cannot create user data directory: /nonstandard/home/snaptest-rebel/snap/hello-world/27: Read-only file system

$ tree
.
└── snap
    └── hello-world
        ├── 27
        └── common

$ exit

There's always the chance that there's some peculiarity that has crept into my system over time, so I can test this on a few of my other machines as time permits.

It seems not. I just stopped using snaps since they don't work for me.
I've got a good reason for not placing certain homedirs under /home,
which is after all just a convention and not a requirement. Haven't had
a single problem with using non-conventional homedirs for 35 years with
nary a problem until now.

On 12/20/2017 09:18 AM, Usievaład Kimajeŭ wrote:
> No updates, I see.
>

Jamie Strandboge (jdstrand) wrote :

There are updated workaround procedures listed here:

https://forum.snapcraft.io/t/how-can-i-use-snap-when-i-dont-use-home-user/3352/2

Note that progress has been made on this bug by Zygmunt through the addition of the /var/lib/snapd/apparmor/snap-confine directory. Currently snapd only uses it itself for NFS home, but the design was intended to also address this bug.

Usievaład Kimajeŭ (anibyl) wrote :

This workaround doesn't work actually.

Jamie Strandboge (jdstrand) wrote :

I'm not sure how far you read in the forum, but see: https://forum.snapcraft.io/t/how-can-i-use-snap-when-i-dont-use-home-user/3352/10. Basically, you need to perform a bind mount for it to work due to changes to snap-confine.

Daan W. (bubukind) wrote :

Just to add to this, I got "Too many symbolic links" because ~/snap was a symlink.
The reason for this was that ~ is on NFS with a tight quota, so this was meant as a workaround.

System V. Unix (sysv) wrote :

Adding the non-standard home directories to {HOMEDIRS}, adding a bind mount from /nonstandard/home to /home still causes errors, and it not a valid workaround.

$ opera
cannot create user data directory: /nonstandard/home/sysv/snap/opera/10: Read-only file system

This is really disappointing in the quality of apparmor/snap and Ubuntu 18.04 release. Do better. People have been using non-standard paths for homedirs now, like,...for 30 years on Unix/Linux/BSD. Deal with it.

Bill Korb (korb) wrote :

I agree with the others that have objected to this, as this has entirely broken Ubuntu 18.04 at our company, too. We have a NAS which hosts all user HOME directories, from wherever they may log in.

Unfortunately, this snap stuff fails miserably in such an environment, and there should be a way for you to accomplish your goal without breaking something that we've been using on various *nixes for 30+ years (e.g. Solaris, AIX, HP-UX, SUSE, RHEL, and even Ubuntu up to recently). We are still running Ubuntu 12.04 in some places as your NFS support has gradually degraded over the years, and we're seriously contemplating moving to CentOS/RHEL which seems to be much more enterprise-friendly.

reetp (jcrisp) wrote :

We have experienced this on Trusty 14.04 I was testing to see what might happen on an upgrade.

We run SSSD for logons and a have directory on disk (not a remote mount) of:

/home/server/files/users/username

I've tried the suggested apparmor fixes with no joy.

sudo dpkg-reconfigure apparmor

cat /etc/apparmor.d/tunables/home.d/ubuntu
# This file is auto-generated. It is recommended you update it using:
# $ sudo dpkg-reconfigure apparmor
#
# The following is a space-separated list of where additional user home
# directories are stored, each must have a trailing '/'. Directories added
# here are appended to @{HOMEDIRS}. See tunables/home for details.
@{HOMEDIRS}+=/home/server/files/users/

sudo apparmor_parser -r /etc/apparmor.d/*snap-confine*
sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/*

Note the following seems to have stripped /users/ off the above directory:

Nov 22 16:42:29 desktop kernel: [ 2875.968601] audit: type=1400 audit(1542901349.258:67): apparmor="DENIED" operation="open" profile="/snap/core/5897/usr/lib/snapd/snap-confine" name="/home/server/files/" pid=6254 comm="snap-confine" requested_mask="r" denied_mask="r" fsuid=5001 ouid=0

I also tried:

/var/lib/snapd/apparmor/snap-confine/my-homes

# home directories are in /foo/bar, not /home
mount options=(rw rbind) /home/server/files/users/ -> /tmp/snap.rootfs_*/home/,

sudo apparmor_parser -r /etc/apparmor.d/*snap-confine*
sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/*

That didn't seem to survive a reboot either.

Package: snapd
Status: install ok installed
Priority: optional
Section: devel
Installed-Size: 80575
Maintainer: Ubuntu Developers <email address hidden>
Architecture: amd64
Version: 2.34.2~14.04

I don't want to do a fstab bind mount as my users get confused easily enough already..... one home is enough.

Sorry, but this is shockingly bad.

Justyn Butler (justyn) wrote :

I have also been unable to get the workarounds to work on Ubuntu 18.04.

I'm genuinely surprised and disappointed at how poor this is.

description: updated
Robert Gerstman (fishratcow) wrote :

 I used bindfs to solve this problem.
The bind option to mount wasn't being accepted and none of the other
solutions posted above worked.

I added this to /etc/fstab for /root with the permissions of 0700.

/somewhere/else/root /root fuse.bindfs perms=0700,nonempty 0 0

Steve Langasek (vorlon) wrote :

> cat /etc/apparmor.d/tunables/home.d/ubuntu
> # This file is auto-generated. It is recommended you update it using:
> # $ sudo dpkg-reconfigure apparmor
> #
> # The following is a space-separated list of where additional user home
> # directories are stored, each must have a trailing '/'. Directories added
> # here are appended to @{HOMEDIRS}. See tunables/home for details.
> @{HOMEDIRS}+=/home/server/files/users/

> sudo apparmor_parser -r /etc/apparmor.d/*snap-confine*
> sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/*

> Note the following seems to have stripped /users/ off the above
> directory:

> Nov 22 16:42:29 desktop kernel: [ 2875.968601] audit: type=1400
> audit(1542901349.258:67): apparmor="DENIED" operation="open"
> profile="/snap/core/5897/usr/lib/snapd/snap-confine"
> name="/home/server/files/" pid=6254 comm="snap-confine"
> requested_mask="r" denied_mask="r" fsuid=5001 ouid=0

What this in fact shows is that something is trying to access /home/server/files/ and is denied access. Do you perhaps have users who have a home directory of /home/server/files? Or, are there symlinks somewhere pointing to /home/server/files as the target? From your description of your setup, it is not expected that snaps are trying to access /home/server/files directly.

reetp (jcrisp) wrote :

>What this in fact shows is that something is trying to access /home/server/files/ and is denied access.

Yes, I figured that bit out..... the question is why.

>Do you perhaps have users who have a home directory of /home/server/files?

Not as far as I am aware - see below.

I do have one 'local' user in

/home/localadmin

That accounts is no SSSD and used for local administration only.

>Or, are there symlinks somewhere pointing to /home/server/files as the target?

Not as far as I am aware

>From your description of your setup, it is not expected that snaps are trying to access /home/server/files directly.

No they should be accessing a user directory which make it even more peculiar.

Each user that logs in has a /home/server/files/users directory with all local files.

They also have one Mounts directory in their home that has mapped shares on the server that are added with SSSD/Pam. When you log out they are disconnected.

john@desktop:~$ ll /home/server/files
total 12
drwxr-xr-x 3 root root 4096 Feb 11 2017 ./
drwxr-xr-x 3 root root 4096 Feb 11 2017 ../
drwxr-xr-x 9 root root 4096 Nov 29 2017 users/

john@desktop:~$ ll /home/server/files/users
total 36
drwxr-xr-x 9 root root 4096 Nov 29 2017 ./
drwxr-xr-x 3 root root 4096 Feb 11 2017 ../
drwx------ 32 denise denise 4096 Mar 8 2018 denise/
drwx------ 59 john john 4096 Nov 25 20:40 john/
drwx------ 58 nicky nicky 4096 Jul 27 11:51 nicky/

This should be correct, but clearly something doesn't agree.

@{HOMEDIRS}+=/home/server/files/users/

Fred Drake (fdrake) wrote :

I'm seeing the "cannot create user data directory" error when my $HOME is an ordinary home directory in /home/DOMAIN/, where we're using AD for authentication, and DOMAIN is the AD domain being used. (Of course, the actual ~/snap/kubectl/677 directory is created at some point.)

Does /home/DOMAIN/ need to be added to @{HOMEDIRS}, or is this another unsupported configuration?

This is all on a vanilla Ubuntu 18.10 (cosmic) desktop install.

Yannis Milios (yannis-milios) wrote :

Hello,

I'm testing snaps in CentOS7 and I ran through this problem. Is there a workaround for CentOS, given that AppArmor is an Ubuntu thing AFAIK (CentOS equivalent is SELinux I think, which is currently disabled on my machine) ?

Thanks,
Yannis

Hi all,

Exactly same problem as #27 : CentOS 7, non-default home dir (corporate environment, too many users to change). Any way to solve? Snaps are currently unusable.

Thanks,
Ivo

Gargoyle (g-rgoyle) wrote :

+1

# Steps to reproduce
1.) Clean Ubuntu 18.04 install + Jenkins + lxd snap.
2.) su - Jenkins
3.) lxc list

Above fails with readonly filesystem error because by default Jenkins' homedir is /var/lib/jenkins.

# Workaround
1.) Shutdown Jenkins
2.) mv /var/lib/jenkins /home/jenkins
3.) edit /etc/password and change jenkins users homedir (or use usermod)
4.) edit /etc/default/jenkins and change $JENKINS_HOME to match (ie JENKINS_HOME=/home/$NAME)

Adding a file /etc/apparmor.d/tunables/home.d/jenkins with the @{HOMEDIRS} as suggested above didn't work for me either.

I would have thought using lxc containers to perform automated builds and testing with jenkins is a fairly common & practical use case, so it would be good to figure out a reliable fix for this.

I have the same issue for about 1500 Users having their home not at /home

Hi,

On OpenSUSE 15.0, i have a homedir mounted on a CIFS network share for all the student and teacher in our university.

I installed two applications : visualfm and cloudcompare.
I have the error for both :

cannot open path of the current working directory: Permission denied

Even if snap is able to create my /homedir/snap directory.

Vijay Raghav (raghav998) wrote :

I have also the same issue (Corporate Environment), now snap is unusable

cement_head (andorjkiss) wrote :

HOLY COWS BATMAN! This is completely ridonculous. We have a system that uses a small (1 TB SSD) for OS/Login and $HOME(s) are on a 25TB RAID10 system. SNAPs are now unusable? This is bonkers....

Zygmunt Krynicki (zyga) wrote :

Dear @cement_head, you can always mount your home directory under /home/$LOGNAME, even if it is on a 25TB RAID10 system. It's not about where it comes from, it's about where it is in the system. That's all.

John Lenton (chipaca) wrote :

Setting this to Won'tFix, as it's unlikely we'll fix it for the foreseeable future -- the workaround for people insisting on non-standard home locations should work for most.

Changed in snappy:
status: New → Won't Fix
no longer affects: snapcraft
no longer affects: snap-confine
John Lenton (chipaca) wrote :

FWIW this is mentioned explicitly in "Limitations in snapd", https://forum.snapcraft.io/t/limitations-in-snapd/9718

ericc (eric-cheminot) wrote :

Hello,

I have a similar behaviour, as mentioned in comment #1 - that is not (I think) covered by the limitations linked just above (#36).

The homedirs is standard (/home), but for certain users, their home dir is a symlink to another folder: /home/user1 -> /accounts/user1. In this case, /home/user1/snap is "standard" layout, but snap keeps pretending that e.g. "/home/user1/snap/spotify/36" is not a directory - which is not true.

Comment #34 is not correct for me - even though the argument is logical, it is unfortunately not true. It may come from the facts that I'm using BTRFS volumes (I've already seen specific behaviours with containers and BTRFS, but I'm not technical enough to draw a conclusion). So, no, it's not "all".

Also, adding /accounts to apparmor homedirs is without effect here.

Oliver Grawert (ogra) wrote :

@ericc if you switch these dirs from symlinks to being bind mounts it will work ...

i guess @zyga should add this to his list of limitations (symlinks do in general not work with apparmor, but bind-mounts are a safe replacement for them)

ericc (eric-cheminot) wrote :

@ogra Thanks, I was not sure I could do this (again, not my domain). I have multiple users having their home under /accounts mount:
  /home/user1 -> (links to) /accounts/user1
  /home/user2 -> /accounts/user2

So the only volume candidate for bind mounting for me would be /accounts. The only solution would be to have a sub-volume for each user. I've just tried with a dummy user, but while I can I can mount @dummy subvol to /home/dummy, it complains that /home/dummy is not a folder when adding "bind" option:
  # mount -o subvol=@dummy /dev/sdc1 /home/dummy => [OK]
  # mount _o bind,subvol=@dummy /dev/sdc1 /home/dummy => [KO, /home/dummy not a folder (or directory, I'm not using an English console]

But... actually, following your suggestion I've insisted on the proposed solution and I can do a mount bind at the user folder level, that is exactly instead of links ((I did not know that). And, yes, it works now! Thanks!

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers