snaps don't work with NFS home

[Robert Redl]

Strictly confined snap commands that don't use networking in their interfaces (eg, 'plugs: [ network ]') do not work for users with NFS home because of AppArmor denials for networking.

WORKAROUND:
Add the following to /etc/apparmor.d/abstractions/base and /etc/apparmor.d/usr.lib.snapd.snap-confine:
network inet,
network inet6,

Then reload policy with:
$ sudo apparmor_parser -r /etc/apparmor.d/usr.lib.snapd.snap-confine
$ sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.*

Unfortunately this adds networking to all profiles on the system. snapd could be updated to conditionally add these rules to snap-confine and non-daemon commands to help users. When fine-grained network mediation is implemented in AppArmor, it may be able to help limit the scope of the added rules (but this would need to be researched, NFS in the kernel is a bit twisty).

= Original report =

Our home directories have the following structure:

/home/u/user.name

where u is the first letter of the users first name. The reason for this structure is the large number of users. The nfs mount point is /home

The file /etc/apparmor.d/tunables/home.d/ubuntu contains the following line:

@{HOMEDIRS}+=/home/u/
(for one example user)

@{HOMEDIRS}+=/home/*/
did also not work.

Starting a snap (in this example case inkscape) results in the following error message:
cannot change current working directory to the original directory: Permission denied

For a self-created snap in classic mode, I get the following error:
cannot create user data directory: /home/u/user.name/snap/mysnap/x1: Permission denied

The journal contains the following messages:
kernel: nfs: RPC call returned error 13
kernel: audit: type=1400 audit(1486481365.925:127): apparmor="DENIED" operation="sendmsg" profile="/usr/lib/snapd/snap-confine" pid=25069 comm="snap-confine" laddr=x.x.x.x lport=782 faddr=x.x.x.x fport=2049 family="inet" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send"

Installed packages:
snapd/xenial-proposed,now 2.22.2 amd64 [installed]
snap-confine/xenial-proposed,now 2.22.2 amd64 [installed]
ubuntu-core-launcher/xenial-proposed,now 2.22.2 amd64 [installed]

[Jamie Strandboge (jdstrand)]

Thank you for filing a bug! This is essentially a duplicate of bug #1620771. You have identified the issue precisely and need to update @{HOMEDIRS} for your site. This can be done in a couple of ways such as the ones you described, but you'll need to also reload the policy after making the changes. Ie, modify one of /etc/apparmor.d/tunables/home, /etc/apparmor.d/tunables/home.d/ubuntu or /etc/apparmor.d/tunables/home.d/something, then do:

$ sudo rm -f /var/cache/apparmor/snap.*
$ sudo apparmor_parser -r --write-cache --cache-loc /var/cache/apparmor /var/lib/snapd/apparmor/profiles/snap.*

Then it should work. Hope this helps!

[Robert Redl (r5r3)]

Thanks for the fast reply! Unfortunately, the problem is not solved.

The
@{HOMEDIRS}+=/home/*/
line solves the location issue (as in bug #1620771 and bug #1592696), but here the location don't seems to be the real issue.

To prove that, I replaced the nfs mounted home directory with a home directory on a local disk with the very same path. Then, snaps work as expected. From that, and from the error message included in my first post, my guess is, that the actual problem is nfs related. The error message says, that apparmor denies access to port 2049 of the remote host (which is the default nfs port).

[Jamie Strandboge (jdstrand)]

Can you paste the output of "grep audit /var/log/syslog" at the time right after the denial?

[Robert Redl (r5r3)]

Feb 9 09:57:30 hostname kernel: [ 2070.523056] audit: type=1400 audit(1486630650.755:1460): apparmor="DENIED" operation="sendmsg" profile="/usr/lib/snapd/snap-confine" pid=15768 comm="snap-confine" laddr=ip_of_local_host lport=917 faddr=ip_of_nfs_server fport=2049 family="inet" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send"
Feb 9 09:57:30 hostname kernel: [ 2070.523098] audit: type=1400 audit(1486630650.755:1461): apparmor="DENIED" operation="sendmsg" profile="/usr/lib/snapd/snap-confine" pid=15768 comm="snap-confine" laddr=ip_of_local_host lport=917 faddr=ip_of_nfs_server fport=2049 family="inet" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send"
Feb 9 09:57:30 hostname kernel: [ 2070.523323] audit: type=1400 audit(1486630650.755:1462): apparmor="DENIED" operation="sendmsg" profile="/usr/lib/snapd/snap-confine" pid=15768 comm="snap-confine" laddr=ip_of_local_host lport=917 faddr=ip_of_nfs_server fport=2049 family="inet" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send"
Feb 9 09:57:30 hostname kernel: [ 2070.523349] audit: type=1400 audit(1486630650.755:1463): apparmor="DENIED" operation="sendmsg" profile="/usr/lib/snapd/snap-confine" pid=15768 comm="snap-confine" laddr=ip_of_local_host lport=917 faddr=ip_of_nfs_server fport=2049 family="inet" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send"
Feb 9 09:57:30 hostname kernel: [ 2070.523576] audit: type=1400 audit(1486630650.755:1464): apparmor="DENIED" operation="mkdir" profile="/usr/lib/snapd/snap-confine" name="/home/r/" pid=15768 comm="snap-confine" requested_mask="c" denied_mask="c" fsuid=10270 ouid=10270

[Jamie Strandboge (jdstrand)]

Ok, that makes a lot of sense. snap-confine needs to be update to work on nfs (eg, add 'network inet, network inet6,'. Based on 'name="/home/r/"' it looks like you are still using the '@{HOMEDIRS}+=/home/u/' change to the home tunable (or perhaps you didn't reload snap-confine's profile after changing it)?

Robert, can you:
1. adjust the tunable to have: @{HOMEDIRS}+=/home/*/
2. adjust /etc/apparmor.d/usr.lib.snapd.snap-confine to have: #include <abstractions/nameservice>
3. reload the snap-confine profile with:
   sudo apparmor_parser -r /etc/apparmor.d/usr.lib.snapd.snap-confine

I suspect you may need to also add to /etc/apparmor.d/usr.lib.snapd.snap-confine:
#include <abstractions/base>

Once you do the above, can you report back? I suspect that any snaps that don't 'plugs: [ network ]' will fail to work for the same reasons as snap-confine. Eg, hello-world will fail, but vlc will not.

[Robert Redl (r5r3)]

1. I already had @{HOMEDIRS}+=/home/*/ and I did not forget to reload. However, the audit message still refers to /home/r/, which is the actual parent directory of my home directory.

2. Including #include <abstractions/nameservice> directly below /usr/lib/snapd/snap-confine flags=(attach_disconnected) works. It don't seems to be necessary to include #include <abstractions/base>

Adding network inet and network inet6 solves the problem as well.

Snaps without network interface: Right, they don't work. A workaround is to install them in devmode.

Enabling network access for all snaps just to make them compatible with NFS don't seems to be a perfect solution from the security perspective. Doesn't that mean, that these snaps can access every network service, not only NFS? Is it possible to explicitly enable NFS (and other network file systems) and not enable network access for all kinds of services?

[Seth Arnold (seth-arnold)]

On Fri, Feb 10, 2017 at 08:15:42AM -0000, Robert Redl wrote:
> 2. Including #include <abstractions/nameservice> directly below
> /usr/lib/snapd/snap-confine flags=(attach_disconnected) works. It don't
> seems to be necessary to include #include <abstractions/base>

You may have trouble killing the processes running in this domain from
unconfined domains without going to some effort to replicate what
<abstractions/base> already provides for you. Give it a read and see what
else you may be missing.

Thanks

[Robert Redl (r5r3)]

Will there be an Update of the installation package? Will the lines

#include <abstractions/nameservice>
#include <abstractions/base>

be included?

[Jamie Strandboge (jdstrand)]

Robert, your comment "Enabling network access for all snaps just to make them compatible with NFS don't seems to be a perfect solution from the security perspective" is exactly right. It is not possible (currently) to only allow networking for NFS. This may be possible at some point in the future with fine-grained network mediation.

Based on your report, in the meantime, you can add this to /etc/apparmor.d/abstractions/base which will apply to all snaps and for snap-confine, to /etc/apparmor.d/usr.lib.snapd.snap-confine:

network inet,
network inet6,

Then reboot (this will trigger a profile recompile for everything). As mentioned, this is unfortunate but the only workaround atm.

[Roger Light (roger.light)]

Thanks for the workaround.

[Jamie Strandboge (jdstrand)]

FYI, https://forum.snapcraft.io/t/snaps-and-nfs-home/438

[Björn Torkelsson (torkel)]

Any news on this bug (for 16.04)?

Unfortunately the workaround does not work for me.

In my case /home is a link to /import/home (i.e home is autofs mounted), so I guess my problem is also related to #1620771?

Btw, I assume that /etc/apparmor.d/usr.lib.snapd.snap-confine in the workaround should really be /etc/apparmor.d/usr.lib.snapd.snap-confine.real ?

[Jamie Strandboge (jdstrand)]

The workaround is brittle with newer snapds since if the core snap is newer than the installed snapd, the profile for snap-confine from the core snap is used. This profile is on read-only media. The /etc/apparmor.d/usr.lib.snapd.snap-confine.real is the profile these days for the snap-confine on the system.

@Bjorn, you are also hitting https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1662552/comments/5

[Jamie Strandboge (jdstrand)]

I've also updated https://forum.snapcraft.io/t/snaps-and-nfs-home/438/8 for additional implementation details. I believe there is enough there for someone to pick up the work. While this is on the snapd team's radar, it is currently not prioritized. If someone wants to implement a fix for this sooner than later, please feel free to ping me or zyga if you have any questions.

[Michael Iatrou (michael.iatrou)]

There is also the case of snaps with classic confinement, which exhibit the same behavior as described here: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1713767

[Zygmunt Krynicki (zyga)]

I'm working on a workaround for this. I have some initial patches that change the policy for snap-confine and other snaps and I'll propose them as soon as I can write some spread tests to ensure this works.

[Zygmunt Krynicki (zyga)]

I have implemented a proof of concept and got initial round of feedback from the team. I will address the mentioned issues, add some more tests and propose a PR for evaluation.

[Zygmunt Krynicki (zyga)]

This is now under review at https://github.com/snapcore/snapd/pull/3958

[Gustavo Niemeyer (niemeyer)]

It's in master now.

[Björn Torkelsson (torkel)]

Any change that this will be fixed in/backported to Xenail?

[Zygmunt Krynicki (zyga)]

Yes, this will be available in Xenial when 2.29 is released to stable.
If you switch the core snap to the "beta" channel with "snap refresh
--beta core" you can get advantage of this straight away.

On Mon, Oct 30, 2017 at 12:42 AM, Björn Torkelsson
<email address hidden> wrote:
> Any change that this will be fixed in/backported to Xenail?
>
> --
> You received this bug notification because you are a member of Snappy
> Developers, which is subscribed to snapd in Ubuntu.
> Matching subscriptions: xxx-bugs-on-snapd
> https://bugs.launchpad.net/bugs/1662552
>
> Title:
> snaps don't work with NFS home
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/snapd/+bug/1662552/+subscriptions

[Oliver Grawert (ogra)]

just a FYI. all snapd development happens focused on the last LTS (xenial) and fixes are either forward or backward ported to the other supported releases ... so xenial indeed gets it first in any case (thats true for all snapd fixes (at least until the next LTS comes around))

[Björn Torkelsson (torkel)]

@ogra Ah, thanks for the info.

@zyga Switched to the beta channel, but still not working :-(

1 torkel@matilda:~⟫ hello-world
cannot perform operation: mount --rbind /home /tmp/snap.rootfs_EilQKW//home: Permission denied

Nov 5 00:52:36 matilda kernel: [119451.849036] audit: type=1400 audit(1509839556.242:534): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="/snap/core/3396/usr/lib/snapd/snap-confine" name="/tmp/snap.rootfs_J0J8C2/home/" pid=13237 comm="snap-confine" srcname="/import/home/" flags="rw, rbind"

After switching to the beta channel I also tried https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1662552/comments/5 but that didn't help.

I assume that there are still some problems when /home is a symbolic link to /import/home (i.e https://bugs.launchpad.net/snapcraft/+bug/1620771)?

[Zygmunt Krynicki (zyga)]

This has been released for a while now. Please reopen if there are more issues.

Note that /home/$anything cannot be a symbolic link. Please use a bind mount if necessary (this is a separate issue).

[Michal Kukuča (michalmaria)]

With Ubuntu Bionic 18.04, snapd version 2.32, Snap still doesn’t work correctly with home folders mounted via NFS. I’m even unable to use basic applications like System monitor:

kernel: nfs: RPC call returned error 13
kernel: audit: type=1400 audit(1526303580.147:309): apparmor=“DENIED” operation=“sendmsg” profile="/snap/core/4571/usr/lib/snapd/snap-confine" pid=32548 comm=“snap-confine” laddr=10.0.1.198 lport=1003 faddr=10.0.1.98 fport=2049 family=“inet” sock_type=“stream” protocol=6 requested_mask=“send” denied_mask="send"
gnome-system-monitor_gnome-system-monitor.desktop[31416]: cannot change current working directory to the original directory: Permission denied
cannot create user data directory: /home/michalmaria/snap/gnome-system-monitor/39: Stale file handle

Our configuration:
1. Ubuntu server 18.04, provides home folders, OpenLDAP+Kerberos+NFS.
2. Ubuntu 18.04 clients. Home folders are mapped directly to /home, everything works (as it always did), but snaps won't run.

[John Lenton (chipaca)]

Michal, have you done the things described in
https://forum.snapcraft.io/t/snaps-and-nfs-home/438 ?
On Mon, 14 May 2018 at 14:46, Michal Kukuča <email address hidden>
wrote:

> With Ubuntu Bionic 18.04, snapd version 2.32, Snap still doesn’t work
> correctly with home folders mounted via NFS. I’m even unable to use
> basic applications like System monitor:

> kernel: nfs: RPC call returned error 13
> kernel: audit: type=1400 audit(1526303580.147:309): apparmor=“DENIED”
operation=“sendmsg” profile="/snap/core/4571/usr/lib/snapd/snap-confine"
pid=32548 comm=“snap-confine” laddr=10.0.1.198 lport=1003 faddr=10.0.1.98
fport=2049 family=“inet” sock_type=“stream” protocol=6
requested_mask=“send” denied_mask="send"
> gnome-system-monitor_gnome-system-monitor.desktop[31416]: cannot change
current working directory to the original directory: Permission denied
> cannot create user data directory:
/home/michalmaria/snap/gnome-system-monitor/39: Stale file handle

> Our configuration:
> 1. Ubuntu server 18.04, provides home folders, OpenLDAP+Kerberos+NFS.
> 2. Ubuntu 18.04 clients. Home folders are mapped directly to /home,
everything works (as it always did), but snaps won't run.

> --
> You received this bug notification because you are a member of Snappy
> Developers, which is subscribed to snapd in Ubuntu.
> https://bugs.launchpad.net/bugs/1662552

> Title:
> snaps don't work with NFS home

> To manage notifications about this bug go to:
> https://bugs.launchpad.net/snapd/+bug/1662552/+subscriptions

[Zygmunt Krynicki (zyga)]

Michal Kukuča, we have a test case but it may be missing something that is evidently going on in your use case. I will review the code, the denial and get back to you shortly.

[Zygmunt Krynicki (zyga)]

Michal, can you please list the contents of this directory:

/var/lib/snapd/apparmor/snap-confine

In addition, can you please provide the fstab entry that mounts the NFS share?

I'm trying to understand if our NFS detection logic is kicking in

[Jason D. Kelleher (jdkelleher)]

The issue appears larger than just NFS mounted $HOME.

After I used the workaround provided to get the filebot snap to run with an NFS mounted $HOME, I was unable to use it to manipulate any files in my NFS mounted media library.

I gave up and just loaded the deb package.

Should this be reported as a separate bug?

[Zygmunt Krynicki (zyga)]

Hello jdkelleher.

Can you please provide more details about your setup? Things like "snap version" output, the location of the NFS mount, the snap you used, "dmesg | grep DENIED" output when you tried.

Ideally please open a new bug to scope the conversation there.

[Andrew Conway (acubuntuone)]

I have the same problem. The fix does not help. I use autofs to mount particular users rather than all of /home, which I think the fix requires. Someone else doing the same thing as me opened a new bug 1782873 with details of setup, but I think the issue is the autofs rather than boot mounting of /home.

[Markus Kuhn (markus-kuhn)]

The real bug here is that AppArmor should restrict NFS access only via the file-path rules, and not via the network rules, since if an application accesses a file via NFS, all related network traffic is initiated and controlled by the kernel (or by kernel helper processes like automount, rpc.gssd and nfsidmap), and not by the application. NFS access really needs to be fixed in AppArmor and anything else is just a temporary workaround. Until then, AppArmor is too blunt a tool for restricting network access.

See also bug #1784499 (AppArmor prevents /usr/bin/man from reading NFS-mounted man pages)

[Craig Hansen (craig-hansen)]

I'm finding that when vlc is installed via snap, it's unable to play a file that is NFS automounted. Fixes that target only /home are missing the point. Is there a more general workaround or fix that addresses Markus Kuhn's and Andrew Conway's observations above?

[Michal Kukuča (michalmaria)]

Sorry guys for not responding to your questions. I somehow didn't get the notifications about the discussion here. Also: I didn't really look further into this, because of a simple workaround I had to implement: remove all snap apps and install them from the repos.

Now I did recently upgrade all our computers to Ubuntu 18.10 and the issue still persists, so I'm using the same "workaround" as well, but I will keep one computer as is (with snap apps), so if someone would like to know how things are working (or not working) here, I will be happy to provide you with more details.

Current setup:
snap 2.36.2
snapd 2.36.2
series 16
ubuntu 18.10
kernel 4.18.0-11-generic

/home is mounted via fstab at boot time like this:
10.0.1.98:/home /home nfs4 rw,hard,bg,sec=krb5 0 0

Running a snap app will not do anything for the user, but will report errors like these in the journal:

gnome-calculator 3.30.1 260 stable/… canonical✓ -

dec 13 15:50:36 jerusalem dbus-daemon[2172]: [session uid=10000 pid=2172] Activating via systemd: service name='org.freedesktop.portal.Documents' unit='xdg-document-portal.service' requested by ':1.124' (uid=10000 pid=4948 comm="/snap/bin/gnome-calculator " label="unconfined")
dec 13 15:50:36 jerusalem systemd[2095]: Starting flatpak document portal service...
dec 13 15:50:36 jerusalem xdg-document-portal[4958]: error: Failed to load db: Chyba pri otváraní súboru /home/michalmaria/.local/share/flatpak/db/documents: Adresár alebo súbor neexistuje
dec 13 15:50:36 jerusalem systemd[2095]: xdg-document-portal.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
dec 13 15:50:36 jerusalem systemd[2095]: xdg-document-portal.service: Failed with result 'exit-code'.
dec 13 15:50:36 jerusalem systemd[2095]: Failed to start flatpak document portal service.
dec 13 15:52:36 jerusalem dbus-daemon[2172]: [session uid=10000 pid=2172] Failed to activate service 'org.freedesktop.portal.Documents': timed out (service_start_timeout=120000ms)
dec 13 15:52:36 jerusalem gnome-calculator_gnome-calculator.desktop[2311]: cmd_run.go:832: WARNING: cannot start document portal: Failed to activate service 'org.freedesktop.portal.Documents': timed out (service_start_timeout=120000ms)
dec 13 15:52:36 jerusalem gnome-calculator_gnome-calculator.desktop[2311]: cannot change current working directory to the original directory: Permission denied

Another example:
spotify 1.0.94.262.g3d5c231c-9 28 stable spotify✓ -
dec 13 15:55:18 jerusalem dbus-daemon[2172]: [session uid=10000 pid=2172] Activating via systemd: service name='org.freedesktop.portal.Documents' unit='xdg-document-portal.service' requested by ':1.138' (uid=10000 pid=5160 comm="/snap/bin/spotify " label="unconfined")
dec 13 15:55:18 jerusalem systemd[2095]: Starting flatpak document portal service...
dec 13 15:55:18 jerusalem xdg-document-portal[5172]: error: Failed to load db: Chyba pri otváraní súboru /home/michalmaria/.local/share/flatpak/db/documents: Adresár alebo súbor neexistuje
dec 13 15:55:18 jerusalem systemd[2095]: xdg-document-portal.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
dec 13...

[Philippe Coval (rzr)]

May I add that snap is not working on HOMEs which have symlinks in path (ie: ln -fs /local/home/user /home/user)

[Marc Kolly (makuser)]

I recently reinstalled around 20 fat-clients to Ubuntu 18.04, which were previously running 14.04 and I am running into the same problem.

Just like michalmaria, our configuration is:
1. Debian 10, provides home folders via NFS and runs Kerberos, OpenLDAP is off-site.
2. Ubuntu 18.04 clients. Home folders are mounted at /user, everything works most of the time, sometimes the UI freezes though requiring a restart and snaps won't run at all, showing the same errors on the journal as michalmaria already posted.

I think the main difference is I mount the homes to /user, leaving /home untouched for local accounts and so the fix which I assume is just for /home will not work?

/etc/apparmor.d/tunables/home.d/site.local contains @{HOMEDIRS}+=/user/

What do you all suggest and is there another bug# that might help more? I am already aware of #1784774, but I am not using autofs, although I could and would switch to that if that at least works...

[Zygmunt Krynicki (zyga)]

Hello

Please forgive me this brief answer. Snaps require user home directories mounted at /home. Using /user regardless of NFS is not supported. I believe another bug tracks that issue

[E. Madison Bray (erik-m-bray+ubuntu)]

I think I've discovered a different wrinkle to this issue that I haven't seen discussed here.

There are, in my experience with trying to work around this, three separate issues.

1. It's necessary to add the root of your NFS home directories to {@HOMEDIRS}. I have done that, and confirmed that the rules in the apparmor profile for snap-confine expand correctly and do allow writing to ~/snap/

2. Next, I still had issues where, when trying to run a snap, the following audit messages were in my syslog:

    Sep 18 12:43:29 <hostname> kernel: [ 4603.119892] nfs: RPC call returned error 13
    Sep 18 12:43:29 <hostname> kernel: [ 4603.119926] audit: type=1400 audit(1600433009.543:113): apparmor="DENIED" operation="sendmsg" profile="/snap/snapd/9279/usr/lib/snapd/snap-confine" pid=4371 comm="snap-confine" laddr=<laddr> lport=837 faddr=<faddr> fport=2049 family="inet" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send"

I took steps similar to https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1662552/comments/9 which is less than ideal, but reported to work. In my case, I added

    network inet,
    network inet6,

to /var/lib/snapd/apparmor/snap-confine/nfs

Mysteriously, upon rebooting, this file *vanishes*, but it seems not before the apparmor caches are regenerated and that file *does* get included (I confirmed this by putting garbage in it, and confirming that AppArmor reported syntax errors when starting up).

I further confirmed that when running the snap, although I still get "permission denied", there are no longer *any* audit messages from apparmor.

3. That's because there appears to be a third, very simple issue, having nothing to do with apparmor. Because snap-confine is setuid it runs as root, and like many NFS mounts mine is configured to treat root as a normal user when reading users' home directories.

Early in its startup, snap-confine runs sc_preserve_and_sanitize_process_state() https://github.com/snapcore/snapd/blob/9defc23e31afee03cf5994f309422681971f42db/cmd/snap-confine/snap-confine.c#L322
This tries to use openat() to open a handle to CWD to store before calling chdir("/"). Since this
happens well before sc_set_effective_identity() it will fail if your CWD happens to be in the NFS mount.

This is easily worked around of course by changing directory outside the NFS mount, but for a typical user they will be typically sitting in their home directory, and this is very confusing.

[Marc Kolly (makuser)]

Hello all,

we have since then upgraded all our machines to Ubuntu 20.04 and are facing the same issues here with our NFS4 homes.

I moved the mountpoint from /user to /home as @zyga suggested, but had no success actually running snaps.

Oct 05 15:55:20 pc-021 systemd[1583]: Started Application launched by gnome-shell.
Oct 05 15:55:20 pc-021 chromium_chromium.desktop[4308]: cannot open path of the current working directory: Permission denied
Oct 05 15:55:20 pc-021 systemd[1583]: gnome-launched-chromium_chromium.desktop-4308.scope: Succeeded.

[Jason D. Kelleher (jdkelleher)]

My solution has been to fully remove all snap functionality.

sudo apt purge snapd
sudo rm -vrf /snap /var/snap /var/lib/snapd /var/cache/snapd /usr/lib/snapd

[Zygmunt Krynicki (zyga)]

Marc Kolly, could you please open a new bug with details of your particular configuration please. Commenting on a bug that is fix released is not useful as we have integration tests that verify this particular particular behavior works. I suspect there is something about your setup we are misidentifying, which is causing further issues.

[Marc Kolly (makuser)]

Yes, I also removed snapd on all machines in the office, because it was a total unstable mess in 18.04 and reinstalled the Gnome calculator and some other apps that had been switched to snaps by default via a method that actually works everywhere.

Unfortunately the chromium package from the Ubuntu apt repository, which is installed *VIA APT* is now installing a snap package. Insane IMHO, but that's how it is.
And unfortunately some gov. sites work poorly in Firefox, so we have to install Chromium as well, to be able to use those and therefore need snap again...

[Zygmunt Krynicki (zyga)]

@Marc, what is the bug report you've opened?

Chromium has no more debian packages as the team supporting it decided that it is exceedingly difficult to maintain (especially for releases other than latest). This is unrelated to snapd. Without snapd it would simply be removed from the archive.

[Markus Kuhn (markus-kuhn)]

Having briefly tried to get snap to work a few years ago on Ubuntu 18.04, before giving up, I have just upgraded to Ubuntu 20.04, and snap appears to cause problems again, especially as larger parts of Ubuntu now seem to rely on snap (e.g. the "Software Install" GUI tool). Therefore I'm interested in hearing what progress there has been since 18.04 with snap in environments with NFS-mounted $HOME with symbolic links in the $HOME path.

My environment is a centrally managed Linux workstation in a major computer-science department, where all user home directories are located on an NFS filer. Specifically, my environment has the following characteristics:

$ getent passwd mgk25
mgk25:*:1597:1597:Markus Kuhn:/home/mgk25:/bin/bash
$ echo $HOME
/home/mgk25
$ ls -l /home
lrwxrwxrwx 1 root root 5 Oct 22 23:35 /home -> homes
$ ls -l /homes
lrwxrwxrwx 1 root root 11 Oct 13 2008 /homes -> /auto/homes
$ ls -ld /auto/homes/mgk25
drwxr-xr-x 223 mgk25 mgk25 65536 Oct 23 17:02 /auto/homes/mgk25
$ findmnt $HOME
TARGET SOURCE FSTYPE OPTIONS
/auto/homes/mgk25 wulbor.cl.cam.ac.uk:/userfiles/mgk25/unix_home nfs4 rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp6,timeo=600,retrans=2,sec=krb5p,clientaddr=2001:630:272:238:3656:c6fd:1234:5678,local_lock=none,addr=2a05:b400:110:a9::4321:abcd

In other words:

- my $HOME contains two symbolic links in its path

- my $HOME is auto-mounted via either NFSv3 or NFSv4.1 using Kerberos authentication (sec=krb5p)

- the filer implements root squashing

- the Kerberos machine credential in /etc/krb5.keytab (used by processes with EUID=root) maps on the filer to user "nobody" and therefore only has "others"-level access to files on the filer.

- local system user/group IDs (any UID/GID < 1000 other than root) do not have any Kerberos ticket and therefore have no access to the filer

Is there any chance that snap could ever work in such an environment?

The very idea of mounting something inside ~/.snap/ seems infeasible in such an environment. So is there any way to use snapd such that it never accesses $HOME?

[Zygmunt Krynicki (zyga)]

Hey Markus.

Thank you for providing details about your environment. Unfortunately we don't have any new ideas on how to solve home-at-nearly arbitrary path and we certainly didn't have time to push this idea forward.

One idea I had a while ago is to mount whatever the original location of HOME, at /var/home/$LOGNAME, offering unique path for any user. This is assuming the path cannot be represented in the original location.

This technically could work, we would have a fixed extension of various key paths and the security subsystem would be able to handle that. What would not be optimal is the disconnect between what a snap application process thinks on the inside of the environment (e.g. HOME is /var/home/zyga) and what the system meta-data says (e.g. /home/zyga).

We have a variation of that problem today (/home/zyga for real and /home/zyga/snap/$SNAP_NAME/$SNAP_REVISION) but we wanted to figure out how to avoid that as well. If the end user experience is acceptable, that would be something worth considering, especially if this makes the snap ecosystem usable.

I cannot comment on the kerberos details. I think snapd could work in this environment as we have supported root squashing NFS home in /home/$LOGNAME before. I think we could only push forward (e.g. handle more paths via /var/home idea) and see what's missing.

I can try proposing an experimental feature for /var/home and see if that helps you out. I could do this assuming you are willing to report back and share your experience and issues you've encountered.

[Markus Kuhn (markus-kuhn)]

I should add that at the moment, my first problem is still that NFS seems not permitted at all:

$ aptitude show snapd
Package: snapd
Version: 2.47.1+20.04
State: installed

# snap install pdftk
2020-10-26T15:12:47Z INFO Waiting for automatic snapd restart...
pdftk 2.02-4 from Scott Moser (smoser) installed

$ /snap/bin/pdftk
2020/10/26 17:28:48.784329 cmd_run.go:570: WARNING: XAUTHORITY environment value is not a clean path: "/auto/homes/mgk25/.Xauthority"
cannot open path of the current working directory: Permission denied

$ dmesg
[ 8862.382001] nfs: RPC call returned error 13
[ 8862.382037] audit: type=1400 audit(1603733328.782:345): apparmor="DENIED" operation="sendmsg" profile="/usr/lib/snapd/snap-confine" pid=54514 comm="snap-confine" laddr=2001:630:272:238:3656:c6fd:1234:5678 lport=844 faddr=2a05:b400:110:a9::4321:abcd fport=2049 family="inet6" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send"

So I suspect the first problem is still that "network inet6" is not allowed, and therefore any form of NFS remains blocked. snapd apparently did not detect that I am using an autofs home directory.

I should probably add that on the centrally-managed Linux desktop that I use, the automount tables are disseminated via LDAP, as described e.g. in

  https://help.ubuntu.com/community/AutofsLDAP

In particular, /etc/nsswitch.conf contains "ldap" in the following three lines:

  passwd: files systemd ldap
  group: files systemd ldap
  automount: ldap

Therefore: can snapd reliably detect autofs mount points that are added via LDAP?

If not, is there some manual switch by which we can tell snapd that all my snaps will need network access, as I am going to use them routinely via NFS?

But lack of "network" permissions is not the only problem:

If I follow the workaround described in the bug-report above, by adding the lines

  # see https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1662552
  network inet,
  network inet6,

to both

  /etc/apparmor.d/abstractions/base
  /etc/apparmor.d/usr.lib.snapd.snap-confine.real

then the error message I get changes to

$ /snap/bin/pdftk
2020/10/26 17:48:11.825599 cmd_run.go:570: WARNING: XAUTHORITY environment value is not a clean path: "/auto/homes/mgk25/.Xauthority"
cannot perform operation: mount --rbind /home /tmp/snap.rootfs_NnRQQa//home: Permission denied
$ dmesg
[10025.430756] audit: type=1400 audit(1603734491.825:351): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="/usr/lib/snapd/snap-confine" name="/tmp/snap.rootfs_NnRQQa/home/" pid=57898 comm="snap-confine" srcname="/auto/homes/" flags="rw, rbind"

We have already in /etc/apparmor.d/tunables/home the lines

  @{HOME}=@{HOMEDIRS}/*/ /root/
  @{HOMEDIRS}=/home/ /auto/homes/

to indicate where our automounted home directories are.

Any idea what me be going wrong there?

P.S.: Note that on my Ubuntu 20.04 system, the file

  /etc/apparmor.d/usr.lib.snapd.snap-confine

mentioned above does not exist, and instead there is

  /etc/apparmor.d/usr.lib.snapd.snap-confine.real

[Andrew Conway (acubuntuone)]

This still doesn't work with 22.04, which is a problem for firefox, which is now installed as a snap. This seems somewhat strange as firefox obviously needs network access, so it is not just the network access that causes problems.

Running firefox from the command line produces an error complaining that it doesn't have read access to the current working directory (which would be true if it were as a different user as a different user doesn't have access to the kerberos ticket).

[Andrew Conway (acubuntuone)]

I am pretty sure this is at least partly a problem with snaps not working with Kerberos, which is the authentication mechanism for NFS. The Kerberos credentials are (with good reason) not stored in the home directory.

I described this in more detail in bug 1784774.

This means that firefox and lxd don't work in 22.04 with authenticated NFS home directories.

[eoli3n (eoli3neoli3n)]

Same problem here, with 22.04 and firefox : https://forum.snapcraft.io/t/cannot-open-path-of-the-current-working-directory-permission-denied-bis/28704

[eoli3n (eoli3neoli3n)]

@acubuntuone we don't use kerberos authentication for NFS mounts here, and I face the same problem.

[Alberto Mardegan (mardy)]

We did some progress in investigating this, and it appears that a problem still exists: if the system is using autofs and snapd is started before any NFS home directory is mounted, snapd will fail to detect the fact that the system is using autofs (or NFS), and will not grant snap-confine those permissions needed to use the network.

Given that this bug is old and that indeed a (partial) fix landed, I suggest to keep it closed; I noticed that someone else filed a new bug which explicitly mentions the autofs case, so I suggest continuing the discussion there:

https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1884299