Snaps don't run with NFS home on AutoFS

[tylerecouture]

My physical computer lab uses AutoFS home drives (per https://help.ubuntu.com/community/Autofs#Wildcard_characters). If any user tries to run chromium browser, it fails.

I assume it is related to these:
https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1662552
But that says a fix was released, but seems toi only work for NFS home drives
https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1782873
That ones is reported as a dupe but it's not, AutoFS home drives still don't work.

$ chromium -v
cannot create user data directory: /home/test.student2/snap/chromium/1193: Stale file handle

$ tail -f /var/log/syslog
Jun 19 11:21:09 tbl-h10-4 kernel: [18949.188657] nfs: RPC call returned error 13
Jun 19 11:21:09 tbl-h10-4 kernel: [18949.188666] nfs: RPC call returned error 13
Jun 19 11:21:09 tbl-h10-4 kernel: [18949.188695] audit: type=1400 audit(1592590869.460:59): apparmor="DENIED" operation="sendmsg" profile="/snap/snapd/8140/usr/lib/snapd/snap-confine" pid=12884 comm="snap-confine" laddr=192.168.43.216 lport=766 faddr=192.168.43.4 fport=2049 family="inet" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send"
Jun 19 11:21:09 tbl-h10-4 kernel: [18949.188697] audit: type=1400 audit(1592590869.460:60): apparmor="DENIED" operation="sendmsg" profile="/snap/snapd/8140/usr/lib/snapd/snap-confine" pid=12884 comm="snap-confine" laddr=192.168.43.216 lport=766 faddr=192.168.43.4 fport=2049 family="inet" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send"

$ lsb_release -rd
Description: Ubuntu 20.04 LTS

$ apt policy chromium-browser
chromium-browser:
  Installed: 81.0.4044.129-0ubuntu0.20.04.1
  Candidate: 81.0.4044.129-0ubuntu0.20.04.1
  Version table:
 *** 81.0.4044.129-0ubuntu0.20.04.1 500
        500 http://ca.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages
        100 /var/lib/dpkg/status
     80.0.3987.163-0ubuntu1 500
        500 http://ca.archive.ubuntu.com/ubuntu focal/universe amd64 Packages

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Alberto Mardegan (mardy)]

A similar issue was discussed in the forums:

https://forum.snapcraft.io/t/cannot-open-path-of-the-current-working-directory-permission-denied-bis/28704

If the system is using autofs and snapd is started before any NFS home directory is mounted, snapd will fail to detect the fact that the system is using autofs (or NFS), and will not grant snap-confine those permissions needed to use the network.

[Andrew Conway (acubuntuone)]

Firefox also doesn't work now it is a snap in 22.04.

I think there are still multiple issues here. The original poster seems to be using NVSv3 I believe based on the RPC errors (NFSv3 uses multiple ports, one of which is called something like RPC, but I am not an expert in this as I have only used NFSv4, which uses a single port). Is this correct tylerecouture?

NFSv3 has no user authentication; NFSv4 uses Kerberos for authentication (and privacy and tamper resistance). I think this brings in additional problems as snaps don't appear to work with Kerberos

e.g.

https://bugs.launchpad.net/ubuntu/+source/chromium-browser/+bug/1849346

https://bugzilla.mozilla.org/show_bug.cgi?id=1734791

I get a very different error - a simple permission denied error rather than an AppArmor problem.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[tylerecouture (tylerecouture)]

I think I'm using 4, but I am set up using autofs and haven't touched NFS directly:

# /etc/auto.home
* -fstype=nfs,rw,async servername:/nfshome/&

`nfsstat -c` tells me: "Client nfs v4"

[Erik Meitner (eamuwmath)]

In our situation home folders are mounted using autofs using NFS v4, Kerberos is NOT in use.

root@jammy:~# lsb_release -rd
Description: Ubuntu 22.04 LTS
Release: 22.04

root@jammy:~# dpkg -l chromium-browser
--snip--
ii chromium-browser 1:85.0.4183.83-0ubuntu2 amd64 Transitional package - chromium-browser ->>
root@jammy:~# snap list |grep chromium
chromium 101.0.4951.64 1993 latest/stable canonical* -

Running chromium as user with NFS home folder at /staff/me:

me@jammy:~$ chromium
cannot open path of the current working directory: Permission denied

/var/log/syslog:
May 12 12:10:31 jammy systemd[15457]: Started snap.chromium.chromium.41586725-30fd-464e-9987-08936a8991eb.scope.
May 12 12:10:31 jammy kernel: [ 1496.959498] nfs: RPC call returned error 13
May 12 12:10:31 jammy kernel: [ 1496.959659] audit: type=1400 audit(1652375431.246:78): apparmor="DENIED" operation="sendmsg" profile="/usr/lib/snapd/snap-confine" pid=56203 comm="snap-confine" laddr=qq.ww.ee.rr lport=964 faddr=zz.xx.cc.vv fport=2049 family="inet" sock_type="stream" protocol=6 requested_mask="send" denied_mask="send"

root@jammy:~# cat /etc/apparmor.d/tunables/home.d/ubuntu
# This file is auto-generated. It is recommended you update it using:
# $ sudo dpkg-reconfigure apparmor
#
# The following is a space-separated list of where additional user home
# directories are stored, each must have a trailing '/'. Directories added
# here are appended to @{HOMEDIRS}. See tunables/home for details.
@{HOMEDIRS}+=/staff/ /fac/ /grad/ /visitor/

root@jammy:~# cat /etc/auto.staff
* -rw,nosuid nfshome.SSSSS.edu:/nfshome/staff/&

[Andrew Conway (acubuntuone)]

I (using Kerberos) don't the get apparmor DENIED messages that Eric (not using Kerberos) did, but I get exactly the same "cannot open path of the current working directory: Permission denied" error.

[Matthieu Herrb (matthieu-o)]

With Ubuntu 22 shipping firefox as a snap, it fails to start here :
"cannot open path of the current working directory: Permission denied"

[Matthieu Herrb (matthieu-o)]

In https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1662552 it is suggested that the problem is triggered by snapd starting too early (before autofs mounts any home directory) is there a way to force autofs to mount an home directory before launching snapd ?

[Andrew Conway (acubuntuone)]

Matthieu, more recently a more likely problem has been characterized by Alberto Mardegan and found the line in question in
https://bugs.launchpad.net/snapd/+bug/1973321

In particular, restarting snapd doesn't help at all for me, so having the directory mounted before snapd starts doesn't help, and the same problem occurs with other file systems. However _starting_ outside the home directory does help. This is the situation for me with Kerberos authenticated NFS, and others with sshfs. I don't know whether it is relevant for people using NFS without authentication. It is easy to test for yourself - restart snapd and see if that helps.

So I think there is progress in understanding the problem, even if not working out how to fix it.

FYI: I initially tried a work around where I used the debian repository for firefox instead of the snap version, but despite giving it a higher priority (confirmed via "sudo apt policy firefox") I found the debian package twice over a week uninstalled and replaced by the snap version that then would not start. I can manually revert it but it is inconvenient. Any suggestions on how to make that work reliably would be appreciated.

[Thomas Poulsen (thomas-lha66)]

I get the same error "cannot open path of the current working directory: Permission denied" when trying to open a file located on a filesystem which is mounted with sshfs.
This is the case for firefox and chromium-browser (both snaps), but not for google-chrome (not snap).
This is on 22.04. It used to work fine for firefox in 21.10, when firefox was not a snap.

[ThommiStephan (thommistephan)]

Similar error here.
Ubuntu 22.04 LTS, home-directories mounted via NFSv3 under /home/user/*

I am not able to start firefox. Error message:

tstephan@ifb2e5fe:~$ firefox
cannot open path of the current working directory: Permission denied

I included @{HOMEDIRS}+=/home/user/ in /etc/apparmor.d/tunables/home.d

[Oleg Davydov (burunduk3)]

I ran into a similar issue when trying to use sshfs-mounted profile for firefox.

My setup:

$ mkdir .remote
$ sshfs <remote> .remote -o idmap=user
$ ln -s .remote/snap

So now /home/ubuntu/snap is a symlink to a directory /home/ubuntu/.remote/snap. When trying to run firefox, it fails with:

$ firefox
cannot create user data directory: /home/ubuntu/snap/firefox/1232: Not a directory

Which is confusing, as /home/ubuntu/snap/firefox/1232 is a directory. Trying to strace what's going on (details in https://pastebin.com/gbeHV8L1) I found out that firefox (or snap) is confused that snap is a symlink to directory, not a directory by itself (and this limitation doesn't makes sense to me, looks like just implementation flaw of creating user directory).

Not sure if this is related to the above, but experience is similar.

[Alberto Mardegan (mardy)]

Добрый день, Олег! The bug you are hitting is actually not related to this one, and is bug 1621102. You can try with a bind-mount instead.

[Zirbo Zirboni (zirbozirboni)]

Hallo, I think I have the same issue.
Our NIS cluster works uses autofs, and in the nodes running ubuntu 22, snaps crash at start giving the error
`cannot open path of the current working directory: Permission denied`
Some users reported getting
`cannot create user data directory: /home/<username>/snap/firefox/1635: Stale file handle`
although I am not sure what they did.

At the moment our workaround is to run
`sudo systemctl restart snapd.service`
after logging in the first time. It seems the snaps do find the homes if snapd is started after autofs is already up and running. But for some weird reason, if a user logs out, and logs in again, snaps still work, but if ANOTHER user logs in, the new user has to restart snapd!