snaps don't start when current working directory is on a remote FS (sshfs, NFS)

[Alberto Mardegan]

All snaps fail to start when the current working directory is under a sshfs tree:

    /tmp/test$ hello-world
    cannot open path of the current working directory: Permission denied

The reason is that sshfs by default disallows the root user (or any user other than the one who created the mount) from accessing the mounted file system, and snap-confine is a setuid program which before dropping its privileges tries to open the current working directory and terminates if that operation fails:

    In sc_preserve_and_sanitize_process_state():

 proc_state->orig_cwd_fd =
     openat(AT_FDCWD, ".",
     O_PATH | O_DIRECTORY | O_CLOEXEC | O_NOFOLLOW);
 if (proc_state->orig_cwd_fd < 0) {
  die("cannot open path of the current working directory");
 }

There are workarounds for this situations: one can add the "user_allow_other" option to /etc/fuse.conf and call sshfs with the "-o allow_root" option, and this will allow snaps to start, yet it's a suboptimal experience.

Possible solutions:
A) Move this part of code a bit later, after having dropping the privileges.
B) Drop the privileges before making this call, and restore them afterwards.
C) Use get_current_dir_name() to get the directory name if openat() fails, then restore it using chdir().

[Alberto Mardegan (mardy)]

Update: none of the solutiones mentioned above work, unfortunately.

The problem is that the FUSE kernel module not only checks the effective user ID, but also the real and saved one:

https://github.com/torvalds/linux/blob/2bf06b8e64280251775011f63d44e7bfc48dbdfd/fs/fuse/dir.c#L1223-L1240

As a privileged process, we can set all of the user and group IDs to arbitrary values, and can therefore pass this FUSE check; but once we do, there's no way to get back to being a privileged process, and therefore this is one of the last operations snap-confine performs before executing the desired program.

One option that we might want to investigate is setting all the needed capabilities on our process at startup (while we are still root), set the SECBIT_KEEP_CAPS bit, and then drop all our user IDs to the user ones. As long as we have all the capabilities set, we should be able to perform all the operations we need to; and then capabilities would be normally lost when we call execve().

[Andrew Conway (acubuntuone)]

In bug https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1784774

Alberto pointed out that the same cause could make snaps fail when starting under a NFS served/Kerberos authenticated home directory, which, like sshfs, has the property that root cannot access data in it.

I have confirmed this.

[Alberto Mardegan (mardy)]

Thanks for confirming this, Andrew!

Can you please tell me, if you implemented your NFS setup based on some default/recommended configuration, or if blocking access to the root user is something that you explicitly activated?

I'm asking this to understand how big the impact of this bug is. Is it possible to disable this restriction and let the root user access the NFS directories?

[Andrew Conway (acubuntuone)]

The setup where root cannot access is the default for NFS (unless you want to run it with no authentication at all).

I have not seen or heard any options for making root able to access Kerberos/nfs. They may exist, but it is not an easy thing to do - the root user has to authenticate itself to the user server as a particular user. It is not obvious which user it should choose. Well, you could come up with a set of heuristics that would often but not always work, and then see if that user has any credentials lying around, and grab them. Alternatively you could make a root user in kerberos (the reason for sudo's existance is because this is a bad idea), and make a file containing the root password in plaintext in a standard place that Kerberos could access if it wants to do something as root, but this seems like a very bad idea.

P.S. It turns out that the workaround of using the debian firefox packages doesn't work reliably - ubuntu 22.04 keeps regularly reinstalling firefox as a snap even if you explicitly give the debian package higher priority.