AppArmor profile prohibits classic snap from inheriting file descriptors

[Anders Kaseorg]

For example, with the ‘node’ classic snap:

$ touch /tmp/test.js
$ /snap/bin/node
Welcome to Node.js v12.13.0.
Type ".help" for more information.
> fd = fs.openSync("/tmp/test.js")
21
> child_process.execFileSync('/snap/bin/node', {stdio: [fd, 'inherit', 'inherit']})
events.js:187
      throw er; // Unhandled 'error' event
      ^

Error: EACCES: permission denied, read
Emitted 'error' event on ReadStream instance at:
    at internal/fs/streams.js:167:12
    at FSReqCallback.wrapper [as oncomplete] (fs.js:470:5) {
  errno: -13,
  code: 'EACCES',
  syscall: 'read'
}
Thrown:
Error: Command failed: /snap/bin/node
    at checkExecSyncError (child_process.js:621:11)
    at Object.execFileSync (child_process.js:639:15) {
  status: 1,
  signal: null,
  output: [ null, null, null ],
  pid: 30020,
  stdout: null,
  stderr: null
}
> .exit
$ dmesg
…
[69583.236304] audit: type=1400 audit(1571966467.652:672): apparmor="DENIED" operation="file_inherit" profile="/snap/snapd/4992/usr/lib/snapd/snap-confine" name="/tmp/test.js" pid=30020 comm="snap-confine" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000

This breaks all sorts of things. I ran into this when trying to use prettier-emacs with the ‘emacs’ and ‘node’ classic snaps.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: snapd 2.41+19.10.1
ProcVersionSignature: Ubuntu 5.3.0-19.20-lowlatency 5.3.1
Uname: Linux 5.3.0-19-lowlatency x86_64
NonfreeKernelModules: openafs
ApportVersion: 2.20.11-0ubuntu9
Architecture: amd64
CurrentDesktop: GNOME
Date: Thu Oct 24 18:07:19 2019
EcryptfsInUse: Yes
InstallationDate: Installed on 2016-02-19 (1343 days ago)
InstallationMedia: Ubuntu-GNOME 16.04 LTS "Xenial Xerus" - Alpha amd64 (20160218)
SourcePackage: snapd
UpgradeStatus: Upgraded to focal on 2019-06-23 (123 days ago)

[Zygmunt Krynicki (zyga)]

So this is an existing issue that we sometimes tried to work around by granting snap-confine more permissions. This is a limitation in apparmor itself, where we cannot say that snap-confine can inherit and pass a file descriptor to another process, whatever that file may be.

I had a quick look if that workaround handles /tmp/* but I couldn't see anything. Perhaps it needs to be added but I'd like to write a test first.

[Ian Johnson (anonymouse67)]

Note that this has been reported on the forum before at
https://forum.snapcraft.io/t/classic-confinement-on-atom-yet-still-getting-denials/4956
and
https://forum.snapcraft.io/t/snapd-2-32-breaks-live-server-installer/4597

[Jamie Strandboge (jdstrand)]

As Zygmunt said, this is a current limitation with apparmor. The problem is because both node and snap-confine are differently confined by apparmor, there is a revalidation that happens when node calls itself since it invokes snap run, which invokes snap-confine which causes the revalidation (because it is differently confined). FD delegation improvements in apparmor are planned that would allow for a better experience.

A workaround could be to avoid snap run and call node from /snap/node/current/... instead of through /snap/bin/node.

[Anders Kaseorg (andersk)]

Marking confirmed based on the forum reports.

To be clear, I’m not using ‘snap run’, just the ‘node’ that snap has put in the PATH, which is /snap/bin/node (a symlink to /usr/bin/snap). Lots of applications expect to be able to run ‘node’ from the PATH, including the ‘node’ snap’s own ‘npm’, ‘npx’, ‘yarn’, and ‘yarnpkg’ scripts.

If /snap/node/current/bin/node is expected to work better, then maybe it’s reasonable to ask why /snap/bin/node goes through /usr/bin/snap at all for a classic snap? Could snap just set it up as a direct symlink to /snap/node/current/bin/node and avoid this problem?

[John Johansen (jjohansen)]

This is traditional MAC behavior and is by design. Uncontrolled inheritance is an information leak/security hole.

The delegation extension that @jdstrand mentioned is an extension that crosses capability systems with a type enforcement system.

Marking this wishlist as it is feature development that is planned for but not yet resourced.

[Anders Kaseorg (andersk)]

John, did you read more than three words of the report? We’re talking about a classic snap inheriting an fd from a classic snap (the same classic snap, in fact) for a file to which they should both have access (because they’re classic snaps). There can’t be information leaking across a security boundary when there’s no security boundary. And if there were a security boundary, it sure wouldn’t be a very good one if you could get around it by invoking the binary at a different path.

[John Johansen (jjohansen)]

Yes I did, and @jdstrand did explain the situation in #4

"There is a revalidation that happens when node calls itself since it invokes snap run, which invokes snap-confine which causes the revalidation (because it is differently confined)."

So there is a security boundary being crossed.

[Anders Kaseorg (andersk)]

Alright, so, when I asked in #5 whether symlinks for classic snaps in /snap/bin could point directly do the target binary instead of indirecting through snap and snap-confine, am I at least asking a legitimate question? Surely making a handful of symlink changes would be easier than developing a delegation extension for AppArmor.

[John Johansen (jjohansen)]

I am not familiar enough with the specifics of how snappy is setting policy to be able to answer your question atm. Whether it is possible will depend on policy.

AppArmor mediation is post symlink so the policy would have to allow access to the target binary.

[Jamie Strandboge (jdstrand)]

> To be clear, I’m not using ‘snap run’, just the ‘node’ that snap has put in the PATH, which is /snap/bin/node (a symlink to /usr/bin/snap). Lots of applications expect to be able to run ‘node’ from the PATH, including the ‘node’ snap’s own ‘npm’, ‘npx’, ‘yarn’, and ‘yarnpkg’ scripts.

Sure, node isn't calling snap run directly, but the act of calling /snap/bin/node (which is a symlink to /usr/bin/snap), correctly makes snap invoke the launcher via snap run.

> If /snap/node/current/bin/node is expected to work better, then maybe it’s reasonable to ask why /snap/bin/node goes through /usr/bin/snap at all for a classic snap? Could snap just set it up as a direct symlink to /snap/node/current/bin/node and avoid this problem?

We don't want snapd to do that since that would bypass the launcher entirely and even for classic snaps, we want the launcher to setup the environment and setup the process to be trackable by the system in various ways. Snaps calling their own binaries from $SNAP is the currently best supported method until we have the updates John mentioned.

[Jamie Strandboge (jdstrand)]

John, I know there are plans for FD delegation and properly mediating this but I wonder if there is any use for a 'file_inherit' rule that is perhaps just very coarse and would allow inheriting the fd. It does seem like this could provide a means of sandbox escape though since a(n unprivileged) process could open something, then launch the (in this case, setuid) confined executable and snap-confine would have access to it. For the case of snap-confine, we only really need for snap-confine to pass through the fd to what it launches, not actually be able to use it....

[Jamie Strandboge (jdstrand)]

Since the issue is that an fd is opened by the first app running in one profile while transitioning to the snap-confine profile, there is an option that would 'work'.

As a POC, I installed the hello-world snap and also created a test-classic snap (just hello-world renamed with 'confinement: classic' and installed with --classic --dangerous). Do nothing else, I then try to reproduce the issue:

$ test-classic.sh
bash-5.0$ exec 3<> /run/user/$(id -u)/test.fd
bash-5.0$ test-classic.env > /dev/null
bash-5.0$

I see in the logs the familiar denial:

Oct 30 13:53:24 foo kernel: audit: type=1400 audit(1572461604.648:3444): apparmor="DENIED" operation="file_inherit" profile="/snap/core/8039/usr/lib/snapd/snap-confine" name="/run/user/1000/test.fd" pid=24957 comm="snap-confine" requested_mask="wr" denied_mask="wr" fsuid=1000 ouid=1000

I then updated /var/lib/snapd/apparmor/profiles/snap.test-classic.sh to have:

  /usr/lib/snapd/snap-confine ix,
  /snap/core/8039/usr/lib/snapd/snap-confine ix,
  ^mount-namespace-capture-helper (complain) {
    file,
    unix,
    signal,
  }

and tried again:

$ test-classic.sh
bash-5.0$ exec 3<> /run/user/$(id -u)/test.fd
bash-5.0$ test-classic.env > /dev/null
bash-5.0$

and I observe in the logs there is no file_inherit denial.

This 'works' because the profile that snap-confine is running under is the same as the classic snap and therefore has all the same accesses that the snap does (I could've chosen the special 'unconfined', but snap-confine will fail to run in that case).

Interestingly, if I run 'hello-world' from the classic snap:

$ test-classic.sh
bash-5.0$ exec 3<> /run/user/$(id -u)/test.fd
bash-5.0$ test-classic.env > /dev/null
bash-5.0$ hello-world.sh
bash-4.3$ cat /proc/self/fd/3
cat: /proc/self/fd/3: Permission denied

hello-world correctly gets the denials (first is inherit, 2nd /apparmor/.null is how apparmor handles the access to the failed inherit fd):

Oct 30 14:20:44 foo kernel: audit: type=1400 audit(1572463244.359:3449): apparmor="DENIED" operation="file_inherit" profile="snap.hello-world.sh" name="/run/user/1000/test.fd" pid=26175 comm="snap-exec" requested_mask="wr" denied_mask="wr" fsuid=1000 ouid=1000
Oct 30 14:21:56 foo kernel: audit: type=1400 audit(1572463316.344:3451): apparmor="DENIED" operation="open" profile="snap.hello-world.sh" name="/apparmor/.null" pid=26244 comm="cat" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

Now, I say 'works' because I don't care for how the snap-confine policy is circumvented in the POC since a classic snap could then try to exploit bugs in the setuid snap-confine. While one could argue that a classic snap already has root on the system, many people will install classic snaps that run as the user (ie, no daemons) and feel a bit safer, but with the POC policy the snap could, running as the user, try to exploit bugs in snap-confine to gain privileges.

There is possibly an acceptable way, but it would need to be investigated to verify it works and for acceptable safety:

1. adjust the classic policy to use:

  /usr/lib/snapd/snap-confine ix,
  /snap/$SNAP_WITH_SNAPD/$SNAP_WITH_SNAPD_REVISION/usr/lib/snapd/snap-confine ix,

2. adjust snap-confine ...

[Jamie Strandboge (jdstrand)]

"
1. adjust the classic policy to use:

  /usr/lib/snapd/snap-confine ix,
  /snap/$SNAP_WITH_SNAPD/$SNAP_WITH_SNAPD_REVISION/usr/lib/snapd/snap-confine ix,
"

This should have been:

1. adjust the classic policy to use:

  /usr/lib/snapd/snap-confine px -> unconfined,
  /snap/$SNAP_WITH_SNAPD/$SNAP_WITH_SNAPD_REVISION/usr/lib/snapd/snap-confine px -> unconfined,

[Michał Sawicz (saviq)]

I just had a similar problem trying to call a confined app from a classic one in Python:

subprocess.run("command", stdin=open("/file/path", "rb"))

The way around it is either reading the file into memory and feeding it in:

subprocess.run("command", input=open("/file/path", "rb").read())

Or looping over the file and writing to the process's stdin:

with subprocess.Popen("command", stdin=subprocess.PIPE) as p, \
     open("/file/path", "rb") as f:
    while True:
        buf = f.read(8192)
        if buf:
            p.stdin.write(buf)
        else:
            break

[Alex Murray (alexmurray)]

> An alternative without modifying snap-confine would be to have two snap-confine profiles, one for
> strict and one for classic, and adjust the classic template to transition to the classic
> snap-confine template which has rules allowing 'rw' access to files and 'unix' for sockets.

To me this sounds like the easier of the two approaches - can you outline any particular (dis)advantages to either approach?

[Jamie Strandboge (jdstrand)]

>> An alternative without modifying snap-confine would be to have two snap-confine profiles, one for
>> strict and one for classic, and adjust the classic template to transition to the classic
>> snap-confine template which has rules allowing 'rw' access to files and 'unix' for sockets.
>
> To me this sounds like the easier of the two approaches - can you outline any particular
> (dis)advantages to either approach?

A pro for modifying snap confine means that it can still run under a restricted profile after it performs aa_change_profile(). The con is there is a short period when it isn't running under confinement. Without the investigation, it is not clear to me that the modification will achieve the objective of avoiding the file_inherit. The code changes to snap-confine would not be significant.

A pro for using a different template is that it requires no code changes and is less complex. A con is that the snap-confine policy is weakened when a classic snap executes another snap. In addition to a lessening of hardening around snap-confine, it is more difficult to reason about the confinement of snap-confine.

Considering that snap-confine is designed to be secure and not dependent on apparmor for its security, it is probably best to use the two-profile approach and make no code changes. This introduces no changes on systems with no classic snaps installed. While this will weaken the hardening of the safety net, there is nothing saying that we have to have a totally wide open policy for the snap-confine under classic. Eg, while we might have a wide 'rw' rules, we don't need 'x' or 'm' rules and we can use explicit deny rules for apparmor policy, libraries snap-confine uses, etc.

[John Johansen (jjohansen)]

In response to Jamie's question in #12 the no answer is no. Delegation works because it allows a subject with explicit access to an object to delegate that access to another. An important part of delegation is that it is not just delegating the object but inheritance and passing of the object is controlled beyond the initial passage of the object.

One of the problems with most traditional capability systems is they don't correctly allow control of the inherited object which has proved to be problematic and also does not map well back to a type system.

Allowing for an fd_inherit rule breaks the inheritance control in apparmor's delegation model.

[John Johansen (jjohansen)]

I should note that this only requires object delegation in apparmor, which is a subset of the full delegation work and will land first.

[Ian Johnson (anonymouse67)]

There is a RFC PR from Jamie up here: https://github.com/snapcore/snapd/pull/10029. We (snapd team) will try to pick this up and get it merged when we have some time (hopefully soon).

[fcole90 (fcole90)]

Possibly related issue: https://github.com/nodejs/node/issues/37982

[Jamie Strandboge (jdstrand)]

FYI, if people need to workaround this to get real work done, you can add something like this to your bashrc:

snap_workaround() {
    fn="/var/lib/snapd/apparmor/snap-confine/lp1849753"
    test -e "$fn" && return

    tmpfn=$(mktemp)
    cat > "$tmpfn" <<EOM
# lp1849753
unix,
owner /** rw,
ptrace readby peer=unconfined,
EOM
    echo "Moving workaround rules to '$fn'"
    sudo mv "$tmpfn" "$fn"
    echo "Reloading snap-confine policy"
    sudo apparmor_parser -r /etc/apparmor.d/*snap-confine* /var/lib/snapd/apparmor/profiles/snap-confine.*
    # another mysterious issue where sometimes all the policy isn't loaded (eg, with rustup)
    echo "Reloading all snap.* policy (work around missing profiles)"
    sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.*
    sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap-update-ns.*
}

[fcole90 (fcole90)]

Thanks for the workaround!

Can you explain in brief how's working? Is it like temporarily transferring the ownership of the process?

[John Johansen (jjohansen)]

It is changing a section (the file /var/lib/snapd/apparmor/snap-confine/lp1849753) used by the snap apparmor profiles and then reloading apparmor profiles into the kernel. This does a live replacement of policy, so processes that are already confined will gain the new permissions as well as new processes. There is No transfer of ownership.

[jehon (jeanhonlet)]

The above script of @jdstrand (comment #22) could be automated by a service.

The basic is to create the file just after snapd.service has deleted it.

Create the service in /usr/share/jehon/etc/systemd/system/lp1849753.service:

----------------------------
[Unit]
Description=Fix the App Armor profile for nodejs
Documentation=https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1849753/comments/22
Requires=snapd.service
After=snapd.service snapd.apparmor.service
PartOf=snapd.service

[Service]
Type=oneshot
ExecStart=<path-to-the-above-script>
RemainAfterExit=yes

[Install]
WantedBy=default.target

----------------------------
I am not 100% sure it does work in all cases, but worth a try

[Stanislav German-Evtushenko (giner)]

Here is a workaround for node (tested on Ubuntu 22.04):

for cmd in node npm yarn; do
mkdir -p ~/bin
cat > "$HOME/bin/$cmd" << EOF
#!/bin/bash

/snap/bin/$cmd "\$@" > >(cat) 2> >(cat >&2)
EOF
chmod +x "$HOME/bin/$cmd"
done

[Jesse Glick (jesse-glick)]

I suspect this is also the cause of a problem affecting Krew plugins when using the kubectl snap (reported in https://github.com/corneliusweig/konfig/issues/14). To reproduce with the `kubectl` snap version 1.26.2 installed in classic confinement, create a file in `$PATH` named `kubectl-xxx`

---%<---
#!/usr/bin/env bash
set -ex
tmp=$(mktemp XXXXXX)
# Works:
echo hello >$tmp
cat $tmp
# Broken:
kubectl config view >$tmp
cat $tmp
--->%---

and try to use it:

---%<---
$ kubectl xxx
++ mktemp XXXXXX
+ tmp=rpbRXH
+ echo hello
+ cat rpbRXH
hello
+ kubectl config view
error: write /dev/stdout: permission denied
--->%---

[Thomas Güttler (thomas-guettler)]

I installed yq and kubectl via brew for linux, and now I don't get the permission denied error any more.

Related: https://stackoverflow.com/questions/75659711/yq-fails-with-permission-denied/76002505#76002505