AppArmor denies crun sending signals to containers (stop, kill)

[Martin Pitt]

[ Impact ]

 * On mantic and noble, when run as root, podman cannot stop any container running in background because crun is being run with a new profile introduced in AppArmor v4.0.0 that doesn't have corresponding signal receive rule container's profile.

 * Without the fix, users would have to resort to figuring out container's PID 1 and killing it as root or by other privileged and unconfined process. This is a regression from basic podman functionality.

 * The fix adds signal receive rules for currently confined OCI runtimes in AppArmor v4.0.0 (runc and crun) to the profile used by podman.

[ Test Plan ]

All commands must be invoked as root.

Run tests below with both crun and runc OCI runtimes. For crun, nothing has to be changed (it's installed and used by default). For runc, first install the runc pakcage, and then insert "--runtime /usr/sbin/runc" arguments after "podman run".

Start container in background and then stop it:

  # Run container in background (-d)
  podman run -d --name foo docker.io/library/nginx:latest
  # Stop the container
  podman stop foo

On success, the last command should print the container name and the container running in background should be stopped (verify with "podman ps").

Additional tests:

Verify that container running in foreground TTY can be stopped.

  # Terminal 1:
  # Run container on this TTY
  podman run -it --name bar --rm docker.io/library/ubuntu:22.04

  # Terminal 2:
  # Stop the container
  podman stop bar

On success, the last command should print the container name, the process running in terminal 1 should stop, and the container should be removed (verify with "podman ps -a").

Verify that container running with dumb init can be killed.

  # Run container in background (-d) with dumb init
  podman run -d --name bar --rm --init ubuntu:22.04 sleep infinity
  # Stop the container
  podman stop bar

On success, the last command should print the container name and the container running in background should be stopped and removed (verify with "podman ps -a").

Verify container processes can signal each other

  # Run container in foreground with processes sending signals between themselves
  podman run ubuntu:22.04 sh -c 'sleep inf & sleep 1 ; kill $!'

On success, the last command should exit after cca 1 second with exit status 0.

[ Where problems could occur ]

 * The fix requires a rebuild of podman that will pull in any other changes in the archive since the last build, which could potentially break some functionality.

[ Original report ]

Mantic's system podman containers are completely broken due to bug 2040082. However, after fixing that (rebuilding with the patch, or a *shht don't try this at home* hack [1]), the AppArmor policy still causes bugs:

  podman run -it --rm docker.io/busybox

Then

  podman stop -l

fails with

   2023-10-25T11:06:33.873998Z: send signal to pidfd: Permission denied

and journal shows

  audit: type=1400 audit(1698231993.870:92): apparmor="DENIED" operation="signal" class="signal" profile="containers-default-0.50.1" pid=4713 comm="3" requested_mask="receive" denied_mask="receive" signal=term peer="/usr/bin/crun"

This leaves the container in a broken state:

  # podman ps -a
  CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
  61749260f9c4 docker.io/library/busybox:latest sh 40 seconds ago Exited (-1) 29 seconds ago confident_bouman

  # podman rm --all
  2023-10-25T11:07:21.428701Z: send signal to pidfd: Permission denied
  Error: cleaning up container 61749260f9c4c96a51dc27fdd9cb8a86d80e4f2aa14eb7ed5b271791ff8008ae: removing container 61749260f9c4c96a51dc27fdd9cb8a86d80e4f2aa14eb7ed5b271791ff8008ae from runtime: `/usr/bin/crun delete --force 61749260f9c4c96a51dc27fdd9cb8a86d80e4f2aa14eb7ed5b271791ff8008ae` failed: exit status 1

  audit: type=1400 audit(1698232041.422:93): apparmor="DENIED" operation="signal" class="signal" profile="containers-default-0.50.1" pid=4839 comm="3" requested_mask="receive" denied_mask="receive" signal=kill peer="/usr/bin/crun"

[1] sed -i 's/~alpha2/0000000/' /usr/sbin/apparmor_parser

Ubuntu 23.10

ii apparmor 4.0.0~alpha2-0ubuntu5 amd64 user-space parser utility for AppArmor
ii golang-github-containers-common 0.50.1+ds1-4 all Common files for github.com/containers repositories
ii podman 4.3.1+ds1-8 amd64 engine to run OCI-based containers in Pods

[Martin Pitt (pitti)]

FTR, after calling `aa-teardown` the stopping and removing works.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Martin Pitt (pitti)]

Confirmed on current noble, even with the -proposed build https://launchpad.net/ubuntu/+source/libpod/4.7.2+ds1-2build1/+build/27032773

[Martin Pitt (pitti)]

I tried a more targeted workaround, with

  aa-complain /etc/apparmor.d/usr.bin.crun

or alternatively (without apparmor-utils, which isn't on the default cloud image):

  sed -i '/flags=/ s/unconfined/complain/' /etc/apparmor.d/usr.bin.crun

but for some reason that breaks podman entirely:

# podman run -it --rm docker.io/busybox
Failed to re-execute libcrun via memory file descriptor
                                                       ERRO[0000] Removing container 7c3c938f8e356a9834de6a114ad8b8353ffac7508c8aac131d588e1358ba2f30 from runtime after creation failed
Error: OCI runtime error: crun: Failed to re-execute libcrun via memory file descriptor

I just noticed that neither podman nor crun ship their own AppArmor profiles, /etc/apparmor.d/usr.bin.crun is shipped by apparmor. So adding a package task, but leaving libpod as "affected", so that it is easier to find.

[Martin Pitt (pitti)]

I also tried

  aa-disable usr.bin.crun

but that doesn't work either. I guess it's not really crun, but profile="containers-default-0.50.1", but that is created dynamically -- it's not anywhere in /etc/apparmor.d/. I grepped the whole file system for that:

  grep: /usr/lib/podman/rootlessport: binary file matches
  grep: /usr/bin/podman: binary file matches
  grep: /usr/bin/buildah: binary file matches

Running an individual container with --security-opt=label=disable also doesn't work, same DENIED and failure.

"man containers.conf" points at apparmor_profile="container‐default", but not how to disable it. I naively tried apparmor_profile="none" but

  Error: AppArmor profile "none" specified but not loaded

But curiously an empty string works. 🎉 So, my official workaround:

  mkdir -p /etc/containers/containers.conf.d
  printf '[CONTAINERS]\napparmor_profile=""\n' > /etc/containers/containers.conf.d/disable-apparmor.conf

[Georgia Garcia (georgiag)]

According to the AppArmor policy [1], the following rule is allowed

signal (receive) peer=unconfined,

And when there was no policy for /usr/bin/crun, the signal that is now being denied would fall under this rule, because crun was unconfined.
A profile for crun was added in Bug 2035315 because applications that make use of unprivileged user namespaces must be confined by an AppArmor profile, so to properly fix this bug the following rule must be added

signal (receive) peer=/usr/bin/crun,

or better yet, because of the AppArmor upstream commit that renames the profile [2]

signal (receive) peer={/usr/bin/,}crun,

[1] https://github.com/containers/common/blob/main/pkg/apparmor/apparmor_linux_template.go#L23C36-L23C36
[2] https://gitlab.com/apparmor/apparmor/-/blob/2594d936ada5df797bc69e78a2ef8c6e6171d454/profiles/apparmor.d/crun

[André Oliveira (oribunokiyuusou)]

I too ran into this issue, but `aa-disable usr.bin.crun` was enough to sort it out.

EDIT: aa-disable allows me to run the crun command manually, but when podman runs it it still fails.

[André Oliveira (oribunokiyuusou)]

This is definitely happening on Mantic, as that's what I'm running. Would be a shame to see this make it's way into Noble as podman gradually replaces docker.

[Tomáš Virtus (virtustom)]

There's a similar issue with runc (and containerd and docker) reported here https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/2039294

I've opened PRs with a fix upstream:
- https://github.com/containerd/containerd/pull/10123
- https://github.com/moby/moby/pull/47749

I think I'll need to work a little bit more on them to dynamically add rules only for profiles that exist on the system, even though it works even if they don't exist. Is this a proper way to fix it? I have gained all my experience with AppArmor in last 2 days.

For podman a similar change should be applied to the profile template defined here https://github.com/containers/common/blob/main/pkg/apparmor/apparmor_linux_template.go. I can do that later.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Lucas Kanashiro (lucaskanashiro)]

Hi Tomáš,

Thanks for investigating this issue and providing the patch (MP) to fix it in Noble. However, before fixing it in Noble, we need to fix it in Oracular (development release). Would you like to provide a patch or MP targeting Oracular?

[Lucas Kanashiro (lucaskanashiro)]

I also see that you are patching golang-github-containers-common. Does that mean that no patch in libpod is needed? If the answer is yes, we need to mark the libpod tasks as Invalid.

[Neil Wilson (neil-aldur)]

To move this on a bit more rapidly as it is a blocking issue for me.

It's the same version in Oracular at present. I've pushed the changes as an MP against ubuntu/devel.

What needs to happen next?

[Neil Wilson (neil-aldur)]

The patch above doesn't work as it stands. We are still getting signal filters in the audit log

May 14 11:13:06 srv-omzr6 kernel: audit: type=1400 audit(1715685186.296:112): apparmor="DENIED" operation="signal" class="signal" profile="containers-default-0.57.4" pid=8031 comm="3" requested_mask="receive" denied_mask="receive" signal=term peer="crun"
May 14 11:13:06 srv-omzr6 kernel: audit: type=1400 audit(1715685186.318:113): apparmor="DENIED" operation="signal" class="signal" profile="containers-default-0.57.4" pid=8033 comm="3" requested_mask="receive" denied_mask="receive" signal=term peer="crun"
May 14 11:13:16 srv-omzr6 kernel: audit: type=1400 audit(1715685196.340:114): apparmor="DENIED" operation="signal" class="signal" profile="containers-default-0.57.4" pid=8035 comm="3" requested_mask="receive" denied_mask="receive" signal=kill peer="crun"
May 14 11:13:21 srv-omzr6 kernel: audit: type=1400 audit(1715685201.413:115): apparmor="DENIED" operation="signal" class="signal" profile="containers-default-0.57.4" pid=7664 comm="conmon" requested_mask="receive" denied_mask="receive" signal=term peer="podman"
May 14 11:14:31 srv-omzr6 kernel: audit: type=1400 audit(1715685271.577:116): apparmor="DENIED" operation="signal" class="signal" profile="containers-default-0.57.4" pid=8049 comm="3" requested_mask="receive" denied_mask="receive" signal=term peer="crun"
May 14 11:14:36 srv-omzr6 kernel: audit: type=1400 audit(1715685276.326:117): apparmor="DENIED" operation="signal" class="signal" profile="containers-default-0.57.4" pid=8052 comm="3" requested_mask="receive" denied_mask="receive" signal=kill peer="crun"
May 14 11:14:41 srv-omzr6 kernel: audit: type=1400 audit(1715685281.392:118): apparmor="DENIED" operation="signal" class="signal" profile="containers-default-0.57.4" pid=7458 comm="conmon" requested_mask="receive" denied_mask="receive" signal=term peer="podman"
May 14 11:14:41 srv-omzr6 kernel: audit: type=1400 audit(1715685281.604:119): apparmor="DENIED" operation="signal" class="signal" profile="containers-default-0.57.4" pid=8055 comm="3" requested_mask="receive" denied_mask="receive" signal=kill peer="crun"

[Tomáš Virtus (virtustom)]

@lucaskanashiro: This patch is for golang-github-containers-common source package. This source package produces golang-github-containers-common-dev binary package, which is just source code on filesystem. But podman binary package, which is produced from libpod source package, has golang-github-containers-common-dev in its Build-Depends. It doesn't depend on golang-github-containers-common-dev during runtime. So when golang-github-containers-common is updated, libpod should be rebuilt. Should I keep libpod as confirmed, and make a patch against libpod to make a rebuild?

@neil-aldur: I'll try to reproduce.

[Neil Wilson (neil-aldur)]

The debdiff I've put together for oracular updates the patch to be a bit more general and cover all the signals I've seen so far in testing. (As well as dropping the other patch that has been incorporated upstream).

  # Allow certain signals from OCI runtimes (podman, runc and crun)
    signal (receive) set=(int, quit, kill, term) peer={/usr/bin/,/usr/sbin/,}runc,
    signal (receive) set=(int, quit, kill, term) peer={/usr/bin/,/usr/sbin/,}crun,
    signal (receive) set=(int, quit, kill, term) peer={/usr/bin/,/usr/sbin/,}podman,

Upstream have said they have no apparmor experience, so I suspect they will take a PR. See https://github.com/containers/common/issues/1898

[Neil Wilson (neil-aldur)]

I've built a backported 4.9.4 libpod for noble based on an updated golang-github-containers-common including the above patch.

It's available from ppa:brightbox/experimental

[Neil Wilson (neil-aldur)]

Adding the podman signal line, and building a libpod that overrides the default packages eliminates the errors I was getting.

All the tests in this ticket pass with the updated packages.

[Tomáš Virtus (virtustom)]

@neil-aldur, did you forget to attach the debdiff?

By restricting the signal set you also restrict what $SIG you can put to "podman kill --signal $SIG".

I did not realize that there's a podman reference profile as well, but since podman doesn't try to kill the container by itself, I wonder if it makes sense to arbitrarily open a policy like this.

Also, whether you changes are good or not, they diverge from the policy changes we have already merged to containerd and moby upstream. Not sure if that's a problem.

Regarding your changes to the changelog entry in your MP: I based my entry on comment on a code comment from ahasenack (https://code.launchpad.net/~fun2program8/ubuntu/+source/crun/+git/crun/+merge/464233, you have to select b879 commit, it's the first code comment). I don't think we should copy the commit message into changelog entries. It's already in the patch.

[Tomáš Virtus (virtustom)]

Also, thanks for linking the podman issue. I'll try to merge patch upstream similar to moby and containerd.

[Neil Wilson (neil-aldur)]

The debdiff is in the MP above.

Podman does try to kill the container itself, as the error trace above testifies.

May 14 11:14:41 srv-omzr6 kernel: audit: type=1400 audit(1715685281.392:118): apparmor="DENIED" operation="signal" class="signal" profile="containers-default-0.57.4" pid=7458 comm="conmon" requested_mask="receive" denied_mask="receive" signal=term peer="podman"

It's trying to kill conmon in some scenarios, which means your policy changes so far are deficient in that regard. We can tighten the signal set there to term and kill, which is certainly no worse than the pre-4.0.0 situation.

I note the point about the signal set on the runtimes, and that should be removed. The stop signals can be set to anything within the container.

I would suggest extending the AARE to cover the binaries as well as the policy name.

[Tomáš Virtus (virtustom)]

Sorry, I missed the conmon-podman denial. Would you mind making a PR to the upstream with your changes with issue you posted linked? I think Lucas will not have time until end of week.

[Neil Wilson (neil-aldur)]

I've pushed the changes based on your comments to the MP above. I've left the signal set for podman as (int, quit, term, kill).

Do you think that signal set should be tighter, or is that a good compromise?

If that seems ok with you, I'll happily handle the PR upstream at GitHub.