apparmor docker

[adam furtenbach]

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 23.10
Release: 23.10
Codename: mantic

Docker version 24.0.5, build 24.0.5-0ubuntu1

Graceful shutdown doesn't work anymore due to SIGTERM and SIGKILL (maybe all signals?) doesn't reach the target process. Works when apparmor is uninstalled.

[17990.085295] audit: type=1400 audit(1697213244.019:981): apparmor="DENIED" operation="signal" class="signal" profile="docker-default" pid=172626 comm="runc" requested_mask="receive" denied_mask="receive" signal=term peer="/usr/sbin/runc"
[17992.112517] audit: type=1400 audit(1697213246.043:982): apparmor="DENIED" operation="signal" class="signal" profile="docker-default" pid=172633 comm="runc" requested_mask="receive" denied_mask="receive" signal=kill peer="/usr/sbin/runc"

[adam furtenbach (zmoosh)]

Applies to all signals.

# docker kill -s sigusr1 dynamodb-local
Error response from daemon: Cannot kill container: dynamodb-local: Cannot kill container fe323ad3ca9648f2e8b59debd22a2439f4709c5fafe3dbf46a0a06f67ba65204: unknown error after kill: runc did not terminate successfully: exit status 1: unable to signal init: permission denied
: unknown

[33054.783037] audit: type=1400 audit(1697228308.520:1037): apparmor="DENIED" operation="signal" class="signal" profile="docker-default" pid=189468 comm="runc" requested_mask="receive" denied_mask="receive" signal=usr1 peer="/usr/sbin/runc"

[Seth Arnold (seth-arnold)]

Are you perhaps mixing Docker packages from one source with Docker AppArmor profiles from another source?

AppArmor policy around signals is a bit more involved than around files:

- The sending process must have permission to send the signal to the recipient
- The receiving process must have permission to receive the signal from the sender

Make sure both your docker-default profile and your /usr/sbin/runc profile have the necessary permissions.

Thanks

[adam furtenbach (zmoosh)]

Hi,

Just did a fresh minimal install of ubuntu-23.10.1-desktop-amd64.iso

apt install -y docker.io
<add user to group>
<restart>
docker run -d --name=nginx nginx
docker kill -s sigint nginx

Error response from daemon: Cannot kill container: nginx: Cannot kill container 3590b8a55fa29e5df34b1ad7444100652ba9912d42e877c475b181909ee9a698: unknown error after kill: runc did not terminate successfully: exit status 1: unable to signal init: permission denied
: unknown

dmesg:
[33054.783037] audit: type=1400 audit(1697228308.520:1037): apparmor="DENIED" operation="signal" class="signal" profile="docker-default" pid=189468 comm="runc" requested_mask="receive" denied_mask="receive" signal=usr1 peer="/usr/sbin/runc"

can't find a docker-default profile in /etc/apparmor.d/
usr.sbin.runc has 0644

# cat usr.sbin.runc
abi <abi/4.0>,
include <tunables/global>

/usr/sbin/runc flags=(unconfined) {
  userns,

  # Site-specific additions and overrides. See local/README for details.
  include if exists <local/usr.sbin.runc>
}

Hope it helps

[John Johansen (jjohansen)]

the docker-default profile is shipped with/part of docker. It is generated and loaded by docker, you can see the docker apparmor code here

https://github.com/moby/moby/tree/master/profiles/apparmor

and the docker-default profile in particular is in
https://github.com/moby/moby/blob/master/profiles/apparmor/template.go

the signal rule needs to be updated, or a new rule added to allow runc

  signal receive signal=usr1 peer="/usr/sbin/runc",

[adam furtenbach (zmoosh)]

Great job, just to be clear copied the same dmesg line from earlier as it was a bit time consuming to move from virtual machine.

runc needs to be able to send any signal.

[35885.316617] audit: type=1400 audit(1700335230.173:423510): apparmor="DENIED" operation="signal" class="signal" profile="docker-default" pid=61658 comm="runc" requested_mask="receive" denied_mask="receive" signal=term peer="/usr/sbin/runc"
[35901.315862] audit: type=1400 audit(1700335246.173:423952): apparmor="DENIED" operation="signal" class="signal" profile="docker-default" pid=61744 comm="runc" requested_mask="receive" denied_mask="receive" signal=kill peer="/usr/sbin/runc"
[35920.822695] audit: type=1400 audit(1700335265.681:424397): apparmor="DENIED" operation="signal" class="signal" profile="docker-default" pid=62025 comm="runc" requested_mask="receive" denied_mask="receive" signal=int peer="/usr/sbin/runc"

Is any action required from me?

[John Johansen (jjohansen)]

I have added docker to this report, and we will need to report this to upstream docker, it likely for the time look at distro patching docker.

Locally you should be able to add the rule you need and use apparmor_parser -r to replace the profile until the bug is fixed. To allow all signals you can just do

  signal receive peer="/usr/sbin/runc",

or for the set currently encountered

  signal receive signal=(usr1 term kill int) peer="/usr/sbin/runc",

[Christian Boltz (cboltz)]

Slightly related:

> /usr/sbin/runc flags=(unconfined) {

Shouldn't that nowadays be(come)

profile runc /usr/sbin/runc flags=(unconfined) {

Ideally please fix this now, so that the upstream docker profile can use peer=runc

[John Johansen (jjohansen)]

Ideally yes. It will become

profile runc /usr/sbin/runc flags=(default_allow) { }

Ubuntu will have to distro patch for awhile.

[Georges Varouchas (gvarouchas)]

This issue is also affecting me, and I do not have experience with apparmor profiles to update the correct file.

Can someone explain in more details a patch that fixes the issue ?
(more precisely: what line should I write ? in what file ?)

Obviously: it is also a pain to have this issue with the stock system configuration, I hope this issue gets fixed sooner than later.

[John Johansen (jjohansen)]

@gvarouchas, you need to be more specific. There are a couple interrelated issues in this bug. What is the exact Denial message you are getting. The will look something like the denial messages in comment 5. You can find them using
  sudo dmesg | grep DENIED
or
  journalctl -g apparmor

[Georges Varouchas (gvarouchas)]

As a temporary patch on my system, I disabled the apparmor rules for /usr/sbin/runc

Following the documentation to disable one single apparmor profile (link: https://help.ubuntu.com/community/AppArmor#Disable_one_profile ) :

```
sudo ln -s /etc/apparmor.d/usr.sbin.runc /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.runc
```

docker can now send signals to its containers.

---

re-activating is documented in the next paragraph in the page above:

```
sudo rm /etc/apparmor.d/disable/usr.sbin.runc
sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.runc
```

docker stop will not be able to send a signal to its containers anymore.

[Tomáš Virtus (virtustom)]

As a temporary workaround, put the file I have attached to /etc/apparmor.d/docker-default and load it with "apparmor_parser -Kr /etc/apparmor.d/docker-default". This will make dockerd skip loading its builtin profile and use this one instead. The only difference between the builtin one and this one is the following rule:

  # runc may send signals to container processes
  signal (receive) peer=runc,

I've opened PRs upstream:
- https://github.com/containerd/containerd/pull/10123
- https://github.com/moby/moby/pull/47749

I think I'll need to work a little bit more on them to add rules only for profiles that exist. (It works even if they don't exist though.)

[Tomáš Virtus (virtustom)]

Forgot to attach the profile. Attached here.

[John Johansen (jjohansen)]

To make this generic so that it will work on older and newer hosts we should probably change the peer expression to

  signal (receive) peer={runc,unconfined},

or possibly, define an @{runc} variable in the preamble and use that. This really only is advantageous, in that it shows semantic intent, if if using the value of unconfined, or if @[runc} is used multiple times within the profile.

@{runc}={peer,unconfined}

   signal (receive) peer=@{runc},

[Garfonso (garfonso)]

I think the addition of @jjohansen is a good idea. On my system (updated from multiple older versions ;-)), the peer is not runc but /usr/sbin/runc and apparmor denied the signal before I changed the line to the full path. I am not sure if that is intended, but that's how it is now working on my system(s).