[uc18] docker overlayfs* seems broken

[Tony Espy]

A customer recently reported that 'sudo docker run hello-world' fails on a pi3 or pi4 running UC18. Looking at the journal, the failure appears to be caused by an apparmor denial related docker's overlay2 storage driver. I've tried both the unified and the Pi3 specific UC18 images and both fail with the same error. The same command works fine on other devices running UC18 (I've tested multipass+macOS, and dragonboard), and also works on a Pi3b running our standard UC16 image.

Here are the details from the UC18 image.

$ snap list
core 16-2.43.3 8691 stable canonical✓ core
core18 20200124 1673 stable canonical✓ base
docker 18.09.9 427 stable canonical✓ -
pi 18-1 27 18-pi canonical✓ gadget
pi-kernel 5.3.0-1019.21~18.04.1 104 18-pi canonical✓ kernel
snapd 2.43.3 6438 stable canonical✓ snapd

And here's the apparmor denial:

Mar 24 19:38:55 localhost sudo[3095]: awe : TTY=pts/0 ; PWD=/home/awe ; USER=root ; COMMAND=/snap/bin/docker run hello-world
Mar 24 19:39:02 localhost audit[2932]: AVC apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common/var-lib-docker/overlay2/afce643d5ac2c31f46b8c867c35abea776166c6da199fab370c30af17d314fd7-init/diff/.dockerenv" pid=2932 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

I've been told this may end up being something that gets worked around in snapd, however as this looks like a regression, I'm erring on the side of caution and filing this bug anyways.

Please let me know if there's anything else I can provide.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jamie Strandboge (jdstrand)]

I can't comment on the interaction of AppArmor and overlay with the available information. I can say that we already have these rules:

const dockerSupportConnectedPlugAppArmorCore = `
# These accesses are necessary for Ubuntu Core 16 and 18, likely due to the
# version of apparmor or the kernel which doesn't resolve the upper layer of an
# overlayfs mount correctly the accesses show up as runc trying to read from
# /system-data/var/snap/docker/common/var-lib-docker/overlay2/$SHA/diff/
/system-data/var/snap/{@{SNAP_NAME},@{SNAP_INSTANCE_NAME}}/common/{,**/} rwl,
/system-data/var/snap/{@{SNAP_NAME},@{SNAP_INSTANCE_NAME}}/@{SNAP_REVISION}/{,**/} rwl,
`

The denial of 'apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common/var-lib-docker/overlay2/afce643d5ac2c31f46b8c867c35abea776166c6da199fab370c30af17d314fd7-init/diff/.dockerenv" pid=2932 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0' doesn't match this though, because '.dockerenv' is a file, not a directory. If I were to guess, I'd guess that perhaps the snap is overlaying a file rather than a dir, but again, I don't know for sure.

It would be fine to adjust the policy to use this instead:

/system-data/var/snap/{@{SNAP_NAME},@{SNAP_INSTANCE_NAME}}/common/{,**} rwl,
/system-data/var/snap/{@{SNAP_NAME},@{SNAP_INSTANCE_NAME}}/@{SNAP_REVISION}/{,**} rwl,

since the snap already has read/write access to these directories when /system-data is not prepended. I've taken a todo to send up a PR for this.

[Maciej Borzecki (maciek-borzecki)]

Ian proposed a PR with the change suggested by Jamie: https://github.com/snapcore/snapd/pull/8426

[Ian Johnson (anonymouse67)]

I noticed that this same denial happens on UC20 on amd64 VM with kernel 5.4.0-20-generic, so perhaps this is just a new kernel behavior, and the linux-raspi2 kernel has newer patches than all the other UC18 kernels we have available.

[Ian Johnson (anonymouse67)]

Denial on UC20 VM:

Apr 06 13:11:35 ubuntu kernel: audit: type=1400 audit(1586178695.710:59): apparmor="DENIED" operation="open" profile="snap.docker.dockerd" name="/system-data/var/snap/docker/common/var-lib-docker/overlay2/ba5b8287c3c94aade8e3254eadbee5d4a4d279b4645e4337af9dd877c97ca1f2-init/diff/etc/resolv.conf" pid=1197 comm="dockerd" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

kernel info:

Linux ubuntu 5.4.0-20-generic #24-Ubuntu SMP Mon Mar 23 20:55:46 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[Juerg Haefliger (juergh)]

This was fixed in snapd: https://github.com/snapcore/snapd/pull/8426