flock does not resolve paths correctly in containers with microk8s

[Ian Johnson]

With the strict version of microk8s, we can "touch" files in directories such as /data/foo and read/write files there, but we cannot use flock or file_lock on such files as it triggers an apparmor denial:

[189905.568751] audit: type=1400 audit(1631746154.821:16659): apparmor="DENIED" operation="file_lock" profile="snap.microk8s.daemon-containerd" name="/data/test" pid=1989145 comm="flock" requested_mask="k" denied_mask="k" fsuid=0 ouid=0

Note that /data inside the container ends up getting mapped to somewhere inside $SNAP_COMMON:

root@machine-1:/var/snap/microk8s/common/default-storage/default-test-jb-pvc-5e1b2f7a-e04d-4404-8933-8d0db02c2389# ls
123
root@machine-1:/var/snap/microk8s/common/default-storage/default-test-jb-pvc-5e1b2f7a-e04d-4404-8933-8d0db02c2389# pwd
/var/snap/microk8s/common/default-storage/default-test-jb-pvc-5e1b2f7a-e04d-4404-8933-8d0db02c2389

@joeborg can provide reproducer details but this seems like a bug in AppArmor where it isn't resolving the path properly.

[Ian Johnson (anonymouse67)]

Of particular note is that there is an overlayfs mount on / for the container, see:

root@test-jb:/# cat /proc/self/mountinfo
2334 1749 0:166 / / rw,relatime - overlay overlay rw,lowerdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/770/fs,upperdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/774/fs,workdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/774/work
2335 2334 0:168 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
2336 2334 0:169 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755,inode64
2337 2336 0:170 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
2338 2336 0:157 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw
2339 2334 0:162 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs ro
2340 2339 0:171 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,mode=755,inode64
2341 2340 0:30 /kubepods/besteffort/poddb94f1c3-fdd4-4a92-8f4d-abf8f59c9fdf/9a5da99c7f9b159290b6e39a7818542063d006f8a4999dacd4cdb2e9a8adc864 /sys/fs/cgroup/systemd ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,xattr,name=systemd
2342 2340 0:40 /kubepods/besteffort/poddb94f1c3-fdd4-4a92-8f4d-abf8f59c9fdf/9a5da99c7f9b159290b6e39a7818542063d006f8a4999dacd4cdb2e9a8adc864 /sys/fs/cgroup/cpuset ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,cpuset,clone_children
2343 2340 0:38 /kubepods/besteffort/poddb94f1c3-fdd4-4a92-8f4d-abf8f59c9fdf/9a5da99c7f9b159290b6e39a7818542063d006f8a4999dacd4cdb2e9a8adc864 /sys/fs/cgroup/net_cls,net_prio ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,net_cls,net_prio
2344 2340 0:37 /kubepods/besteffort/poddb94f1c3-fdd4-4a92-8f4d-abf8f59c9fdf/9a5da99c7f9b159290b6e39a7818542063d006f8a4999dacd4cdb2e9a8adc864 /sys/fs/cgroup/perf_event ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,perf_event
2345 2340 0:36 /kubepods/besteffort/poddb94f1c3-fdd4-4a92-8f4d-abf8f59c9fdf/9a5da99c7f9b159290b6e39a7818542063d006f8a4999dacd4cdb2e9a8adc864 /sys/fs/cgroup/pids ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,pids
2346 2340 0:42 /kubepods/besteffort/poddb94f1c3-fdd4-4a92-8f4d-abf8f59c9fdf/9a5da99c7f9b159290b6e39a7818542063d006f8a4999dacd4cdb2e9a8adc864 /sys/fs/cgroup/blkio ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,blkio
2347 2340 0:41 /kubepods/besteffort/poddb94f1c3-fdd4-4a92-8f4d-abf8f59c9fdf/9a5da99c7f9b159290b6e39a7818542063d006f8a4999dacd4cdb2e9a8adc864 /sys/fs/cgroup/memory ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,memory
2348 2340 0:43 / /sys/fs/cgroup/rdma ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,rdma
2349 2340 0:34 /kubepods/besteffort/poddb94f1c3-fdd4-4a92-8f4d-abf8f59c9fdf/9a5da99c7f9b159290b6e39a7818542063d006f8a4999dacd4cdb2e9a8adc864 /sys/fs/cgroup/devices ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,devices
2350 2340 0:33 /kubepods/besteffort/poddb94f1c3-fdd4-4a92-8f4d-abf8f59c9fdf/9a5da99c7f9b159290b6e39a7818542063d006f8a4999dacd4cdb2e9a8adc864 /sys/fs/cgroup/cpu,cpuacct ro,nosuid,nodev,noexec,relatime - cgroup cgroup rw,cpu,cpuacct
2351 2340 0:39 /kubepods/besteffort/poddb94...

[Joseph Borg (joeborg)]

Reproduction steps:
---------------------

$ sudo snap install microk8s --channel latest/edge/strict
$ sudo microk8s enable storage

```test.yaml

apiVersion: v1
kind: Pod
metadata:
  name: test-jb
  labels:
    name: test-jb
spec:
  containers:
  - name: test-jb
    image: ubuntu:latest
    command: ["sleep", "999999999"]
    volumeMounts:
    - name: data
      mountPath: /data
  restartPolicy: "OnFailure"
  volumes:
  - name: "data"
    persistentVolumeClaim:
      claimName: test-jb
---
kind: "PersistentVolumeClaim"
apiVersion: "v1"
metadata:
  name: test-jb
  annotations:
    volume.alpha.kubernetes.io/storage-class: "generic"
spec:
  accessModes:
    - "ReadWriteOnce"
  resources:
    requests:
      storage: "10Gi"

```

$ sudo microk8s apply -f ./test.yaml # Wait a few seconds for this to come up.
$ sudo microk8s kubectl exec -it pod/test-jb -- bash
$$ cd /data
$$ flock test
flock: test: Permission denied
$$ ls
test

[Seth Arnold (seth-arnold)]

Does your policy grant 'k' permissions to the file in question?

If it does, it might be worth trying to prepare a reproducer with just overlayfs, none of the container stuff; stacking filesystems are awkward at best (I thought we had a bug for overlayfs, but my firefox history isn't being helpful, so now I'm doubting my memory) and it might be possible to reproduce this with a five-line shell script and overlayfs alone.

Thanks

[Alberto Mardegan (mardy)]

> Does your policy grant 'k' permissions to the file in question?

I think it does, as the snap.microk8s.daemon-containerd profile contains these rules:

  # Read-only system area for other versions
  # bind mount used here (see 'parallel installs', above)
  /var/snap/{@{SNAP_NAME},@{SNAP_INSTANCE_NAME}}/ r,
  /var/snap/{@{SNAP_NAME},@{SNAP_INSTANCE_NAME}}/** mrkix,

  # Writable system area only for this version
  # bind mount used here (see 'parallel installs', above)
  /var/snap/{@{SNAP_NAME},@{SNAP_INSTANCE_NAME}}/@{SNAP_REVISION}/** wl,
  /var/snap/{@{SNAP_NAME},@{SNAP_INSTANCE_NAME}}/common/** wl,

Also, running this command outside of the snap works:

    sudo aa-exec -p snap.microk8s.daemon-containerd \
        flock /var/snap/microk8s/common/default-storage/default-test-jb-pvc-aefec156-04d8-4cfe-a661-5df36eeca724/test echo ok