Working on creating a confined snap for kubelet. We're seeing a lot of denials. At least the following is needed to make kubelet work with the attached script:
1. adjust kubelet to 'plugs: [ log-observe, mount-observe ]'
2. adjust kubelet to make /var/log/containers snap-specific
3. modprobe llc stp bridge br_netfilter
4. create a kubernetes-support interface that allows (at least):
# what is this for?
#include <abstractions/dbus-strict>
capability sys_resource,
@{PROC}/diskstats r,
@{PROC}/@{pid}/cmdline r,
@{PROC}/@{pid}/cgroup r,
/sys/fs/cgroup/{,**} r,
/sys/kernel/mm/hugepages/ r,
@{PROC}/sys/kernel/random/boot_id r, # fixed already
@{PROC}/sys/kernel/panic_on_oops rw,
@{PROC}/sys/kernel/panic rw,
@{PROC}/sys/kernel/keys/root_maxbytes r,
@{PROC}/sys/kernel/keys/root_maxkeys r,
@{PROC}/sys/vm/panic_on_oom r,
@{PROC}/sys/vm/overcommit_memory rw,
@{PROC}/@{pid}/oom_score_adj rw,
# modprobe llc, stp, bridge, br_netfilter
/sys/module/llc/initstate r,
/sys/module/stp/initstate r,
/sys/module/bridge/initstate r,
/sys/module/br_netfilter/initstate r,
@{PROC}/sys/net/bridge/bridge-nf-call-iptables rw,
# seccomp blocks module loading, this is for listing
/sys/module/apparmor/parameters/enabled r,
/bin/kmod ixr,
/etc/modprobe.d/{,**} r,
ptrace (read, trace) peer=docker-default,
ptrace (read, trace) peer=unconfined, # hrmm
ptrace (read, trace) peer=snap.docker.dockerd,
/bin/journalctl ixr,
# make snap-specific
/var/log/containers/{,**} rw,
I'll put up a preliminary PR that implements the apparmor and kernel module policy so that people can play with this. In the meantime, after updating the kubelet snap to plugs log-observe and mount-observe and connect them, people can:
$ sudo modprobe llc stp bridge br_netfilter
# add the above policy to /var/lib/snapd/apparmor/profiles/snap.kubelet.kubelet the run:
$ sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.kubelet.kubelet
Locally at this point kubelet is spinning look for the api service so I can't see what other accesses are required.
= Original description =
Working on creating a confined snap for kubelet. We're seeing a lot of errors trying to open files relating to cgroups:
/proc/self/cgroup
/sys/fs/cgroup/cpu,cpuacct/cpu.shares
/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us
/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us
/sys/fs/cgroup/memory/memory.limit_in_bytes
/sys/fs/cgroup/memory/memory.soft_limit_in_bytes
/sys/fs/cgroup/blkio
/sys/fs/cgroup/memory
/sys/fs/cgroup/cpuset
The last three result in a hard failure of kubelet. There may be other files as well.
Based on snappy-debug output, it looks like it's opening these files with the "r" flag, but I imagine it may need write access to some of these as well. I'm not sure.
For some context, kubelet is the main process that runs on each node in a Kubernetes cluster. Its main purpose is to orchestrate Docker containers, and it looks like it's using cgroups for tight control over the utilization of hardware resources.
Manipulating cgroups is not currently supported by a generic interface. I suspect this will require a kubernetes specific interface, but do you have a simple reproducer to demonstrate the functionality you require?