Need an interface for kubernetes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
snapd |
Fix Released
|
Medium
|
Jamie Strandboge | ||
snapd (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Trusty |
Fix Released
|
Undecided
|
Unassigned | ||
Xenial |
Fix Released
|
Undecided
|
Unassigned | ||
Yakkety |
Fix Released
|
Undecided
|
Unassigned | ||
Zesty |
Fix Released
|
Undecided
|
Unassigned | ||
Artful |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
Working on creating a confined snap for kubelet. We're seeing a lot of denials. At least the following is needed to make kubelet work with the attached script:
1. adjust kubelet to 'plugs: [ log-observe, mount-observe ]'
2. adjust kubelet to make /var/log/containers snap-specific
3. modprobe llc stp bridge br_netfilter
4. create a kubernetes-support interface that allows (at least):
# what is this for?
#include <abstractions/
capability sys_resource,
@{PROC}/diskstats r,
@{PROC}
@{PROC}
/sys/fs/
/sys/kernel/
@{PROC}
@{PROC}
@{PROC}
@{PROC}
@{PROC}
@{PROC}
@{PROC}
@{PROC}
# modprobe llc, stp, bridge, br_netfilter
/sys/module/
/sys/module/
/sys/module/
/sys/module/
@{PROC}
# seccomp blocks module loading, this is for listing
/sys/module/
/bin/kmod ixr,
/etc/modprobe.
ptrace (read, trace) peer=docker-
ptrace (read, trace) peer=unconfined, # hrmm
ptrace (read, trace) peer=snap.
/bin/journalctl ixr,
# make snap-specific
/var/log/
I'll put up a preliminary PR that implements the apparmor and kernel module policy so that people can play with this. In the meantime, after updating the kubelet snap to plugs log-observe and mount-observe and connect them, people can:
$ sudo modprobe llc stp bridge br_netfilter
# add the above policy to /var/lib/
$ sudo apparmor_parser -r /var/lib/
Locally at this point kubelet is spinning look for the api service so I can't see what other accesses are required.
= Original description =
Working on creating a confined snap for kubelet. We're seeing a lot of errors trying to open files relating to cgroups:
/proc/self/cgroup
/sys/fs/
/sys/fs/
/sys/fs/
/sys/fs/
/sys/fs/
/sys/fs/
/sys/fs/
/sys/fs/
The last three result in a hard failure of kubelet. There may be other files as well.
Based on snappy-debug output, it looks like it's opening these files with the "r" flag, but I imagine it may need write access to some of these as well. I'm not sure.
For some context, kubelet is the main process that runs on each node in a Kubernetes cluster. Its main purpose is to orchestrate Docker containers, and it looks like it's using cgroups for tight control over the utilization of hardware resources.
summary: |
- Need an interface to access cgroups + Need an interface for kubernetes |
description: | updated |
Manipulating cgroups is not currently supported by a generic interface. I suspect this will require a kubernetes specific interface, but do you have a simple reproducer to demonstrate the functionality you require?