| 2017-02-14 16:47:58 |
George Kraft |
bug |
|
|
added bug |
| 2017-02-14 16:49:41 |
George Kraft |
attachment added |
|
Output of `snappy-debug.security scanlog kubelet` https://bugs.launchpad.net/snapd/+bug/1664638/+attachment/4818922/+files/kubelet-snappy-debug |
|
| 2017-02-28 18:00:17 |
Jamie Strandboge |
tags |
|
snapd-interface |
|
| 2017-02-28 18:02:54 |
Jamie Strandboge |
snapd: importance |
Undecided |
Medium |
|
| 2017-02-28 18:02:54 |
Jamie Strandboge |
snapd: status |
New |
Incomplete |
|
| 2017-02-28 20:59:39 |
George Kraft |
attachment added |
|
kubelet-confinement-test.sh https://bugs.launchpad.net/snapd/+bug/1664638/+attachment/4828591/+files/kubelet-confinement-test.sh |
|
| 2017-03-10 17:34:18 |
Jamie Strandboge |
summary |
Need an interface to access cgroups |
Need an interface for kubernetes |
|
| 2017-03-10 17:49:52 |
Jamie Strandboge |
description |
Working on creating a confined snap for kubelet. We're seeing a lot of errors trying to open files relating to cgroups:
/proc/self/cgroup
/sys/fs/cgroup/cpu,cpuacct/cpu.shares
/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us
/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us
/sys/fs/cgroup/memory/memory.limit_in_bytes
/sys/fs/cgroup/memory/memory.soft_limit_in_bytes
/sys/fs/cgroup/blkio
/sys/fs/cgroup/memory
/sys/fs/cgroup/cpuset
The last three result in a hard failure of kubelet. There may be other files as well.
Based on snappy-debug output, it looks like it's opening these files with the "r" flag, but I imagine it may need write access to some of these as well. I'm not sure.
For some context, kubelet is the main process that runs on each node in a Kubernetes cluster. Its main purpose is to orchestrate Docker containers, and it looks like it's using cgroups for tight control over the utilization of hardware resources. |
Working on creating a confined snap for kubelet. We're seeing a lot of denials. At least the following is needed to make kubelet work with the attached script:
1. adjust kubelet to 'plugs: [ mount-observe ]'
2. adjust kubelet to make /var/log/containers snap-specific
3. modprobe llc stp bridge br_netfilter
4. create a kubernetes-support interface that allows (at least):
# what is this for?
#include <abstractions/dbus-strict>
capability dac_override, # why?
capability sys_resource,
@{PROC}/diskstats r,
@{PROC}/@{pid}/cmdline r,
@{PROC}/@{pid}/cgroup r,
/sys/fs/cgroup/{,**} r,
/sys/kernel/mm/hugepages/ r,
@{PROC}/sys/kernel/random/boot_id r, # fixed already
@{PROC}/sys/kernel/panic_on_oops rw,
@{PROC}/sys/kernel/panic rw,
@{PROC}/sys/kernel/keys/root_maxbytes r,
@{PROC}/sys/kernel/keys/root_maxkeys r,
@{PROC}/sys/vm/panic_on_oom r,
@{PROC}/sys/vm/overcommit_memory rw,
@{PROC}/@{pid}/oom_score_adj rw,
# modprobe llc, stp, bridge, br_netfilter
/sys/module/llc/initstate r,
/sys/module/stp/initstate r,
/sys/module/bridge/initstate r,
/sys/module/br_netfilter/initstate r,
@{PROC}/sys/net/bridge/bridge-nf-call-iptables rw,
# seccomp blocks module loading, this is for listing
/sys/module/apparmor/parameters/enabled r,
/bin/kmod ixr,
/etc/modprobe.d/{,**} r,
ptrace (read, trace) peer=docker-default,
ptrace (read, trace) peer=unconfined, # hrmm
ptrace (read, trace) peer=snap.docker.dockerd,
/bin/journalctl ixr,
/run/log/journal/{,**} r,
/var/log/kern.log r,
# make snap-specific
/var/log/containers/{,**} rw,
I'll put up a preliminary PR that implements the apparmor and kernel module policy so that people can play with this. In the meantime, after updating the kubelet snap to plugs mount-observe and connect it, people can:
$ sudo modprobe llc stp bridge br_netfilter
# add the above policy to /var/lib/snapd/apparmor/profiles/snap.kubelet.kubelet the run:
$ sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.kubelet.kubelet
Locally at this point kubelet is spinning look for the api service so I can't see what other accesses are required.
= Original description =
Working on creating a confined snap for kubelet. We're seeing a lot of errors trying to open files relating to cgroups:
/proc/self/cgroup
/sys/fs/cgroup/cpu,cpuacct/cpu.shares
/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us
/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us
/sys/fs/cgroup/memory/memory.limit_in_bytes
/sys/fs/cgroup/memory/memory.soft_limit_in_bytes
/sys/fs/cgroup/blkio
/sys/fs/cgroup/memory
/sys/fs/cgroup/cpuset
The last three result in a hard failure of kubelet. There may be other files as well.
Based on snappy-debug output, it looks like it's opening these files with the "r" flag, but I imagine it may need write access to some of these as well. I'm not sure.
For some context, kubelet is the main process that runs on each node in a Kubernetes cluster. Its main purpose is to orchestrate Docker containers, and it looks like it's using cgroups for tight control over the utilization of hardware resources. |
|
| 2017-03-10 17:50:01 |
Jamie Strandboge |
snapd: status |
Incomplete |
In Progress |
|
| 2017-03-10 17:50:01 |
Jamie Strandboge |
snapd: assignee |
|
Jamie Strandboge (jdstrand) |
|
| 2017-03-10 18:50:02 |
Jamie Strandboge |
description |
Working on creating a confined snap for kubelet. We're seeing a lot of denials. At least the following is needed to make kubelet work with the attached script:
1. adjust kubelet to 'plugs: [ mount-observe ]'
2. adjust kubelet to make /var/log/containers snap-specific
3. modprobe llc stp bridge br_netfilter
4. create a kubernetes-support interface that allows (at least):
# what is this for?
#include <abstractions/dbus-strict>
capability dac_override, # why?
capability sys_resource,
@{PROC}/diskstats r,
@{PROC}/@{pid}/cmdline r,
@{PROC}/@{pid}/cgroup r,
/sys/fs/cgroup/{,**} r,
/sys/kernel/mm/hugepages/ r,
@{PROC}/sys/kernel/random/boot_id r, # fixed already
@{PROC}/sys/kernel/panic_on_oops rw,
@{PROC}/sys/kernel/panic rw,
@{PROC}/sys/kernel/keys/root_maxbytes r,
@{PROC}/sys/kernel/keys/root_maxkeys r,
@{PROC}/sys/vm/panic_on_oom r,
@{PROC}/sys/vm/overcommit_memory rw,
@{PROC}/@{pid}/oom_score_adj rw,
# modprobe llc, stp, bridge, br_netfilter
/sys/module/llc/initstate r,
/sys/module/stp/initstate r,
/sys/module/bridge/initstate r,
/sys/module/br_netfilter/initstate r,
@{PROC}/sys/net/bridge/bridge-nf-call-iptables rw,
# seccomp blocks module loading, this is for listing
/sys/module/apparmor/parameters/enabled r,
/bin/kmod ixr,
/etc/modprobe.d/{,**} r,
ptrace (read, trace) peer=docker-default,
ptrace (read, trace) peer=unconfined, # hrmm
ptrace (read, trace) peer=snap.docker.dockerd,
/bin/journalctl ixr,
/run/log/journal/{,**} r,
/var/log/kern.log r,
# make snap-specific
/var/log/containers/{,**} rw,
I'll put up a preliminary PR that implements the apparmor and kernel module policy so that people can play with this. In the meantime, after updating the kubelet snap to plugs mount-observe and connect it, people can:
$ sudo modprobe llc stp bridge br_netfilter
# add the above policy to /var/lib/snapd/apparmor/profiles/snap.kubelet.kubelet the run:
$ sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.kubelet.kubelet
Locally at this point kubelet is spinning look for the api service so I can't see what other accesses are required.
= Original description =
Working on creating a confined snap for kubelet. We're seeing a lot of errors trying to open files relating to cgroups:
/proc/self/cgroup
/sys/fs/cgroup/cpu,cpuacct/cpu.shares
/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us
/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us
/sys/fs/cgroup/memory/memory.limit_in_bytes
/sys/fs/cgroup/memory/memory.soft_limit_in_bytes
/sys/fs/cgroup/blkio
/sys/fs/cgroup/memory
/sys/fs/cgroup/cpuset
The last three result in a hard failure of kubelet. There may be other files as well.
Based on snappy-debug output, it looks like it's opening these files with the "r" flag, but I imagine it may need write access to some of these as well. I'm not sure.
For some context, kubelet is the main process that runs on each node in a Kubernetes cluster. Its main purpose is to orchestrate Docker containers, and it looks like it's using cgroups for tight control over the utilization of hardware resources. |
Working on creating a confined snap for kubelet. We're seeing a lot of denials. At least the following is needed to make kubelet work with the attached script:
1. adjust kubelet to 'plugs: [ log-observe, mount-observe ]'
2. adjust kubelet to make /var/log/containers snap-specific
3. modprobe llc stp bridge br_netfilter
4. create a kubernetes-support interface that allows (at least):
# what is this for?
#include <abstractions/dbus-strict>
capability sys_resource,
@{PROC}/diskstats r,
@{PROC}/@{pid}/cmdline r,
@{PROC}/@{pid}/cgroup r,
/sys/fs/cgroup/{,**} r,
/sys/kernel/mm/hugepages/ r,
@{PROC}/sys/kernel/random/boot_id r, # fixed already
@{PROC}/sys/kernel/panic_on_oops rw,
@{PROC}/sys/kernel/panic rw,
@{PROC}/sys/kernel/keys/root_maxbytes r,
@{PROC}/sys/kernel/keys/root_maxkeys r,
@{PROC}/sys/vm/panic_on_oom r,
@{PROC}/sys/vm/overcommit_memory rw,
@{PROC}/@{pid}/oom_score_adj rw,
# modprobe llc, stp, bridge, br_netfilter
/sys/module/llc/initstate r,
/sys/module/stp/initstate r,
/sys/module/bridge/initstate r,
/sys/module/br_netfilter/initstate r,
@{PROC}/sys/net/bridge/bridge-nf-call-iptables rw,
# seccomp blocks module loading, this is for listing
/sys/module/apparmor/parameters/enabled r,
/bin/kmod ixr,
/etc/modprobe.d/{,**} r,
ptrace (read, trace) peer=docker-default,
ptrace (read, trace) peer=unconfined, # hrmm
ptrace (read, trace) peer=snap.docker.dockerd,
/bin/journalctl ixr,
# make snap-specific
/var/log/containers/{,**} rw,
I'll put up a preliminary PR that implements the apparmor and kernel module policy so that people can play with this. In the meantime, after updating the kubelet snap to plugs log-observe and mount-observe and connect them, people can:
$ sudo modprobe llc stp bridge br_netfilter
# add the above policy to /var/lib/snapd/apparmor/profiles/snap.kubelet.kubelet the run:
$ sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.kubelet.kubelet
Locally at this point kubelet is spinning look for the api service so I can't see what other accesses are required.
= Original description =
Working on creating a confined snap for kubelet. We're seeing a lot of errors trying to open files relating to cgroups:
/proc/self/cgroup
/sys/fs/cgroup/cpu,cpuacct/cpu.shares
/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_period_us
/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us
/sys/fs/cgroup/memory/memory.limit_in_bytes
/sys/fs/cgroup/memory/memory.soft_limit_in_bytes
/sys/fs/cgroup/blkio
/sys/fs/cgroup/memory
/sys/fs/cgroup/cpuset
The last three result in a hard failure of kubelet. There may be other files as well.
Based on snappy-debug output, it looks like it's opening these files with the "r" flag, but I imagine it may need write access to some of these as well. I'm not sure.
For some context, kubelet is the main process that runs on each node in a Kubernetes cluster. Its main purpose is to orchestrate Docker containers, and it looks like it's using cgroups for tight control over the utilization of hardware resources. |
|
| 2017-04-28 14:17:42 |
Andy Whitcroft |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2017-04-28 14:17:43 |
Andy Whitcroft |
bug |
|
|
added subscriber SRU Verification |
| 2017-04-28 14:17:45 |
Andy Whitcroft |
tags |
snapd-interface |
snapd-interface verification-needed |
|
| 2017-04-28 14:18:12 |
Andy Whitcroft |
bug task added |
|
snapd (Ubuntu) |
|
| 2017-04-28 14:19:03 |
Andy Whitcroft |
snapd (Ubuntu Zesty): status |
New |
Fix Committed |
|
| 2017-04-28 17:59:26 |
Jamie Strandboge |
tags |
snapd-interface verification-needed |
snapd-interface verification-done-zesty |
|
| 2017-04-28 18:01:26 |
Jamie Strandboge |
attachment added |
|
test-kubernetes-support_0_all.snap https://bugs.launchpad.net/snapd/+bug/1664638/+attachment/4869255/+files/test-kubernetes-support_0_all.snap |
|
| 2017-04-28 18:01:45 |
Jamie Strandboge |
snapd: status |
In Progress |
Fix Released |
|
| 2017-04-28 18:03:08 |
Jamie Strandboge |
nominated for series |
|
Ubuntu Artful |
|
| 2017-04-28 18:03:08 |
Jamie Strandboge |
bug task added |
|
snapd (Ubuntu Artful) |
|
| 2017-04-28 18:03:15 |
Jamie Strandboge |
snapd (Ubuntu Artful): status |
New |
Fix Committed |
|
| 2017-04-29 06:28:38 |
Andy Whitcroft |
snapd (Ubuntu Yakkety): status |
New |
Fix Committed |
|
| 2017-04-29 06:28:42 |
Andy Whitcroft |
tags |
snapd-interface verification-done-zesty |
snapd-interface verification-done-zesty verification-needed |
|
| 2017-04-29 08:34:35 |
Andy Whitcroft |
snapd (Ubuntu Xenial): status |
New |
Fix Committed |
|
| 2017-04-29 08:36:17 |
Andy Whitcroft |
snapd (Ubuntu Trusty): status |
New |
Fix Committed |
|
| 2017-05-01 15:29:16 |
Jamie Strandboge |
tags |
snapd-interface verification-done-zesty verification-needed |
snapd-interface verification-done-yakkety verification-done-zesty verification-needed |
|
| 2017-05-01 15:33:32 |
Jamie Strandboge |
tags |
snapd-interface verification-done-yakkety verification-done-zesty verification-needed |
snapd-interface verification-done-xenial verification-done-yakkety verification-done-zesty verification-needed |
|
| 2017-05-01 15:37:28 |
Jamie Strandboge |
tags |
snapd-interface verification-done-xenial verification-done-yakkety verification-done-zesty verification-needed |
snapd-interface verification-done-trusty verification-done-xenial verification-done-yakkety verification-done-zesty |
|
| 2017-05-08 20:07:13 |
George Kraft |
attachment added |
|
test-confined-kubelet-with-juju.sh https://bugs.launchpad.net/snapd/+bug/1664638/+attachment/4873686/+files/test-confined-kubelet-with-juju.sh |
|
| 2017-05-12 17:05:51 |
Launchpad Janitor |
snapd (Ubuntu Artful): status |
Fix Committed |
Fix Released |
|
| 2017-05-16 11:19:33 |
Launchpad Janitor |
snapd (Ubuntu Zesty): status |
Fix Committed |
Fix Released |
|
| 2017-05-16 11:19:41 |
Andy Whitcroft |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2017-05-16 11:19:56 |
Launchpad Janitor |
snapd (Ubuntu Yakkety): status |
Fix Committed |
Fix Released |
|
| 2017-05-16 11:20:14 |
Launchpad Janitor |
snapd (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2017-05-16 11:20:22 |
Launchpad Janitor |
snapd (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
