docker in security.privileged=true containers cannot start containers: write /sys/fs/cgroup/devices/docker/.../devices.allow: operation not permitted

[Tero Marttila]

Hi,

Running Ubuntu xenial with current 4.4.0-22-generic kernel and lxd 2.0.2-0ubuntu1~16.04.1, running Ubuntu's patched docker.io package within an unprivileged container (`lxc launch -p default -p docker ubuntu:xenial docker-test`) works, but fails once configuring the container with `lxc config set docker-test security.privileged true`:

 root@docker-test:~# docker run --rm -it debian:jessie bash
 docker: Error response from daemon: Cannot start container 07f5ddd392059c60aa12dd2f7292e54e01b153f2e203180f963989257fec9202: [10] System error: write /sys/fs/cgroup/devices/docker/07f5ddd392059c60aa12dd2f7292e54e01b153f2e203180f963989257fec9202/devices.allow: operation not permitted.

Upgrading to yakkety's docker.io=1.11.2-0ubuntu4 gives a slightly better error:

 docker: Error response from daemon: rpc error: code = 2 desc = "oci runtime error: failed to write c 10:200 rwm to devices.allow: write /sys/fs/cgroup/devices/docker/f05ecde20639572f27ac1ecf582b034d313b7d6573bddc2b57bd49ba1326e36d/devices.allow: operation not permitted".

Where:

  lrwxrwxrwx 1 root root 0 Jun 16 18:28 /sys/dev/char/10:200 -> ../../devices/virtual/misc/tun

It looks like containerd/runc per default wants to allow access to /dev/net/tun for containers:

https://github.com/docker/docker/blob/master/vendor/src/github.com/opencontainers/runc/libcontainer/configs/device_defaults.go#L101

Adding the tuntap device to the docker profile (and restarting the container):

 lxc profile device add docker tuntap unix-char path=/dev/net/tun

allows the device within the devices cgroup hierarchy:

 root@docker-test:~# cat /sys/fs/cgroup/devices/devices.list
 c *:* m
 b *:* m
 c 5:0 rwm
 c 5:1 rwm
 c 1:5 rwm
 c 1:7 rwm
 c 1:3 rwm
 c 1:8 rwm
 c 1:9 rwm
 c 5:2 rwm
 c 136:* rwm
 c 10:229 rwm
 c 10:200 rwm

and fixes docker run:

 root@docker-test:~# docker run --rm -it debian:jessie bash
 root@7ecba0a17fdd:/#

---

On the lxd host:

$ lsb_release -rd
Description: Ubuntu 16.04 LTS
Release: 16.04

$ apt-cache policy lxd
lxd:
  Installed: 2.0.2-0ubuntu1~16.04.1
  Candidate: 2.0.2-0ubuntu1~16.04.1
  Version table:
 *** 2.0.2-0ubuntu1~16.04.1 500
        500 http://apt/ubuntu xenial-security/main amd64 Packages
        500 http://apt/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     2.0.0-0ubuntu4 500
        500 http://apt/ubuntu xenial/main amd64 Packages

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Brandon Raabe (brandocorp)]

Just wanted to say that this fixed my issues:

lxc profile device add <container> tuntap unix-char path=/dev/net/tun

[Stéphane Graber (stgraber)]

Ah, good to know, closing this then. We've changed LXD to start mounting /dev/net/tun for you by default since it's perfectly safe.