dockerd umask inside lxd container

[Dylan Bridgman]

[Summary]

dockerd has a umask of 0177 when running inside a lxd container. This causes the files created in /var/lib/docker to only be accessible by root which in term causes permission errors for non-root users within docker containers when using the vfs storage driver.

[Machines]

Installed docker.io both inside a lxd container and inside a VirtualBox virtual machine.

1. LXD

Host running: Ubuntu 16.04.2 LTS, lxd 2.15-0ubuntu6~ubuntu16.04.1
LXD container: Ubuntu 16.04.2 LTS, docker.io 1.12.6-0ubuntu1~16.04.1
Docker: Storage driver default 'vfs'

2. VirtualBox

Host running: MacOS Sierra 10.12.6, VirtualBox 5.1.26
Virtual machine: Ubuntu 16.04.3 LTS, docker.io 1.12.6-0ubuntu1~16.04.1
Docker: Storage driver set to vfs in /etc/docker/daemon.json:

{
   "storage-driver": "vfs"
}

[Test]

Used gdb to check the effective umask of dockerd in each case:

1. LXD

# gdb --pid 1234
...
(gdb) call/o umask(0)
$1 = 0177
(gdb)

2. VirtualBox

# gdb --pid 1234
...
(gdb) call/o umask(0)
$1 = 022
(gdb)

[Permissons]

1. LXD

/var/lib/docker# ls -l
total 22
drwx------ 4 root root 4 Aug 3 10:11 containers
drw------- 3 root root 3 Jul 31 14:20 image
drw------- 3 root root 3 Jul 31 14:20 network
drw------- 2 root root 2 Jul 31 14:20 swarm
drwx------ 2 root root 2 Aug 3 09:23 tmp
drw------- 2 root root 2 Jul 31 14:20 trust
drw------- 3 root root 3 Jul 31 14:21 vfs
drw------- 2 root root 3 Aug 3 09:22 volumes

Images in vfs/dir/xxxx are mode drw-------

2. VirtualBox

/var/lib/docker# ls -l
total 36
drwx------ 5 root root 4 Aug 3 10:11 aufs
drwx------ 2 root root 4 Aug 3 10:11 containers
drwx------ 4 root root 3 Jul 31 14:20 image
drwxr-x--- 3 root root 3 Jul 31 14:20 network
drwx------ 2 root root 2 Jul 31 14:20 swarm
drwx------ 2 root root 2 Aug 3 09:23 tmp
drwx------ 2 root root 2 Jul 31 14:20 trust
drwx------ 3 root root 3 Jul 31 14:21 vfs
drwx------ 2 root root 3 Aug 3 09:22 volumes

Images in vfs/dir/xxxxx are mode drwxr-xr-x

[Dylan Bridgman (dylanbr)]

Step-by-step on how to replicate:

[Host machine]
## Install latest backports lxd
$ sudo apt install lxd/xenial-backports lxd-client/xenial-backports
...

## Initialise lxd default storage pool
$ sudo lxd init
Do you want to configure a new storage pool (yes/no) [default=yes]? yes
Name of the new storage pool [default=default]: default
Name of the storage backend to use (dir, btrfs, lvm) [default=dir]: dir
... (network setup)

## Create container for docker with nesting enabled
$ lxc launch ubuntu-daily:16.04 docktest -c security.nesting=true

## Enter a bash shell inside the container
$ lxc exec docktest -- bash

[LXD container]
## Install docker
# apt install docker.io -y
... (install)

## Confirm vfs storage driver
# docker info | grep Storage
Storage Driver: vfs

## Run Ubuntu inside docker
# docker run -it ubuntu
# docker create ubuntu
56a163f835da8a3b6b37d86b8ee1ab16e3b413efe9a57b22f8f0e6b919d43c30

[Docker container inside LXD container]
## Add a non-root test user
# adduser test
Adding user `test' ...
Adding new group `test' (1000) ...
Adding new user `test' (1000) with group `test' ...
Creating home directory `/home/test' ...
Copying files from `/etc/skel' ...
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for test
Enter the new value, or press ENTER for the default
 Full Name []: Test user
 Room Number []:
 Work Phone []:
 Home Phone []:
 Other []:
Is the information correct? [Y/n] y

## Attempt to su to test user
# su - test
Unable to cd to '/home/test'
... (error due to permisson issue)

[Dylan Bridgman (dylanbr)]

Please ignore these two lines, just running "docker run" is required:

-# docker create ubuntu
-56a163f835da8a3b6b37d86b8ee1ab16e3b413efe9a57b22f8f0e6b919d43c30

[Dylan Bridgman (dylanbr)]

A workaround may be used to temporarily fix the problem (following on from the replication instructions above):

[LXD container]
## Install gdb
# apt install gdb -y

## Find docker pid
# cat /run/docker.pid
2624

## Attach gdb and set umask to 022
# gdb --pid 2624
... (attach process)
(gdb) call/o umask(022)
$1 = 0177
(gdb) quit
...
Quit anyway? (y or n) y
Detaching from program: /usr/bin/dockerd, process 2624

## Fix existing vfs images
# cd /var/lib/docker/vfs/dir
# chmod 755 *

## Run Ubuntu inside docker
# docker run -it ubuntu

[Docker container inside LXD container]
## Add a non-root test user
# adduser test
Adding user `test' ...
Adding new group `test' (1000) ...
Adding new user `test' (1000) with group `test' ...
Creating home directory `/home/test' ...
Copying files from `/etc/skel' ...
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for test
Enter the new value, or press ENTER for the default
 Full Name []: Test user
 Room Number []:
 Work Phone []:
 Home Phone []:
 Other []:
Is the information correct? [Y/n] y

## Attempt to su to test user
# su - test
test:~$
(success)

[Dylan Bridgman (dylanbr)]

Additional note:

This does not affect docker when using the overlayfs storage driver. I am using the vfs storage driver because I am using ZFS which is not supported by overlayfs.

If one is using ext4 (or another overlayfs supported filesystem) then this can also be fixed by doing the following:

[Host machine]
## Load overlayfs
# modprobe overlay
... (also add to /etc/modules to have load on boot)

[LXD container]
## Restart docker daemon (if container was already running)
# service docker restart

## Confirm overlay storage driver is active
# docker info | grep Storage
Storage Driver: overlay