can't delete docker containers in lxd: Driver btrfs failed to remove root filesystem

Bug #1574363 reported by Reinhard Tartler on 2016-04-24
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
docker.io (Ubuntu)
Undecided
Unassigned

Bug Description

System uses / on btrfs.

System: Xenial

Testcase:

apt install lxd
lxc launch -e ubuntu:x x
lxc exec apt update
lxc exec apt install docker.io
lxc exec docker -ti run debian:latest uptime

lxc exec bash
root@x:~# docker rm $(docker ps -a -q)
Failed to remove container (79afad5c75bc): Error response from daemon: Driver btrfs failed to remove root filesystem 79afad5c75bc356024f8b4f09634ca7cc561ec9358a79d1c7d8e15e0b1079240: Failed to destroy btrfs snapshot /var/lib/docker/btrfs/subvolumes for 861a950836c1704b016db1f6bb9c0cb77544cc19bfca6425be5a8ee3cf6597f9: operation not permitted

Stéphane Graber (stgraber) wrote :

Docker is supposed to always be using aufs inside LXD containers, the output above suggests it's not the case.

btrfs doesn't allow for unprivileged btrfs subvolume delete by default, only subvolume creation (yes, that's a bit odd). So the failure you're getting is absolutely normal except for the fact that I was told Docker would always do aufs inside unprivileged LXD containers.

As a workaround to keep having docker do btrfs inside LXD, you may want to change the host btrfs mountpoint to have user_subvol_rm_allowed as a mount option, then restart the LXD container and it should be able to both create and delete subvolumes.

Note that a bunch more btrfs functions will still fail in the container, so it'll likely only make things slightly better for you.

affects: lxd (Ubuntu) → docker.io (Ubuntu)
Reinhard Tartler (siretart) wrote :

Thank you for this clarification.

I'm not sure how to enable the aufs backend. Reading in the docker documentation at https://docs.docker.com/engine/userguide/storagedriver/selectadriver/, I understand with lxc running on btrfs, there is no other option than using the btrfs option.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in docker.io (Ubuntu):
status: New → Confirmed
Henry Lawson (henrylawson) wrote :

Not sure if Docker, LXD, or btrfs. "user_subvol_rm_allowed" is enabled, issue only started appearing recently, unsure if due to Docker upgrade or LXD upgrade. Observed this error earlier but "user_subvol_rm_allowed" resolved it. However, with it enabled it still occurs now.

me@host: lxd --version
2.4.1

root@docker:~# docker info
Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 1
Server Version: 1.12.1
Storage Driver: btrfs
 Build Version: Btrfs v4.7
 Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 4.4.0-43-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 4 GiB
Name: docker
ID: FOAC:LFCO:6B2U:HWVU:NGEC:GZAS:ZKCI:EBIU:JDBP:EGB4:2NMN:4KXI
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

root@docker:~# docker --version
Docker version 1.12.1, build 23cf638

root@docker:~# mount | grep btrfs
/dev/md2 on / type btrfs (rw,relatime,space_cache,user_subvol_rm_allowed,subvolid=49300,subvol=/var/lib/lxd/containers/docker/rootfs)
/dev/md2 on /dev/lxd type btrfs (rw,relatime,space_cache,user_subvol_rm_allowed,subvolid=5,subvol=/var/lib/lxd/devlxd)
/dev/md2 on /dev/.lxd-mounts type btrfs (rw,relatime,space_cache,user_subvol_rm_allowed,subvolid=5,subvol=/var/lib/lxd/shmounts/docker)
/dev/md2 on /var/lib/docker/btrfs type btrfs (rw,relatime,space_cache,user_subvol_rm_allowed,subvolid=49300,subvol=/var/lib/lxd/containers/docker/rootfs/var/lib/docker/btrfs)

And finally, the error:

docker rm 681fbe3f6870
Error response from daemon: Driver btrfs failed to remove root filesystem 0c214cb543ccd067e5208c8d5374b8345ec534c54b488719f17adb5f6cd44821: Failed to rescan btrfs quota for &{}: operation not permitted

Henry Lawson (henrylawson) wrote :

I also observe sporadic errors with:

Error response from daemon: stat /var/lib/docker/btrfs/subvolumes/a28d807f83e6131a6e43e062a73e7cbff496746e2dfceecc39a27483cabcd0d4: no such file or directory

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers