docker.io 20.10.x does not create sub-directories in /var/lib/docker on start (but does on restart)

[Claudio Kuenzler]

Dear maintainers,

I noticed a changed behaviour when starting the Docker service after the docker.io package was upgraded from 19.3.6 to 20.10.7.

When the Docker host was reset/cleaned, which involves a 'rm -rf /var/lib/docker/*', the sub-directories are not automatically created anymore after a start of the service:

root@dockerhost:~# rm -rf /var/lib/docker/*
root@dockerhost:~# service docker start
root@dockerhost:~# ll /var/lib/docker/
total 0

Without these directories, Docker containers cannot be created. Example with a 'docker pull':

root@dockerhost:~# docker pull rancher/rke-tools:v0.1.78
v0.1.78: Pulling from rancher/rke-tools
540db60ca938: Pulling fs layer
0ae30075c5da: Pulling fs layer
9da81141e74e: Pulling fs layer
b2e41dd2ded0: Pulling fs layer
7f40e809fb2d: Pulling fs layer
758848c48411: Pulling fs layer
4aa8101d7589: Pulling fs layer
68bd44136930: Pulling fs layer
ae3790b81ced: Pulling fs layer
f9c62a2baf95: Pulling fs layer
759049e20249: Pulling fs layer
6713b6a87a77: Pulling fs layer
e1631138f00b: Pulling fs layer
063b41c39c12: Pulling fs layer
c7844f3999c4: Pulling fs layer
d8473758fa62: Pulling fs layer
f83b3d05af4e: Pulling fs layer
open /var/lib/docker/tmp/GetImageBlob573083372: no such file or directory

However a restart does re-create the missing folders:

root@dockerhost:~# service docker restart
root@dockerhost:~# ll /var/lib/docker/
total 44
drwx--x--x 4 root root 4096 Nov 11 13:49 buildkit
drwx--x--- 2 root root 4096 Nov 11 13:49 containers
drwx------ 3 root root 4096 Nov 11 13:49 image
drwxr-x--- 3 root root 4096 Nov 11 13:49 network
drwx--x--- 3 root root 4096 Nov 11 13:49 overlay2
drwx------ 4 root root 4096 Nov 11 13:49 plugins
drwx------ 2 root root 4096 Nov 11 13:49 runtimes
drwx------ 2 root root 4096 Nov 11 13:49 swarm
drwx------ 2 root root 4096 Nov 11 13:49 tmp
drwx------ 2 root root 4096 Nov 11 13:49 trust
drwx-----x 2 root root 4096 Nov 11 13:49 volumes

This happens with docker.io 20.10.x on both Ubuntu 18.04 Bionic and 20.04 Focal.

Compared to the behaviour of docker.io 19.3.x:

root@focal:~# docker info | head
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.8

root@focal:~# service docker stop
root@focal:~# rm -rf /var/lib/docker/*
root@focal:~# service docker start
root@focal:~# ll /var/lib/docker/
drwx--x--x 14 root root 4096 Nov 12 07:33 ./
drwxr-xr-x 29 root root 4096 Oct 29 14:06 ../
drwx------ 2 root root 4096 Nov 12 07:33 builder/
drwx--x--x 4 root root 4096 Nov 12 07:33 buildkit/
drwx------ 2 root root 4096 Nov 12 07:33 containers/
drwx------ 3 root root 4096 Nov 12 07:33 image/
drwxr-x--- 3 root root 4096 Nov 12 07:33 network/
drwx------ 3 root root 4096 Nov 12 07:33 overlay2/
drwx------ 4 root root 4096 Nov 12 07:33 plugins/
drwx------ 2 root root 4096 Nov 12 07:33 runtimes/
drwx------ 2 root root 4096 Nov 12 07:33 swarm/
drwx------ 2 root root 4096 Nov 12 07:33 tmp/
drwx------ 2 root root 4096 Nov 12 07:33 trust/
drwx------ 2 root root 4096 Nov 12 07:33 volumes/

Conclusion: The start script in docker.io 20.10.x should create the sub-directories in /var/lib/docker if they are missing (same as service restart).

cheers,
ck

[Tianon Gravi (tianon)]

In your second (working) example, I see you run "service docker stop" before deleting the contents of "/var/lib/docker" but I do not see that in your first example, which would definitely explain the behavior you see ("service docker start" would be a no-op because Docker's already running, and you've just yanked out all the data from underneath it, so it's not very surprising that it sees errors trying to access said data).

[Claudio Kuenzler (napsty)]

> I see you run "service docker stop" before deleting the contents of "/var/lib/docker" but I do not see that in your first example,

service docker stop is part of the general clean up strategies, for Rancher related Docker hosts anyway (see https://www.claudiokuenzler.com/blog/674/reset-clean-up-a-rancher-docker-host). So yes, this command definitely happened. The command would also throw a ton of errors, if Docker service would still be running due to busy/opened files.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.