Can't run any container: shim error: open pid: no such file or directory

[Chris West]

Trying to run any container exits with this error message:

$ docker run -itd busybox
6f97c56ca785505aa37a72143934f668703ea737f1a08f79fe01c327978dcd2e
docker: Error response from daemon: shim error: open pid: no such file or directory.

The daemon logs aren't any more interesting:
Nov 23 13:27:16 astoria dockerd[12388]: time="2016-11-23T13:27:16.555673074Z" level=error msg="containerd: start container" error="shim error: open pid: no such file or directory" id=6f97c56ca785505aa37a72143934f668703ea737f1a08f79fe01c327978dcd2e
Nov 23 13:27:16 astoria dockerd[12388]: time="2016-11-23T13:27:16.560280252Z" level=error msg="Create container failed with error: shim error: open pid: no such file or directory"
Nov 23 13:27:16 astoria dockerd[12388]: time="2016-11-23T13:27:16.737068835Z" level=error msg="Handler for POST /v1.24/containers/6f97c56ca785505aa37a72143934f668703ea737f1a08f79fe01c327978dcd2e/start returned error: shim error: open pid: no such file or directory"

There's some upstream discussion about this being about incompatibilities between versions of components, with a recommendation that one upgrades everything and prays the problem goes away: https://github.com/docker/docker/issues/23958

I have recently updated from 16.04 to 16.10. I do not have any 16.04 packages left installed (except the ansible ppa!):

% sudo apt list --installed | egrep 'docker|containerd|runc|xenial'

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

ansible/xenial,xenial,now 2.2.0.0-1ppa~xenial all [installed]
containerd/yakkety,now 0.2.3-0ubuntu1 amd64 [installed]
docker.io/yakkety,now 1.12.1-0ubuntu15 amd64 [installed]
runc/yakkety,now 1.0.0~rc1-0ubuntu1 amd64 [installed]

I am using the btrfs storage backend on the stock kernel, with the default settings from apparmor and systemd, as far as I'm aware. Purging and reinstalling the docker.io package does not fix the issue. Deleting all images and containers makes no difference.

% docker info
Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 2
Server Version: 1.12.1
Storage Driver: btrfs
 Build Version: Btrfs v4.7.3
 Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 4.8.0-27-generic
Operating System: Ubuntu 16.10
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 23.49 GiB
Name: astoria
ID: ZHHL:W5GB:BSOZ:3FUA:2IPK:HLD4:KZGX:BJLI:UQQ7:V32B:JCCG:32NM
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

% docker version
Client:
 Version: 1.12.1
 API version: 1.24
 Go version: go1.6.3
 Git commit: 23cf638
 Built: Mon, 10 Oct 2016 21:38:17 +1300
 OS/Arch: linux/amd64

Server:
 Version: 1.12.1
 API version: 1.24
 Go version: go1.6.3
 Git commit: 23cf638
 Built: Mon, 10 Oct 2016 21:38:17 +1300
 OS/Arch: linux/amd64

ProblemType: Bug
DistroRelease: Ubuntu 16.10
Package: docker.io 1.12.1-0ubuntu15
ProcVersionSignature: Ubuntu 4.8.0-27.29-generic 4.8.1
Uname: Linux 4.8.0-27-generic x86_64
NonfreeKernelModules: nvidia_uvm nvidia
ApportVersion: 2.20.3-0ubuntu8
Architecture: amd64
Date: Wed Nov 23 13:26:40 2016
SourcePackage: docker.io
UpgradeStatus: Upgraded to yakkety on 2016-10-13 (40 days ago)
upstart.docker.override: manual

[Tianon Gravi (tianon)]

Interesting. I've seen some success in situations like this from simply restarting the Docker daemon -- mind giving that a shot?

[Joshua Powers (powersj)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Could you please let us know the results of trying Tianon's suggestion?

I'm marking this bug as Incomplete. We'd be grateful if you would then provide a more complete description of the problem, explain why you believe this is a bug in Ubuntu rather than a problem specific to your system, and then change the bug status back to New.

If indeed this is a local configuration problem, you can find pointers to get help for this sort of problem here: http://www.ubuntu.com/support/community

[Launchpad Janitor (janitor)]

[Expired for docker.io (Ubuntu) because there has been no activity for 60 days.]