Cannot start containers after docker.io update

[Christian Henz]

(This is on xenial, but I've had similar problems on bionic as well).

The docker.io package was updated last night (via unattended-upgrades), and this morning I could no longer start containers, the error message being:

    docker: Error response from daemon: failed to start shim: exec: "docker-containerd-shim": executable file not found in $PATH: unknown.

To get it to work again I

- pruned all stopped containers
- restarted dockerd
- restarted containerd

Not sure if all of these steps were needed.

An issue like this seems to pop up basically whenever the docker.io package is updated.

[Sitsofe Wheeler (sitsofe)]

I've also seen this on multiple machines since yesterday and just like the reporter the machines are Xenial and Bionic:

From /var/log/apt.history.log on a Xenial machine:

Commandline: /usr/bin/unattended-upgrade
Install: containerd:amd64 (1.2.6-0ubuntu1~16.04.3, automatic)
Upgrade: apport:amd64 (2.20.1-0ubuntu2.18, 2.20.1-0ubuntu2.19), gvfs-libs:amd64 (1.28.2-1ubuntu1~16.04.1, 1.28.2-1ubuntu1~16.04.3), python3-apport:amd64 (2.20.1-0ubuntu2.18, 2.20.1-0ubuntu2.19), runc:amd64 (1.0.0~rc2+docker1.13.1-0ubuntu1~16.04.2, 1.0.0~rc7+git20190403.029124da-0ubuntu1~16.04.4), libglib2.0-bin:amd64 (2.48.2-0ubuntu4.2, 2.48.2-0ubuntu4.3), libglib2.0-dev:amd64 (2.48.2-0ubuntu4.2, 2.48.2-0ubuntu4.3), libglib2.0-data:amd64 (2.48.2-0ubuntu4.2, 2.48.2-0ubuntu4.3), gvfs:amd64 (1.28.2-1ubuntu1~16.04.1, 1.28.2-1ubuntu1~16.04.3), gvfs-common:amd64 (1.28.2-1ubuntu1~16.04.1, 1.28.2-1ubuntu1~16.04.3), python3-problem-report:amd64 (2.20.1-0ubuntu2.18, 2.20.1-0ubuntu2.19), gvfs-daemons:amd64 (1.28.2-1ubuntu1~16.04.1, 1.28.2-1ubuntu1~16.04.3), libglib2.0-0:amd64 (2.48.2-0ubuntu4.2, 2.48.2-0ubuntu4.3), docker.io:amd64 (18.06.1-0ubuntu1.2~16.04.1, 18.09.7-0ubuntu1~16.04.4)

From /var/log/apt.history.log on a Bionic machine:
Commandline: /usr/bin/unattended-upgrade
Install: containerd:amd64 (1.2.6-0ubuntu1~18.04.2, automatic), runc:amd64 (1.0.0~rc7+git20190403.029124da-0ubuntu1~18.04.2, automatic)
Upgrade: docker.io:amd64 (18.06.1-0ubuntu1.2~18.04.1, 18.09.7-0ubuntu1~18.04.3)

Commands like docker ps work but it is NOT possible to stop already running containers and trying to exec into them causes error messages like this:

$ docker exec -it myrunningcontainer /bin/bash
OCI runtime state failed: exec: "docker-runc": executable file not found in $PATH: : unknown

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sitsofe Wheeler (sitsofe)]

Further information:
Doing things like docker stop will hang and won't actually stop the running container (but won't spit out an error message or exit with a non-zero return code). docker kill fails to change the state of the system in a similar way.

In my case restarting the docker daemon just left all the old containers running we were forced to reboot the machine in order to recover.

[REIGNIER (rreignier)]

The issue was encounter on several machines today also.
A reboot of the system allowed to restart the containers.

[swip (swip88)]

if need disable unattended-upgrade, add APT::Periodic::Unattended-Upgrade "0"; in /etc/apt/apt.conf.d/10periodic

[caleb collins-parks (caleb15)]

I just ran into this problem. Rebooting the machine fixed it.