[SRU] update containerd:amd64 1.3.3-0 stops docker daemon

[Bernd]

[Impact]

Docker uses containerd under the hood. When containerd is upgraded it stops and restarts its service; docker stops when containerd stops but doesn’t restart. Particularly when doing unattended upgrades, an SRU fix rolled out for containerd can result in unexpected and widespread service outages for docker.

[Test Case]

$ sudo apt install docker.io
$ sudo systemctl start docker
$ systemctl status docker | grep Active
     Active: active (running) since[...]
$ systemctl status containerd | grep Active
     Active: active (running) since[...]

$ docker pull ubuntu/redis:latest
$ docker run -e REDIS_PASSWORD=1234 --network host \
--name test-redis -d ubuntu/redis:latest
$ telnet localhost 6379
$ docker container logs test-redis

$ sudo apt install --reinstall containerd
$ systemctl status containerd | grep Active
     Active: active (running) since
$ systemctl status docker | grep Active
     Active: inactive (dead) since [...]; 8s ago
$ docker container logs test-redis

[Where Problems Could Occur]

The challenge with this issue is addressing all important corner cases, and as such the biggest risk is that we miss a corner case and fail to keep the two services running when they should. Areas to watch will be failures during start/stop/restart/upgrade type operations. Issues during runtime are unlikely to relate to this change.

[Original Report]

hello we have had in several vms, the problem that after updating to " containerd:amd64 1.3.3" the docker daemon is stopped and so are all running containers

the vms run with ubuntu 18.04.1/18.04.2

journal log during the update:

```
-- Logs begin at Tue 2020-01-14 09:58:27 CET, end at Fri 2020-04-03 11:30:39 CEST. --
Apr 03 06:09:09 server dockerd[1751]: time="2020-04-03T06:09:09+02:00" level=info msg="level=info ts=2020-04-03T04:09:09.648358396Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:09 server dockerd[1751]: time="2020-04-03T06:09:09+02:00" level=info msg="level=info ts=2020-04-03T04:09:09.660781933Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=\" \"" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:14 server interface_rules[1485]: {"time":"2020-04-03 06:09:14+02:00","message":"interface enp0s3 works fine"}
Apr 03 06:09:14 server interface_rules[1485]: {"time":"2020-04-03 06:09:14+02:00","message":"interface enp0s8 works fine"}
Apr 03 06:09:24 server dockerd[1751]: time="2020-04-03T06:09:24+02:00" level=info msg="level=info ts=2020-04-03T04:09:24.650166885Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:24 server dockerd[1751]: time="2020-04-03T06:09:24+02:00" level=info msg="level=info ts=2020-04-03T04:09:24.663239678Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=\" \"" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:31 server systemd[1]: Starting Daily apt upgrade and clean activities...
Apr 03 06:09:39 server dockerd[1751]: time="2020-04-03T06:09:39+02:00" level=info msg="level=info ts=2020-04-03T04:09:39.64981528Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:39 server dockerd[1751]: time="2020-04-03T06:09:39+02:00" level=info msg="level=info ts=2020-04-03T04:09:39.665086076Z caller=loki.go:55 container_id=1eefb971e84cdaa4eadb0ac890fe9adf5275a4e9cbcf9635cf941b3aa469dd1f msg=\"ignoring empty line\" line=\" \"" plugin=255e05f2d4535991812f30e4cb5ea39fe0b498ca612c26dd89909de318b7f68d
Apr 03 06:09:43 server systemd[1]: Stopping Docker Application Container Engine...
Apr 03 06:09:43 server dockerd[1751]: time="2020-04-03T06:09:43.571408106+02:00" level=info msg="Processing signal 'terminated'"
Apr 03 06:09:43 server dockerd[1751]: time="2020-04-03T06:09:43.614841270+02:00" level=info msg="Daemon shutdown complete"
Apr 03 06:09:43 server dockerd[1751]: time="2020-04-03T06:09:43.614972021+02:00" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=moby
Apr 03 06:09:43 server systemd[1]: Stopped Docker Application Container Engine.
Apr 03 06:09:43 server systemd[1]: Stopping containerd container runtime...
Apr 03 06:09:43 server systemd[1]: Closed Docker Socket for the API.
Apr 03 06:09:43 server containerd[1736]: time="2020-04-03T06:09:43.896546147+02:00" level=info msg="Stop CRI service"
Apr 03 06:09:43 server containerd[1736]: time="2020-04-03T06:09:43.897819071+02:00" level=info msg="Stop CRI service"
Apr 03 06:09:43 server containerd[1736]: time="2020-04-03T06:09:43.899864471+02:00" level=info msg="Event monitor stopped"
Apr 03 06:09:43 server containerd[1736]: time="2020-04-03T06:09:43.900291027+02:00" level=info msg="Stream server stopped"
Apr 03 06:09:43 server systemd[1]: Stopped containerd container runtime.
Apr 03 06:09:47 server systemd[1]: Reloading.
Apr 03 06:09:48 server systemd[1]: Reloading.
Apr 03 06:09:49 server systemd[1]: Reloading.
Apr 03 06:09:49 server systemd[1]: Starting containerd container runtime...
Apr 03 06:09:49 server containerd[1904]: time="2020-04-03T06:09:49.951538613+02:00" level=info msg="starting containerd" revision= version="1.3.3-0ubuntu1~18.04.2"
Apr 03 06:09:49 server systemd[1]: Started containerd container runtime.
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.021260455+02:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.021461398+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.022876671+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.022992123+02:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.023036103+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.031683852+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.031948994+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.032592571+02:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.034455570+02:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.034808794+02:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.034914325+02:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.034952775+02:00" level=info msg="metadata content store policy set" policy=shared
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035299108+02:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035362929+02:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035585401+02:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035642662+02:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035676962+02:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035751653+02:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035797984+02:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035899825+02:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035955175+02:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.035994665+02:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.504599571+02:00" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.504847245+02:00" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.507240519+02:00" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.514716375+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515138699+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515385472+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515439083+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515474183+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515521113+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515569144+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515602784+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515645985+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515812346+02:00" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.515960658+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.516016449+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.516062909+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.516218490+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.cri\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.516437722+02:00" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Engine: PodAnnotations:[] Root: Options:<nil> PrivilegedWithoutHostDevices:false} UntrustedWorkloadRuntime:{Type: Engine: PodAnnotations:[] Root: Options:<nil> PrivilegedWithoutHostDevices:false} Runtimes:map[runc:{Type:io.containerd.runc.v1 Engine: PodAnnotations:[] Root: Options:<nil> PrivilegedWithoutHostDevices:false}] NoPivot:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginConfTemplate:} Registry:{Mirrors:map[docker.io:{Endpoints:[https://registry-1.docker.io]}] Configs:map[] Auths:map[]} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:0 StreamIdleTimeout:4h0m0s EnableSelinux:false SandboxImage:k8s.gcr.io/pause:3.1 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false} ContainerdRootDir:/var/lib/containerd ContainerdEndpoint:/run/containerd/containerd.sock RootDir:/var/lib/containerd/io.containerd.grpc.v1.cri StateDir:/run/containerd/io.containerd.grpc.v1.cri}"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.516732916+02:00" level=info msg="Connect containerd service"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.517528134+02:00" level=info msg="Get image filesystem path \"/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs\""
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.517888717+02:00" level=error msg="Failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.518436663+02:00" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.518837507+02:00" level=info msg="Start subscribing containerd event"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519057900+02:00" level=info msg="Start recovering state"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519208531+02:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519444914+02:00" level=info msg=serving... address=/run/containerd/containerd.sock
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519481864+02:00" level=info msg="Start event monitor"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519573005+02:00" level=info msg="Start snapshots syncer"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519657075+02:00" level=info msg="Start streaming server"
Apr 03 06:09:50 server containerd[1904]: time="2020-04-03T06:09:50.519497564+02:00" level=info msg="containerd successfully booted in 0.569453s"
Apr 03 06:10:08 server systemd[1]: Reloading.
Apr 03 06:10:08 server systemd[1]: Stopping LSB: automatic crash report generation...
Apr 03 06:10:08 server apport[2377]: * Stopping automatic crash report generation: apport
Apr 03 06:10:08 server apport[2377]: ...done.
Apr 03 06:10:08 server systemd[1]: Stopped LSB: automatic crash report generation.
Apr 03 06:10:09 server systemd[1]: Reloading.
Apr 03 06:10:09 server systemd[1]: Reloading.
Apr 03 06:10:11 server systemd[1]: Reloading.
Apr 03 06:10:11 server systemd[1]: Reloading.
Apr 03 06:10:12 server systemd[1]: Reloading.
Apr 03 06:10:12 server systemd[1]: Starting LSB: automatic crash report generation...
Apr 03 06:10:12 server apport[2666]: * Starting automatic crash report generation: apport
Apr 03 06:10:12 server apport[2666]: ...done.
Apr 03 06:10:12 server systemd[1]: Started LSB: automatic crash report generation.
Apr 03 06:10:12 server systemd[1]: Reloading.
Apr 03 06:10:18 server systemd[1]: Started Daily apt upgrade and clean activities.

```

[Bernd (badsmoke)]

containerd:amd64 1.3.3-0ubuntu1~18.04.2

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1870514/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[r0mulux (r-marie)]

Duplicate of https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1870876?comments=all

[Gert van den Berg (mohag1)]

This also took Docker down on 18.04 with the update for USN-4653-1

[Balint Reczey (rbalint)]

I'm adding the rls-ff-incoming tag because the issue seem to be showing up on every containerd update.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sitsofe Wheeler (sitsofe)]

We have seen this on a bionic (18.04) and xenial (16.04) hosts. Does this definitely not impact focal as well?

[Chris J Arges (arges)]

After unattended-updates upgraded containerd our docker services were no longer accessible as the docker daemon had stopped. This happened with the update to 1.3.3-0ubuntu1~18.04.3 on Bionic.

[Krystian Szczesny (krystians)]

Confirmed on Ubuntu 20.04 LTS (GNU/Linux 5.4.0-1015-aws x86_64) with containerd 1.3.3-0ubuntu2.1

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[BeeJayBlazkowicz (beejayblazkowicz)]

I can confirm this aswell for 18.04, happened this morning...

[Stephen Buergler (sixcorners)]

What does rls-ff-incoming mean? Is this something that can be fixed everywhere? I'm pretty sure it's something that happens to mysql also: https://askubuntu.com/q/1037285
A workaround for this seems to be to use the docker snap package. snapd seems to keep stuff running.. at least when there isn't some horrible apparmor issue.

[Stefan Lasiewski (stefanlasiewski)]

> A workaround for this seems to be to use the docker snap package.

Use caution here. Snap tends to upgrade/restart things on its own schedule, so you might run into the same issue.

I use Apt's pinning feature to pin the Docker version on production hosts:

```
/etc/apt/preferences.d# cat docker.pref
Explanation: docker: docker
Package: docker
Pin: version x:yz-ubuntu-bionic
Pin-Priority: 1001
/etc/apt/preferences.d#
```

[Richard Harding (rharding)]

If you are currently affected:

    sudo systemctl restart docker

It looks like the packaging will stop docker when containerd is stopped, but does not start it on start and that's dropped folks into this situation. We're working out a fix for that.

[Tianon Gravi (tianon)]

Aha, this sounds like it's probably the same root cause that led to https://github.com/docker/docker-ce-packaging/pull/508.

[Svetlana Podvalniuk (svetlana41)]

We have the same problem docker daemon was stopped on two EC2 instances and never started again due to network interface being unavailable. As everyone previously mentioned starting docker again fixes the issue, best here is to disable updates it seems like.

[Richard Harding (rharding)]

@Tianon, yes we're tracking the upstream changes but are not finding them sufficient to address the issue. We're finding we need more than just that change to BindsTo

[Hyper_Sloth (info-sbphotographics)]

Confirmed on 20.04 LTS. https://www.reddit.com/r/docker/comments/k4jiyp/woke_up_to_my_docker_daemon_being_stopped_no_logs

[Hyper_Sloth (info-sbphotographics)]

Getting about 30 other people commenting on the thread above reporting the same issue

[Philip Lewis (mrdecisive)]

As a workaround I opted for using a systemd unit file to create a loose dependency between containerd and dockerd.

Here's how:

As root:

Make the override directory (permission 755)

mkdir /etc/systemd/system/containerd.service.d/

Create the override file in the above directory (permissions 644)

vi override.conf

Add the following to the file

[Unit]
Before=docker.service
Wants=docker.service

Restart systemd daemon

systemctl daemon-reload

Check it works

Stop containerd

systemctl stop containerd

Check dockerd has stopped

systemctl status docker

Start containerd

systemctl start containerd

Check dockerd has started

systemctl status docker
++++++++++++++++++++++++++

Some links to other documentation on the subject

https://www.shellhacks.com/systemd-service-file-example/

https://man7.org/linux/man-pages/man5/systemd.service.5.html

[Bryce Harrington (bryce)]

I'd like to give you all an update and outline our plans for this.

The Canonical server team has made analysis of this issue a top priority. We've identified and tested out several possible theories. Our findings suggest that the breakage involves two distinct issues, one the BindTo= issue mentioned above, the other caused by a bug in the docker.io package causing the service to stop on package upgrade; see specifically the service stop command at the end of /var/lib/dpkg/info/docker.io.prerm. We'll use LP: #1870514 to track the former issue, and #1906364 the latter. LP: #1658691 gives some past background for reference.

The tricky part is that unfortunately any change we make to docker.io requires the running of the prerm script (the version of the script already present on your system, not the one we'd be installing), and thus triggers the bug. In other words, updating your system to prevent the bug will cause one more docker stop. Thereafter, the upgrade will not restart the service when rolling out CVE fixes to either containerd or docker.io; it may prompt to do so if running interactively (e.g. https://imgur.com/2Za5dbQ.png), otherwise it should respect the debconf setting.

We would appreciate feedback, testing and/or review of the proposed fix, available in this PPA:

   https://launchpad.net/~bryce/+archive/ubuntu/containerd-sru-lp1870514-docker-dh/

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Launchpad Janitor (janitor)]

This bug was fixed in the package docker.io - 19.03.13-0ubuntu4

---------------
docker.io (19.03.13-0ubuntu4) hirsute; urgency=medium

  * d/p/do_not_bind_docker_to_containerd.patch: Update docker.io to not
    stop when containerd is upgraded, by using Wants= rather than BindTo=.
    (LP: #1870514)
  * d/rules: Fix docker.io to not restart its service during package
    upgrades, to prevent service downtime from automatic updates via
    unattended-upgrade.
    (LP: #1906364)

 -- Bryce Harrington <email address hidden> Fri, 04 Dec 2020 23:02:49 +0000

[Timo Aaltonen (tjaalton)]

Hello Bernd, or anyone else affected,

Accepted docker.io into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/docker.io/19.03.13-0ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Bernd, or anyone else affected,

Accepted containerd into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/containerd/1.3.7-0ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Bernd, or anyone else affected,

Accepted containerd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/containerd/1.3.3-0ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Bernd, or anyone else affected,

Accepted docker.io into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/docker.io/19.03.8-0ubuntu1.20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Bernd, or anyone else affected,

Accepted docker.io into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/docker.io/19.03.6-0ubuntu1~18.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Bernd, or anyone else affected,

Accepted containerd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/containerd/1.3.3-0ubuntu1~18.04.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Bernd, or anyone else affected,

Accepted containerd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/containerd/1.2.6-0ubuntu1~16.04.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Bernd, or anyone else affected,

Accepted docker.io into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/docker.io/18.09.7-0ubuntu1~16.04.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

fixed in hirsute by 1.3.7-0ubuntu5

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted containerd (1.2.6-0ubuntu1~16.04.6) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

docker.io/18.09.7-0ubuntu1~16.04.7 (amd64, arm64, s390x, i386, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#containerd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted docker.io (18.09.7-0ubuntu1~16.04.7) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

docker.io/18.09.7-0ubuntu1~16.04.7 (amd64, arm64, s390x, i386, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#docker.io

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted docker.io (19.03.6-0ubuntu1~18.04.3) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-fan/0.12.10 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#docker.io

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Orgad Shaneh (orgads)]

With the latest version on groovy, I've noticed the following message on upgrade:
/var/lib/dpkg/info/docker.io.postinst: 31: zfs: not found

[Lucas Kanashiro (lucaskanashiro)]

@orgads I filed another bug for the zfs issue you mentioned here:

https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1910133

From what I can see this error is reported in Groovy and Hirsute while configuring the package but it does not break the package installation.

[Lucas Kanashiro (lucaskanashiro)]

The autopkgtest regressions reported above were handled. The ubuntu-fan test failure was fixed after re-triggering it manually, and the docker.io test in Xenial was indeed failing and I just uploaded version 18.09.7-0ubuntu1~16.04.8 fixing it. With that in place all the tests should be passing.

[Lucas Kanashiro (lucaskanashiro)]

I have tested the scenario described in the Test Case section in the bug description through all the target releases and they all seem good:

# Groovy

ubuntu@docker-groovy:~$ sudo apt install -t groovy-proposed docker.io
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  aufs-tools cgroupfs-mount | cgroup-lite debootstrap docker-doc rinse zfs-fuse | zfsutils
The following NEW packages will be installed:
  docker.io
0 upgraded, 1 newly installed, 0 to remove and 15 not upgraded.
Need to get 38.9 MB of archives.
After this operation, 192 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu groovy-proposed/universe amd64 docker.io amd64 19.03.13-0ubuntu3.1 [38.9 MB]
Fetched 38.9 MB in 2min 55s (222 kB/s)
Preconfiguring packages ...
Selecting previously unselected package docker.io.
(Reading database ... 96842 files and directories currently installed.)
Preparing to unpack .../docker.io_19.03.13-0ubuntu3.1_amd64.deb ...
Unpacking docker.io (19.03.13-0ubuntu3.1) ...
Setting up docker.io (19.03.13-0ubuntu3.1) ...
/var/lib/dpkg/info/docker.io.postinst: 31: zfs: not found
docker.service is a disabled or a static unit, not starting it.
Processing triggers for man-db (2.9.3-2) ...
ubuntu@docker-groovy:~$ sudo docker run -e REDIS_PASSWORD=1234 --network host --name test-redis -d ubuntu/redis:latest
060cdb80dd35caae1938792e3e07ef4615d111ce78f5f428463ab50e462e1240
ubuntu@docker-groovy:~$ sudo docker container logs test-redis

1:C 04 Jan 2021 19:47:38.016 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 04 Jan 2021 19:47:38.016 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 04 Jan 2021 19:47:38.016 # Configuration loaded
                _._
           _.-``__ ''-._
      _.-`` `. `_. ''-._ Redis 5.0.7 (00000000/0) 64 bit
  .-`` .-```. ```\/ _.,_ ''-._
 ( ' , .-` | `, ) Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
 | `-._ `._ / _.-' | PID: 1
  `-._ `-._ `-./ _.-' _.-'
 |`-._`-._ `-.__.-' _.-'_.-'|
 | `-._`-._ _.-'_.-' | http://redis.io
  `-._ `-._`-.__.-'_.-' _.-'
 |`-._`-._ `-.__.-' _.-'_.-'|
 | `-._`-._ _.-'_.-' |
  `-._ `-._`-.__.-'_.-' _.-'
      `-._ `-.__.-' _.-'
          `-._ _.-'
              `-.__.-'

1:M 04 Jan 2021 19:47:38.017 # Server initialized
1:M 04 Jan 2021 19:47:38.017 # WARNING overcommit_memory i...

[Łukasz Zemczak (sil2100)]

Ok, the verification here seems sane - at least currently I don't know what more can be done to ensure this does not cause another outage during upgrade. It also baked quite a while in -proposed already, so I'll simply proceed with the release.

Please keep a lookout for any new reports regarding this update.

[Łukasz Zemczak (sil2100)]

...actually, before proceeding, I see that containerd FTBFS on riscv now. Seems like a test issue, let me re-run the build. Please always be sure to check if the SRU builds on all previous architectures.

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for containerd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package docker.io - 19.03.8-0ubuntu1.20.04.2

---------------
docker.io (19.03.8-0ubuntu1.20.04.2) focal; urgency=medium

  * d/p/do_not_bind_docker_to_containerd.patch: Update docker.io to not
    stop when containerd is upgraded, by using Wants= rather than BindTo=.
    (LP: #1870514)
  * d/rules: Fix docker.io to not restart its service during package
    upgrades, to prevent service downtime from automatic updates via
    unattended-upgrade.
    (LP: #1906364)

 -- Bryce Harrington <email address hidden> Fri, 04 Dec 2020 23:02:49 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package containerd - 1.3.3-0ubuntu1~18.04.4

---------------
containerd (1.3.3-0ubuntu1~18.04.4) bionic; urgency=medium

  * d/control: add a Breaks for docker.io lower than 19.03.6-0ubuntu1~18.04.3
    (LP: #1870514). The previous versions stop the docker daemon when a
    containerd update is performed, this Breaks statement will make sure we
    have a newer version which has the appropriate fix.

 -- Lucas Kanashiro <email address hidden> Fri, 11 Dec 2020 09:19:07 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package docker.io - 19.03.6-0ubuntu1~18.04.3

---------------
docker.io (19.03.6-0ubuntu1~18.04.3) bionic; urgency=medium

  [ Bryce Harrington ]
  * d/p/do_not_bind_docker_to_containerd.patch: Update docker.io to not
    stop when containerd is upgraded, by using Wants= rather than BindTo=.
    (LP: #1870514)
  * d/rules: Fix docker.io to not restart its service during package
    upgrades, to prevent service downtime from automatic updates via
    unattended-upgrade.
    (LP: #1906364)

  [ Lucas Kanashiro ]
  * Do not build depend on libbtrfs-dev, it is not available in Bionic.

 -- Lucas Kanashiro <email address hidden> Thu, 10 Dec 2020 10:23:49 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package containerd - 1.3.7-0ubuntu3.2

---------------
containerd (1.3.7-0ubuntu3.2) groovy; urgency=medium

  * d/control: add a Breaks for docker.io lower than 19.03.13-0ubuntu3.1
    (LP: #1870514). The previous versions stop the docker daemon when a
    containerd update is performed, this Breaks statement will make sure we
    have a newer version which has the appropriate fix.

 -- Lucas Kanashiro <email address hidden> Fri, 11 Dec 2020 09:19:07 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package docker.io - 18.09.7-0ubuntu1~16.04.7

---------------
docker.io (18.09.7-0ubuntu1~16.04.7) xenial; urgency=medium

  * d/p/do_not_bind_docker_to_containerd.patch: Update docker.io to not
    stop when containerd is upgraded, by using Wants= rather than BindTo=.
    (LP: #1870514)
  * d/rules: Fix docker.io to not restart its service during package
    upgrades, to prevent service downtime from automatic updates via
    unattended-upgrade.
    (LP: #1906364)

 -- Bryce Harrington <email address hidden> Fri, 04 Dec 2020 23:02:49 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package containerd - 1.3.3-0ubuntu2.2

---------------
containerd (1.3.3-0ubuntu2.2) focal; urgency=medium

  * d/control: add a Breaks for docker.io lower than 19.03.8-0ubuntu1.20.04.2
    (LP: #1870514). The previous versions stop the docker daemon when a
    containerd update is performed, this Breaks statement will make sure we
    have a newer version which has the appropriate fix.

 -- Lucas Kanashiro <email address hidden> Fri, 11 Dec 2020 09:19:07 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package docker.io - 19.03.13-0ubuntu3.1

---------------
docker.io (19.03.13-0ubuntu3.1) groovy; urgency=medium

  * d/p/do_not_bind_docker_to_containerd.patch: Update docker.io to not
    stop when containerd is upgraded, by using Wants= rather than BindTo=.
    (LP: #1870514)
  * d/rules: Fix docker.io to not restart its service during package
    upgrades, to prevent service downtime from automatic updates via
    unattended-upgrade.
    (LP: #1906364)

 -- Bryce Harrington <email address hidden> Fri, 04 Dec 2020 23:02:49 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package containerd - 1.2.6-0ubuntu1~16.04.6

---------------
containerd (1.2.6-0ubuntu1~16.04.6) xenial; urgency=medium

  * d/control: add a Breaks for docker.io lower than 18.09.7-0ubuntu1~16.04.7
    (LP: #1870514). The previous versions stop the docker daemon when a
    containerd update is performed, this Breaks statement will make sure we
    have a newer version which has the appropriate fix.

 -- Lucas Kanashiro <email address hidden> Fri, 11 Dec 2020 09:19:07 -0300