[SRU] unattended-upgrade still restarts blacklisted daemons

[sascha arthur]

[Impact]

Docker uses containerd under the hood. When containerd is upgraded it stops and restarts its service; docker stops when containerd stops but doesn’t restart. Particularly when doing unattended upgrades, an SRU fix rolled out for containerd can result in unexpected and widespread service outages for docker.

[Test Case]

$ sudo apt install docker.io
$ sudo systemctl start docker
$ systemctl status docker | grep Active
     Active: active (running) since[...]
$ systemctl status containerd | grep Active
     Active: active (running) since[...]

$ docker pull ubuntu/redis:latest
$ docker run -e REDIS_PASSWORD=1234 --network host \
--name test-redis -d ubuntu/redis:latest
$ telnet localhost 6379
$ docker container logs test-redis

$ sudo apt install --reinstall containerd
$ systemctl status containerd | grep Active
     Active: active (running) since
$ systemctl status docker | grep Active
     Active: inactive (dead) since [...]; 8s ago
$ docker container logs test-redis

[Where Problems Could Occur]

The challenge with this issue is addressing all important corner cases, and as such the biggest risk is that we miss a corner case and fail to keep the two services running when they should. Areas to watch will be failures during start/stop/restart/upgrade type operations. Issues during runtime are unlikely to relate to this change.

[Original Report]

Hello,

Today plenty of our systems running ubuntu 20.04 were restarting the docker daemon, even if i blacklisted the docker package. Since docker has an dependency on containerd thats the reason why it was restarted. IMO the blacklist should also check the full tree of dependencies... This should NOT happen!

From the log you find:

2020-12-01 06:40:13,881 INFO Starting unattended upgrades script
2020-12-01 06:40:13,882 INFO Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,a=focal-security, o=UbuntuESMApps,a=focal-apps-security, o=UbuntuESM,a=focal-infra-security
2020-12-01 06:40:13,882 INFO Initial blacklist: docker docker.io
2020-12-01 06:40:13,882 INFO Initial whitelist (not strict):
2020-12-01 06:40:19,139 INFO Packages that will be upgraded: containerd qemu-block-extra qemu-kvm qemu-system-common qemu-system-data qemu-system-gui qemu-system-x86 qemu-utils
2020-12-01 06:40:19,140 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2020-12-01 06:40:46,996 INFO All upgrades installed
2020-12-01 06:40:50,732 INFO Starting unattended upgrades script
2020-12-01 06:40:50,732 INFO Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,a=focal-security, o=UbuntuESMApps,a=focal-apps-security, o=UbuntuESM,a=focal-infra-security
2020-12-01 06:40:50,733 INFO Initial blacklist: docker docker.io
2020-12-01 06:40:50,733 INFO Initial whitelist (not strict):

Also this happened for us on plenty of our servers almost at the same (why the unattended updates are not spread over time?), which destroyed the second time an production environment.

This is not how unattended-upgraded should be, sadly this package lost our trust and we disable it and schedule the 'unattended updates' now on our own.

PS: Not to say that on some servers the docker daemon did not even restart..

[Orgad Shaneh (orgads)]

We have at least 4 servers on which docker daemon was shutdown due to this unattended upgrade, and was not restarted.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Balint Reczey (rbalint)]

I'm sorry you experienced problems when unattended-upgrades applied security fixes.

Failure to restart a service is either an issue with the package or with the local configuration, thus I mark docker.io and containerd being affected.

The blacklist is listing _packages_ not services and not applied transitively to dependencies nor reverse dependencies, thus unattended-upgraded was and is expected to upgrade containerd.

Please list containerd in the blacklist if you would like to prevent it to be upgraded and trigger a docker restart.

[Balint Reczey (rbalint)]

Containerd unsuccessfully restarting docker is tracked at LP: #1870514.

[sascha arthur (sarthur)]

Sorry but im not really on your side here and it also doesnt really solve the problem.

I added docker.io to the blacklist because i want to take care manually about the upgrades, it should leave in any case this package alone.

Adding here dependencies (containerd) for the packages will never solve the issue, also i have to maintain this list, because dependencies (at least in theorie) could change over time.

Just to prevent docker.io from restart i have to add a huge set of packages (out of the list here https://packages.ubuntu.com/focal/docker.io) to prevent it from updating.

On top i have to go through all of the first level packages, to check if it has dependencies on the next level, because those could trigger a restart of the first level package, which will chain on the package which i try to prevent.

As far as i get it currently, if unattended-upgrades is not considering dependencies of blacklisted packages, theres no way except building an incredibly huge list of packages which you dont want to update, and regulary recalculate this list, because of maybe changed dependencies..

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Bryce Harrington (bryce)]

This sounds like another dupe of https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1870514

[Bryce Harrington (bryce)]

I'd like to give you all an update and outline our plans for this.

The Canonical server team has made analysis of this issue a top priority. We've identified and tested out several possible theories. Our findings suggest that the breakage involves two distinct issues, one the BindTo= issue mentioned above, the other caused by a bug in the docker.io package causing the service to stop on package upgrade; see specifically the service stop command at the end of /var/lib/dpkg/info/docker.io.prerm. We'll use LP: #1870514 to track the former issue, and #1906364 the latter. LP: #1658691 gives some past background for reference.

The tricky part is that unfortunately any change we make to docker.io requires the running of the prerm script (the version of the script already present on your system, not the one we'd be installing), and thus triggers the bug. In other words, updating your system to prevent the bug will cause one more docker stop. Thereafter, the upgrade will not restart the service when rolling out CVE fixes to either containerd or docker.io; it may prompt to do so if running interactively (e.g. https://imgur.com/2Za5dbQ.png), otherwise it should respect the debconf setting.

We would appreciate feedback, testing and/or review of the proposed fix, available in this PPA:

   https://launchpad.net/~bryce/+archive/ubuntu/containerd-sru-lp1870514-docker-dh/

[sascha arthur (sarthur)]

Thanks @bryce for the detailed updates.

For us personally its the second time we run into a docker daemon restart on production (including all container restarts).

The only reason why this time it got much more attention is that a bug prevented a clean restart of the daemon, leaving containers offline.

This fact tells us that most of the people never noticed that their containers were also during the last update restartet. (Which is fine because mostly a short restart, for example of a webserver nobody would notice)

This is also fine of most of our servers because they running lightweight containers.

The only reason why we noticed it (this and last time) is that we running heavy database containers (for example elasticsearch with couple of TBs) which have a restart time of minutes instead of seconds.

Especially its critical (and loosing consistency) when containers of the same kind get restarted at the same time.

All of this chain is triggered by the unattended updates. Thats why we excluded the automatic updates for the docker package.

But this doesnt help if depending package updates still restarting the docker daemon/containers.

Since the unattended package is still marked as 'wont-fix', and the dependencies problem is explained above..

The only way to safely prevent docker daemon restarts is to fully disable the whole unattended updates? Can someone confirm this?

[Launchpad Janitor (janitor)]

This bug was fixed in the package docker.io - 19.03.13-0ubuntu4

---------------
docker.io (19.03.13-0ubuntu4) hirsute; urgency=medium

  * d/p/do_not_bind_docker_to_containerd.patch: Update docker.io to not
    stop when containerd is upgraded, by using Wants= rather than BindTo=.
    (LP: #1870514)
  * d/rules: Fix docker.io to not restart its service during package
    upgrades, to prevent service downtime from automatic updates via
    unattended-upgrade.
    (LP: #1906364)

 -- Bryce Harrington <email address hidden> Fri, 04 Dec 2020 23:02:49 +0000

[Timo Aaltonen (tjaalton)]

Hello sascha, or anyone else affected,

Accepted docker.io into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/docker.io/19.03.13-0ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello sascha, or anyone else affected,

Accepted docker.io into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/docker.io/19.03.8-0ubuntu1.20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello sascha, or anyone else affected,

Accepted docker.io into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/docker.io/19.03.6-0ubuntu1~18.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello sascha, or anyone else affected,

Accepted docker.io into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/docker.io/18.09.7-0ubuntu1~16.04.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted docker.io (18.09.7-0ubuntu1~16.04.7) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

docker.io/18.09.7-0ubuntu1~16.04.7 (amd64, arm64, s390x, i386, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#docker.io

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted docker.io (19.03.6-0ubuntu1~18.04.3) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-fan/0.12.10 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#docker.io

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[sascha arthur (sarthur)]

Thanks for beeing fully ignored..

Well see us on the next production outtake..

[Lucas Kanashiro (lucaskanashiro)]

Hi @sarthur, I am sorry if you felt ignored, this is not the Ubuntu community intention. What you described seems to not be supported by unattended-upgrades, therefore, it'd be a whishlist bug (feature request) against it. I'd recommend to file this bug against Debian and discuss it with the maintainers (they are the upstream), and if/when the feature lands there we will be happy to sync it.

[Lucas Kanashiro (lucaskanashiro)]

The autopkgtest regressions reported above were handled. The ubuntu-fan test failure was fixed after re-triggering it manually, and the docker.io test in Xenial was indeed failing and I just uploaded version 18.09.7-0ubuntu1~16.04.8 fixing it. With that in place all the tests should be passing.

[Lucas Kanashiro (lucaskanashiro)]

I have tested the scenario described in the Test Case section in the bug description through all the target releases and they all seem good:

# Groovy

ubuntu@docker-groovy:~$ sudo apt install -t groovy-proposed docker.io
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  aufs-tools cgroupfs-mount | cgroup-lite debootstrap docker-doc rinse zfs-fuse | zfsutils
The following NEW packages will be installed:
  docker.io
0 upgraded, 1 newly installed, 0 to remove and 15 not upgraded.
Need to get 38.9 MB of archives.
After this operation, 192 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu groovy-proposed/universe amd64 docker.io amd64 19.03.13-0ubuntu3.1 [38.9 MB]
Fetched 38.9 MB in 2min 55s (222 kB/s)
Preconfiguring packages ...
Selecting previously unselected package docker.io.
(Reading database ... 96842 files and directories currently installed.)
Preparing to unpack .../docker.io_19.03.13-0ubuntu3.1_amd64.deb ...
Unpacking docker.io (19.03.13-0ubuntu3.1) ...
Setting up docker.io (19.03.13-0ubuntu3.1) ...
/var/lib/dpkg/info/docker.io.postinst: 31: zfs: not found
docker.service is a disabled or a static unit, not starting it.
Processing triggers for man-db (2.9.3-2) ...
ubuntu@docker-groovy:~$ sudo docker run -e REDIS_PASSWORD=1234 --network host --name test-redis -d ubuntu/redis:latest
060cdb80dd35caae1938792e3e07ef4615d111ce78f5f428463ab50e462e1240
ubuntu@docker-groovy:~$ sudo docker container logs test-redis

1:C 04 Jan 2021 19:47:38.016 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 04 Jan 2021 19:47:38.016 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 04 Jan 2021 19:47:38.016 # Configuration loaded
                _._
           _.-``__ ''-._
      _.-`` `. `_. ''-._ Redis 5.0.7 (00000000/0) 64 bit
  .-`` .-```. ```\/ _.,_ ''-._
 ( ' , .-` | `, ) Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
 | `-._ `._ / _.-' | PID: 1
  `-._ `-._ `-./ _.-' _.-'
 |`-._`-._ `-.__.-' _.-'_.-'|
 | `-._`-._ _.-'_.-' | http://redis.io
  `-._ `-._`-.__.-'_.-' _.-'
 |`-._`-._ `-.__.-' _.-'_.-'|
 | `-._`-._ _.-'_.-' |
  `-._ `-._`-.__.-'_.-' _.-'
      `-._ `-.__.-' _.-'
          `-._ _.-'
              `-.__.-'

1:M 04 Jan 2021 19:47:38.017 # Server initialized
1:M 04 Jan 2021 19:47:38.017 # WARNING overcommit_memory i...

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for docker.io has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package docker.io - 19.03.8-0ubuntu1.20.04.2

---------------
docker.io (19.03.8-0ubuntu1.20.04.2) focal; urgency=medium

  * d/p/do_not_bind_docker_to_containerd.patch: Update docker.io to not
    stop when containerd is upgraded, by using Wants= rather than BindTo=.
    (LP: #1870514)
  * d/rules: Fix docker.io to not restart its service during package
    upgrades, to prevent service downtime from automatic updates via
    unattended-upgrade.
    (LP: #1906364)

 -- Bryce Harrington <email address hidden> Fri, 04 Dec 2020 23:02:49 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package docker.io - 19.03.6-0ubuntu1~18.04.3

---------------
docker.io (19.03.6-0ubuntu1~18.04.3) bionic; urgency=medium

  [ Bryce Harrington ]
  * d/p/do_not_bind_docker_to_containerd.patch: Update docker.io to not
    stop when containerd is upgraded, by using Wants= rather than BindTo=.
    (LP: #1870514)
  * d/rules: Fix docker.io to not restart its service during package
    upgrades, to prevent service downtime from automatic updates via
    unattended-upgrade.
    (LP: #1906364)

  [ Lucas Kanashiro ]
  * Do not build depend on libbtrfs-dev, it is not available in Bionic.

 -- Lucas Kanashiro <email address hidden> Thu, 10 Dec 2020 10:23:49 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package docker.io - 18.09.7-0ubuntu1~16.04.7

---------------
docker.io (18.09.7-0ubuntu1~16.04.7) xenial; urgency=medium

  * d/p/do_not_bind_docker_to_containerd.patch: Update docker.io to not
    stop when containerd is upgraded, by using Wants= rather than BindTo=.
    (LP: #1870514)
  * d/rules: Fix docker.io to not restart its service during package
    upgrades, to prevent service downtime from automatic updates via
    unattended-upgrade.
    (LP: #1906364)

 -- Bryce Harrington <email address hidden> Fri, 04 Dec 2020 23:02:49 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package docker.io - 19.03.13-0ubuntu3.1

---------------
docker.io (19.03.13-0ubuntu3.1) groovy; urgency=medium

  * d/p/do_not_bind_docker_to_containerd.patch: Update docker.io to not
    stop when containerd is upgraded, by using Wants= rather than BindTo=.
    (LP: #1870514)
  * d/rules: Fix docker.io to not restart its service during package
    upgrades, to prevent service downtime from automatic updates via
    unattended-upgrade.
    (LP: #1906364)

 -- Bryce Harrington <email address hidden> Fri, 04 Dec 2020 23:02:49 +0000