| 2020-12-01 07:34:53 |
sascha arthur |
bug |
|
|
added bug |
| 2020-12-01 07:36:58 |
sascha arthur |
description |
Hello,
Today plenty of our systems running ubuntu 20.04 were restarting the docker daemon, even if i blacklisted the docker package. Since docker has an dependency on containerd thats the reason why it was restarted. IMO the blacklist should also check the full tree of dependencies... This should NOT happen!
From the log you find:
2020-12-01 06:40:13,881 INFO Starting unattended upgrades script
2020-12-01 06:40:13,882 INFO Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,a=focal-security, o=UbuntuESMApps,a=focal-apps-security, o=UbuntuESM,a=focal-infra-security
2020-12-01 06:40:13,882 INFO Initial blacklist: docker docker.io
2020-12-01 06:40:13,882 INFO Initial whitelist (not strict):
2020-12-01 06:40:19,139 INFO Packages that will be upgraded: containerd qemu-block-extra qemu-kvm qemu-system-common qemu-system-data qemu-system-gui qemu-system-x86 qemu-utils
2020-12-01 06:40:19,140 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2020-12-01 06:40:46,996 INFO All upgrades installed
2020-12-01 06:40:50,732 INFO Starting unattended upgrades script
2020-12-01 06:40:50,732 INFO Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,a=focal-security, o=UbuntuESMApps,a=focal-apps-security, o=UbuntuESM,a=focal-infra-security
2020-12-01 06:40:50,733 INFO Initial blacklist: docker docker.io
2020-12-01 06:40:50,733 INFO Initial whitelist (not strict):
Also this happened for us on plenty of our servers almost at the same (why the unattended updates are not spread over time?), which destroyed the second time an production environment.
This is not how unattended-upgraded should be, sadly this package lost our trust and we disable it and schedule the 'unattended updates' now on our own. |
Hello,
Today plenty of our systems running ubuntu 20.04 were restarting the docker daemon, even if i blacklisted the docker package. Since docker has an dependency on containerd thats the reason why it was restarted. IMO the blacklist should also check the full tree of dependencies... This should NOT happen!
From the log you find:
2020-12-01 06:40:13,881 INFO Starting unattended upgrades script
2020-12-01 06:40:13,882 INFO Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,a=focal-security, o=UbuntuESMApps,a=focal-apps-security, o=UbuntuESM,a=focal-infra-security
2020-12-01 06:40:13,882 INFO Initial blacklist: docker docker.io
2020-12-01 06:40:13,882 INFO Initial whitelist (not strict):
2020-12-01 06:40:19,139 INFO Packages that will be upgraded: containerd qemu-block-extra qemu-kvm qemu-system-common qemu-system-data qemu-system-gui qemu-system-x86 qemu-utils
2020-12-01 06:40:19,140 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2020-12-01 06:40:46,996 INFO All upgrades installed
2020-12-01 06:40:50,732 INFO Starting unattended upgrades script
2020-12-01 06:40:50,732 INFO Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,a=focal-security, o=UbuntuESMApps,a=focal-apps-security, o=UbuntuESM,a=focal-infra-security
2020-12-01 06:40:50,733 INFO Initial blacklist: docker docker.io
2020-12-01 06:40:50,733 INFO Initial whitelist (not strict):
Also this happened for us on plenty of our servers almost at the same (why the unattended updates are not spread over time?), which destroyed the second time an production environment.
This is not how unattended-upgraded should be, sadly this package lost our trust and we disable it and schedule the 'unattended updates' now on our own.
PS: Not to say that on some servers the docker daemon did not even restart.. |
|
| 2020-12-01 07:41:57 |
Orgad Shaneh |
bug |
|
|
added subscriber Orgad Shaneh |
| 2020-12-01 07:57:26 |
Launchpad Janitor |
unattended-upgrades (Ubuntu): status |
New |
Confirmed |
|
| 2020-12-01 08:58:16 |
Balint Reczey |
bug task added |
|
docker.io (Ubuntu) |
|
| 2020-12-01 09:11:29 |
Balint Reczey |
bug task added |
|
containerd (Ubuntu) |
|
| 2020-12-01 09:11:53 |
Balint Reczey |
unattended-upgrades (Ubuntu): status |
Confirmed |
Won't Fix |
|
| 2020-12-01 12:11:14 |
Launchpad Janitor |
containerd (Ubuntu): status |
New |
Confirmed |
|
| 2020-12-01 12:11:14 |
Launchpad Janitor |
docker.io (Ubuntu): status |
New |
Confirmed |
|
| 2020-12-02 18:53:05 |
Sergio Durigan Junior |
bug |
|
|
added subscriber Sergio Durigan Junior |
| 2020-12-04 23:51:56 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~bryce/ubuntu/+source/docker.io/+git/docker.io/+merge/394913 |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
nominated for series |
|
Ubuntu Xenial |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
bug task added |
|
unattended-upgrades (Ubuntu Xenial) |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
bug task added |
|
docker.io (Ubuntu Xenial) |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
bug task added |
|
containerd (Ubuntu Xenial) |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
nominated for series |
|
Ubuntu Hirsute |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
bug task added |
|
unattended-upgrades (Ubuntu Hirsute) |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
bug task added |
|
docker.io (Ubuntu Hirsute) |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
bug task added |
|
containerd (Ubuntu Hirsute) |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
nominated for series |
|
Ubuntu Bionic |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
bug task added |
|
unattended-upgrades (Ubuntu Bionic) |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
bug task added |
|
docker.io (Ubuntu Bionic) |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
bug task added |
|
containerd (Ubuntu Bionic) |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
nominated for series |
|
Ubuntu Groovy |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
bug task added |
|
unattended-upgrades (Ubuntu Groovy) |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
bug task added |
|
docker.io (Ubuntu Groovy) |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
bug task added |
|
containerd (Ubuntu Groovy) |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
nominated for series |
|
Ubuntu Focal |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
bug task added |
|
unattended-upgrades (Ubuntu Focal) |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
bug task added |
|
docker.io (Ubuntu Focal) |
|
| 2020-12-05 01:32:16 |
Bryce Harrington |
bug task added |
|
containerd (Ubuntu Focal) |
|
| 2020-12-05 01:32:44 |
Bryce Harrington |
unattended-upgrades (Ubuntu Groovy): status |
New |
Won't Fix |
|
| 2020-12-05 01:33:07 |
Bryce Harrington |
bug task deleted |
containerd (Ubuntu) |
|
|
| 2020-12-05 01:33:14 |
Bryce Harrington |
unattended-upgrades (Ubuntu Focal): status |
New |
Won't Fix |
|
| 2020-12-05 01:33:19 |
Bryce Harrington |
unattended-upgrades (Ubuntu Bionic): status |
New |
Won't Fix |
|
| 2020-12-05 01:33:23 |
Bryce Harrington |
unattended-upgrades (Ubuntu Xenial): status |
New |
Won't Fix |
|
| 2020-12-05 01:34:26 |
Bryce Harrington |
docker.io (Ubuntu Xenial): importance |
Undecided |
High |
|
| 2020-12-05 01:34:26 |
Bryce Harrington |
docker.io (Ubuntu Xenial): status |
New |
In Progress |
|
| 2020-12-05 01:34:26 |
Bryce Harrington |
docker.io (Ubuntu Xenial): assignee |
|
Bryce Harrington (bryce) |
|
| 2020-12-05 01:34:38 |
Bryce Harrington |
docker.io (Ubuntu Xenial): importance |
High |
Critical |
|
| 2020-12-05 01:34:53 |
Bryce Harrington |
docker.io (Ubuntu Bionic): importance |
Undecided |
Critical |
|
| 2020-12-05 01:34:53 |
Bryce Harrington |
docker.io (Ubuntu Bionic): status |
New |
In Progress |
|
| 2020-12-05 01:35:07 |
Bryce Harrington |
docker.io (Ubuntu Focal): importance |
Undecided |
Critical |
|
| 2020-12-05 01:35:07 |
Bryce Harrington |
docker.io (Ubuntu Focal): status |
New |
In Progress |
|
| 2020-12-05 01:35:20 |
Bryce Harrington |
docker.io (Ubuntu Groovy): importance |
Undecided |
Critical |
|
| 2020-12-05 01:35:20 |
Bryce Harrington |
docker.io (Ubuntu Groovy): status |
New |
In Progress |
|
| 2020-12-05 01:35:33 |
Bryce Harrington |
docker.io (Ubuntu Hirsute): importance |
Undecided |
Critical |
|
| 2020-12-05 01:35:33 |
Bryce Harrington |
docker.io (Ubuntu Hirsute): status |
Confirmed |
In Progress |
|
| 2020-12-05 01:35:33 |
Bryce Harrington |
docker.io (Ubuntu Hirsute): assignee |
|
Bryce Harrington (bryce) |
|
| 2020-12-05 01:35:58 |
Bryce Harrington |
bug task deleted |
containerd (Ubuntu Xenial) |
|
|
| 2020-12-05 01:36:05 |
Bryce Harrington |
bug task deleted |
containerd (Ubuntu Bionic) |
|
|
| 2020-12-05 01:36:10 |
Bryce Harrington |
bug task deleted |
containerd (Ubuntu Focal) |
|
|
| 2020-12-05 01:36:16 |
Bryce Harrington |
bug task deleted |
containerd (Ubuntu Groovy) |
|
|
| 2020-12-05 01:36:27 |
Bryce Harrington |
bug task deleted |
containerd (Ubuntu Hirsute) |
|
|
| 2020-12-05 11:19:03 |
SeySayux |
bug |
|
|
added subscriber SeySayux |
| 2020-12-05 16:02:11 |
Sascha Lucas |
bug |
|
|
added subscriber Sascha Lucas |
| 2020-12-09 16:19:44 |
Joshua Powers |
bug |
|
|
added subscriber Joshua Powers |
| 2020-12-10 07:16:52 |
Launchpad Janitor |
docker.io (Ubuntu Hirsute): status |
In Progress |
Fix Released |
|
| 2020-12-10 12:38:19 |
Lucas Kanashiro |
description |
Hello,
Today plenty of our systems running ubuntu 20.04 were restarting the docker daemon, even if i blacklisted the docker package. Since docker has an dependency on containerd thats the reason why it was restarted. IMO the blacklist should also check the full tree of dependencies... This should NOT happen!
From the log you find:
2020-12-01 06:40:13,881 INFO Starting unattended upgrades script
2020-12-01 06:40:13,882 INFO Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,a=focal-security, o=UbuntuESMApps,a=focal-apps-security, o=UbuntuESM,a=focal-infra-security
2020-12-01 06:40:13,882 INFO Initial blacklist: docker docker.io
2020-12-01 06:40:13,882 INFO Initial whitelist (not strict):
2020-12-01 06:40:19,139 INFO Packages that will be upgraded: containerd qemu-block-extra qemu-kvm qemu-system-common qemu-system-data qemu-system-gui qemu-system-x86 qemu-utils
2020-12-01 06:40:19,140 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2020-12-01 06:40:46,996 INFO All upgrades installed
2020-12-01 06:40:50,732 INFO Starting unattended upgrades script
2020-12-01 06:40:50,732 INFO Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,a=focal-security, o=UbuntuESMApps,a=focal-apps-security, o=UbuntuESM,a=focal-infra-security
2020-12-01 06:40:50,733 INFO Initial blacklist: docker docker.io
2020-12-01 06:40:50,733 INFO Initial whitelist (not strict):
Also this happened for us on plenty of our servers almost at the same (why the unattended updates are not spread over time?), which destroyed the second time an production environment.
This is not how unattended-upgraded should be, sadly this package lost our trust and we disable it and schedule the 'unattended updates' now on our own.
PS: Not to say that on some servers the docker daemon did not even restart.. |
[Impact]
Docker uses containerd under the hood. When containerd is upgraded it stops and restarts its service; docker stops when containerd stops but doesn’t restart. Particularly when doing unattended upgrades, an SRU fix rolled out for containerd can result in unexpected and widespread service outages for docker.
[Test Case]
$ sudo apt install docker.io
$ sudo systemctl start docker
$ systemctl status docker | grep Active
Active: active (running) since[...]
$ systemctl status containerd | grep Active
Active: active (running) since[...]
$ docker pull ubuntu/redis:latest
$ docker run -e REDIS_PASSWORD=1234 --network host \
--name test-redis -d ubuntu/redis:latest
$ telnet localhost 6379
$ docker container logs test-redis
$ sudo apt install --reinstall containerd
$ systemctl status containerd | grep Active
Active: active (running) since
$ systemctl status docker | grep Active
Active: inactive (dead) since [...]; 8s ago
$ docker container logs test-redis
[Where Problems Could Occur]
The challenge with this issue is addressing all important corner cases, and as such the biggest risk is that we miss a corner case and fail to keep the two services running when they should. Areas to watch will be failures during start/stop/restart/upgrade type operations. Issues during runtime are unlikely to relate to this change.
[Original Report]
Hello,
Today plenty of our systems running ubuntu 20.04 were restarting the docker daemon, even if i blacklisted the docker package. Since docker has an dependency on containerd thats the reason why it was restarted. IMO the blacklist should also check the full tree of dependencies... This should NOT happen!
From the log you find:
2020-12-01 06:40:13,881 INFO Starting unattended upgrades script
2020-12-01 06:40:13,882 INFO Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,a=focal-security, o=UbuntuESMApps,a=focal-apps-security, o=UbuntuESM,a=focal-infra-security
2020-12-01 06:40:13,882 INFO Initial blacklist: docker docker.io
2020-12-01 06:40:13,882 INFO Initial whitelist (not strict):
2020-12-01 06:40:19,139 INFO Packages that will be upgraded: containerd qemu-block-extra qemu-kvm qemu-system-common qemu-system-data qemu-system-gui qemu-system-x86 qemu-utils
2020-12-01 06:40:19,140 INFO Writing dpkg log to /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
2020-12-01 06:40:46,996 INFO All upgrades installed
2020-12-01 06:40:50,732 INFO Starting unattended upgrades script
2020-12-01 06:40:50,732 INFO Allowed origins are: o=Ubuntu,a=focal, o=Ubuntu,a=focal-security, o=UbuntuESMApps,a=focal-apps-security, o=UbuntuESM,a=focal-infra-security
2020-12-01 06:40:50,733 INFO Initial blacklist: docker docker.io
2020-12-01 06:40:50,733 INFO Initial whitelist (not strict):
Also this happened for us on plenty of our servers almost at the same (why the unattended updates are not spread over time?), which destroyed the second time an production environment.
This is not how unattended-upgraded should be, sadly this package lost our trust and we disable it and schedule the 'unattended updates' now on our own.
PS: Not to say that on some servers the docker daemon did not even restart.. |
|
| 2020-12-10 12:38:53 |
Lucas Kanashiro |
summary |
unattended-upgrade still restarts blacklisted daemons |
[SRU] unattended-upgrade still restarts blacklisted daemons |
|
| 2020-12-10 20:57:10 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~lucaskanashiro/ubuntu/+source/docker.io/+git/docker.io/+merge/395167 |
|
| 2020-12-10 20:57:48 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~lucaskanashiro/ubuntu/+source/docker.io/+git/docker.io/+merge/395168 |
|
| 2020-12-10 20:58:25 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~lucaskanashiro/ubuntu/+source/docker.io/+git/docker.io/+merge/395169 |
|
| 2020-12-10 21:15:22 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~lucaskanashiro/ubuntu/+source/docker.io/+git/docker.io/+merge/395171 |
|
| 2020-12-11 20:35:59 |
Lucas Kanashiro |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2020-12-18 11:57:59 |
Timo Aaltonen |
docker.io (Ubuntu Groovy): status |
In Progress |
Fix Committed |
|
| 2020-12-18 11:58:04 |
Timo Aaltonen |
bug |
|
|
added subscriber SRU Verification |
| 2020-12-18 11:58:10 |
Timo Aaltonen |
tags |
|
verification-needed verification-needed-groovy |
|
| 2020-12-18 12:14:37 |
Timo Aaltonen |
docker.io (Ubuntu Focal): status |
In Progress |
Fix Committed |
|
| 2020-12-18 12:14:45 |
Timo Aaltonen |
tags |
verification-needed verification-needed-groovy |
verification-needed verification-needed-focal verification-needed-groovy |
|
| 2020-12-18 12:19:45 |
Timo Aaltonen |
docker.io (Ubuntu Bionic): status |
In Progress |
Fix Committed |
|
| 2020-12-18 12:19:54 |
Timo Aaltonen |
tags |
verification-needed verification-needed-focal verification-needed-groovy |
verification-needed verification-needed-bionic verification-needed-focal verification-needed-groovy |
|
| 2020-12-18 12:31:03 |
Timo Aaltonen |
docker.io (Ubuntu Xenial): status |
In Progress |
Fix Committed |
|
| 2020-12-18 12:31:15 |
Timo Aaltonen |
tags |
verification-needed verification-needed-bionic verification-needed-focal verification-needed-groovy |
verification-needed verification-needed-bionic verification-needed-focal verification-needed-groovy verification-needed-xenial |
|
| 2020-12-23 08:35:12 |
Mathew Hodson |
bug task deleted |
unattended-upgrades (Ubuntu) |
|
|
| 2020-12-23 08:35:24 |
Mathew Hodson |
bug task deleted |
unattended-upgrades (Ubuntu Xenial) |
|
|
| 2020-12-23 08:35:31 |
Mathew Hodson |
bug task deleted |
unattended-upgrades (Ubuntu Bionic) |
|
|
| 2020-12-23 08:35:47 |
Mathew Hodson |
bug task deleted |
unattended-upgrades (Ubuntu Focal) |
|
|
| 2020-12-23 08:35:53 |
Mathew Hodson |
bug task deleted |
unattended-upgrades (Ubuntu Groovy) |
|
|
| 2020-12-23 08:35:59 |
Mathew Hodson |
bug task deleted |
unattended-upgrades (Ubuntu Hirsute) |
|
|
| 2021-01-04 20:09:00 |
Lucas Kanashiro |
tags |
verification-needed verification-needed-bionic verification-needed-focal verification-needed-groovy verification-needed-xenial |
verification-done verification-done-bionic verification-done-focal verification-done-groovy verification-done-xenial |
|
| 2021-01-07 09:54:42 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2021-01-07 09:58:07 |
Launchpad Janitor |
docker.io (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2021-01-07 09:58:36 |
Launchpad Janitor |
docker.io (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2021-01-07 10:07:40 |
Launchpad Janitor |
docker.io (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2021-01-07 10:14:41 |
Launchpad Janitor |
docker.io (Ubuntu Groovy): status |
Fix Committed |
Fix Released |
|
