| 2018-06-22 12:47:11 |
Cs-gon |
bug |
|
|
added bug |
| 2018-06-22 12:47:36 |
Cs-gon |
tags |
|
bionic |
|
| 2018-06-22 13:51:55 |
Cs-gon |
tags |
bionic |
apport-collected bionic |
|
| 2018-06-22 13:51:56 |
Cs-gon |
description |
When using unattended-upgrades with "InstallOnShutdown" on Bionic, the package installation on various packages hangs until the systemd ShutdownTimeout (30min) is expired and systemd kills all processes and powers off/reboots the system.
This leaves packages in an unconfigured, broken state. At least sometimes this cannot be fixed with a "dpkg --configure -a", but instead requires the user to manually reinstall the package that caused the hang.
This appears to be a deadlock, because the hanging commands are always "systemctl stop ..." or "systemctl restart ...", etc.. If I understand this correctly, those systemctl commands block because systemd tries to shutdown the system and tries to satisfy all dependencies for the shutdown targets before those systemctl commands could get executed, which creates a deadlock.
Steps to reproduce:
- Install 18.04
- activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
- disable bionic-updates in /etc/apt/sources.list (more on that later)
- execute "unattended-upgrade --download-only"
- reboot the system
-> The upgrade on shutdown hangs when configuring the apport package. The hanging command is "systemctl stop apport-forward.socket". The system hangs until it the systemd ShutdownTimeout expires and systemd forcefully reboots the system.
After the system is rebooted the apport package is in "iHR" state, and needs to be reinstalled to fix this.
I disabled the bionic-updates pocket in sources.list, because in the default configuration unattended-updates does not use bionic-updates, and seems to have skipped installation of apport from bionic-security (supposedly because an already newer version of apport was in bionic-updates). If my understanding of why apport initially did not get installed is correct, then this would be another problem, because it would mean that unattended-upgrades potentially does not install all available security updates when bionic-updates is enabled in sources.list (which is the default).
The problem can also be reproduced without disabling bionic-updates in sources.list, but instead enabling bionic-updates in /etc/apt/apt.conf.d/50unattended-upgrades. Then, in my case, the upgrade did hang when installing the package snapd (the hanging command was "systemctl stop snapd.autoimport.service snapd.core-fixup.service snapd.service snapd.snap-repair.service snapd.snap-repair.service snapd.socket snapd.system-shutdown.service"). This leads to the same problems as described above. |
When using unattended-upgrades with "InstallOnShutdown" on Bionic, the package installation on various packages hangs until the systemd ShutdownTimeout (30min) is expired and systemd kills all processes and powers off/reboots the system.
This leaves packages in an unconfigured, broken state. At least sometimes this cannot be fixed with a "dpkg --configure -a", but instead requires the user to manually reinstall the package that caused the hang.
This appears to be a deadlock, because the hanging commands are always "systemctl stop ..." or "systemctl restart ...", etc.. If I understand this correctly, those systemctl commands block because systemd tries to shutdown the system and tries to satisfy all dependencies for the shutdown targets before those systemctl commands could get executed, which creates a deadlock.
Steps to reproduce:
- Install 18.04
- activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
- disable bionic-updates in /etc/apt/sources.list (more on that later)
- execute "unattended-upgrade --download-only"
- reboot the system
-> The upgrade on shutdown hangs when configuring the apport package. The hanging command is "systemctl stop apport-forward.socket". The system hangs until it the systemd ShutdownTimeout expires and systemd forcefully reboots the system.
After the system is rebooted the apport package is in "iHR" state, and needs to be reinstalled to fix this.
I disabled the bionic-updates pocket in sources.list, because in the default configuration unattended-updates does not use bionic-updates, and seems to have skipped installation of apport from bionic-security (supposedly because an already newer version of apport was in bionic-updates). If my understanding of why apport initially did not get installed is correct, then this would be another problem, because it would mean that unattended-upgrades potentially does not install all available security updates when bionic-updates is enabled in sources.list (which is the default).
The problem can also be reproduced without disabling bionic-updates in sources.list, but instead enabling bionic-updates in /etc/apt/apt.conf.d/50unattended-upgrades. Then, in my case, the upgrade did hang when installing the package snapd (the hanging command was "systemctl stop snapd.autoimport.service snapd.core-fixup.service snapd.service snapd.snap-repair.service snapd.snap-repair.service snapd.socket snapd.system-shutdown.service"). This leads to the same problems as described above.
---
ApportVersion: 2.20.9-0ubuntu7.1
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2018-06-22 (0 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
Package: unattended-upgrades 1.1ubuntu1
PackageArchitecture: all
ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17
Tags: bionic
Uname: Linux 4.15.0-20-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True |
|
| 2018-06-22 13:51:57 |
Cs-gon |
attachment added |
|
.var.log.apt.history.log.txt https://bugs.launchpad.net/bugs/1778219/+attachment/5155525/+files/.var.log.apt.history.log.txt |
|
| 2018-06-22 13:51:58 |
Cs-gon |
attachment added |
|
.var.log.unattended-upgrades.unattended-upgrades.log.txt https://bugs.launchpad.net/bugs/1778219/+attachment/5155526/+files/.var.log.unattended-upgrades.unattended-upgrades.log.txt |
|
| 2018-06-22 13:51:59 |
Cs-gon |
attachment added |
|
Dependencies.txt https://bugs.launchpad.net/bugs/1778219/+attachment/5155527/+files/Dependencies.txt |
|
| 2018-06-22 13:52:00 |
Cs-gon |
attachment added |
|
ProcCpuinfoMinimal.txt https://bugs.launchpad.net/bugs/1778219/+attachment/5155528/+files/ProcCpuinfoMinimal.txt |
|
| 2018-06-22 13:52:00 |
Cs-gon |
attachment added |
|
ProcEnviron.txt https://bugs.launchpad.net/bugs/1778219/+attachment/5155529/+files/ProcEnviron.txt |
|
| 2018-06-22 13:59:29 |
Cs-gon |
description |
When using unattended-upgrades with "InstallOnShutdown" on Bionic, the package installation on various packages hangs until the systemd ShutdownTimeout (30min) is expired and systemd kills all processes and powers off/reboots the system.
This leaves packages in an unconfigured, broken state. At least sometimes this cannot be fixed with a "dpkg --configure -a", but instead requires the user to manually reinstall the package that caused the hang.
This appears to be a deadlock, because the hanging commands are always "systemctl stop ..." or "systemctl restart ...", etc.. If I understand this correctly, those systemctl commands block because systemd tries to shutdown the system and tries to satisfy all dependencies for the shutdown targets before those systemctl commands could get executed, which creates a deadlock.
Steps to reproduce:
- Install 18.04
- activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
- disable bionic-updates in /etc/apt/sources.list (more on that later)
- execute "unattended-upgrade --download-only"
- reboot the system
-> The upgrade on shutdown hangs when configuring the apport package. The hanging command is "systemctl stop apport-forward.socket". The system hangs until it the systemd ShutdownTimeout expires and systemd forcefully reboots the system.
After the system is rebooted the apport package is in "iHR" state, and needs to be reinstalled to fix this.
I disabled the bionic-updates pocket in sources.list, because in the default configuration unattended-updates does not use bionic-updates, and seems to have skipped installation of apport from bionic-security (supposedly because an already newer version of apport was in bionic-updates). If my understanding of why apport initially did not get installed is correct, then this would be another problem, because it would mean that unattended-upgrades potentially does not install all available security updates when bionic-updates is enabled in sources.list (which is the default).
The problem can also be reproduced without disabling bionic-updates in sources.list, but instead enabling bionic-updates in /etc/apt/apt.conf.d/50unattended-upgrades. Then, in my case, the upgrade did hang when installing the package snapd (the hanging command was "systemctl stop snapd.autoimport.service snapd.core-fixup.service snapd.service snapd.snap-repair.service snapd.snap-repair.service snapd.socket snapd.system-shutdown.service"). This leads to the same problems as described above.
---
ApportVersion: 2.20.9-0ubuntu7.1
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2018-06-22 (0 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
Package: unattended-upgrades 1.1ubuntu1
PackageArchitecture: all
ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17
Tags: bionic
Uname: Linux 4.15.0-20-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True |
When using unattended-upgrades with "InstallOnShutdown" on Bionic, the package installation on various packages hangs until the systemd ShutdownTimeout (30min) is expired and systemd kills all processes and powers off/reboots the system.
This leaves packages in an unconfigured, broken state. At least sometimes this cannot be fixed with a "dpkg --configure -a", but instead requires the user to manually reinstall the package that caused the hang.
This appears to be a deadlock, because the hanging commands are always "systemctl stop ..." or "systemctl restart ...", etc.. If I understand this correctly, those systemctl commands block because systemd tries to shutdown the system and tries to satisfy all dependencies for the shutdown targets before those systemctl commands could get executed, which creates a deadlock.
Steps to reproduce:
- Install 18.04
- activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
- disable bionic-updates in /etc/apt/sources.list (more on that later)
- execute "unattended-upgrade --download-only"
- reboot the system
-> The upgrade on shutdown hangs when configuring the apport package. The hanging command is "systemctl stop apport-forward.socket". The system hangs until the systemd ShutdownTimeout expires and systemd forcefully reboots the system.
After the system is rebooted the apport package is in "iUR" state, and needs to be reinstalled to fix this.
I disabled the bionic-updates pocket in sources.list, because in the default configuration unattended-updates does not use bionic-updates, and seems to have skipped installation of apport from bionic-security (supposedly because an already newer version of apport was in bionic-updates). If my understanding of why apport initially did not get installed is correct, then this would be another problem, because it would mean that unattended-upgrades potentially does not install all available security updates when bionic-updates is enabled in sources.list (which is the default).
The problem can also be reproduced without disabling bionic-updates in sources.list, but instead enabling bionic-updates in /etc/apt/apt.conf.d/50unattended-upgrades. Then, in my case, the upgrade did hang when installing the package snapd (the hanging command was "systemctl stop snapd.autoimport.service snapd.core-fixup.service snapd.service snapd.snap-repair.service snapd.snap-repair.service snapd.socket snapd.system-shutdown.service"). This leads to the same problems as described above.
---
ApportVersion: 2.20.9-0ubuntu7.1
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2018-06-22 (0 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
Package: unattended-upgrades 1.1ubuntu1
PackageArchitecture: all
ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17
Tags: bionic
Uname: Linux 4.15.0-20-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True |
|
| 2018-06-22 22:46:03 |
Balint Reczey |
bug task added |
|
init-system-helpers (Ubuntu) |
|
| 2018-06-22 22:50:57 |
Balint Reczey |
bug task added |
|
snapd (Ubuntu) |
|
| 2018-06-22 22:51:27 |
Balint Reczey |
unattended-upgrades (Ubuntu): status |
New |
Confirmed |
|
| 2018-06-26 16:40:25 |
Brian Murray |
tags |
apport-collected bionic |
apport-collected bionic rls-cc-incoming |
|
| 2018-06-29 12:22:06 |
Francis Ginther |
tags |
apport-collected bionic rls-cc-incoming |
apport-collected bionic id-5b3568d38c766fbe4a6b1710 rls-cc-incoming |
|
| 2018-07-20 11:09:34 |
Balint Reczey |
init-system-helpers (Ubuntu): status |
New |
In Progress |
|
| 2018-07-20 11:09:38 |
Balint Reczey |
init-system-helpers (Ubuntu): assignee |
|
Balint Reczey (rbalint) |
|
| 2018-07-20 11:09:51 |
Balint Reczey |
init-system-helpers (Ubuntu): importance |
Undecided |
Critical |
|
| 2018-07-20 11:10:07 |
Balint Reczey |
init-system-helpers (Ubuntu): importance |
Critical |
High |
|
| 2018-07-25 09:15:09 |
Balint Reczey |
init-system-helpers (Ubuntu): status |
In Progress |
Invalid |
|
| 2018-07-25 09:15:12 |
Balint Reczey |
unattended-upgrades (Ubuntu): status |
Confirmed |
In Progress |
|
| 2018-07-25 09:36:17 |
Balint Reczey |
unattended-upgrades (Ubuntu): importance |
Undecided |
High |
|
| 2018-07-25 09:45:02 |
Balint Reczey |
snapd (Ubuntu): importance |
Undecided |
Low |
|
| 2018-07-25 14:18:59 |
Balint Reczey |
description |
When using unattended-upgrades with "InstallOnShutdown" on Bionic, the package installation on various packages hangs until the systemd ShutdownTimeout (30min) is expired and systemd kills all processes and powers off/reboots the system.
This leaves packages in an unconfigured, broken state. At least sometimes this cannot be fixed with a "dpkg --configure -a", but instead requires the user to manually reinstall the package that caused the hang.
This appears to be a deadlock, because the hanging commands are always "systemctl stop ..." or "systemctl restart ...", etc.. If I understand this correctly, those systemctl commands block because systemd tries to shutdown the system and tries to satisfy all dependencies for the shutdown targets before those systemctl commands could get executed, which creates a deadlock.
Steps to reproduce:
- Install 18.04
- activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
- disable bionic-updates in /etc/apt/sources.list (more on that later)
- execute "unattended-upgrade --download-only"
- reboot the system
-> The upgrade on shutdown hangs when configuring the apport package. The hanging command is "systemctl stop apport-forward.socket". The system hangs until the systemd ShutdownTimeout expires and systemd forcefully reboots the system.
After the system is rebooted the apport package is in "iUR" state, and needs to be reinstalled to fix this.
I disabled the bionic-updates pocket in sources.list, because in the default configuration unattended-updates does not use bionic-updates, and seems to have skipped installation of apport from bionic-security (supposedly because an already newer version of apport was in bionic-updates). If my understanding of why apport initially did not get installed is correct, then this would be another problem, because it would mean that unattended-upgrades potentially does not install all available security updates when bionic-updates is enabled in sources.list (which is the default).
The problem can also be reproduced without disabling bionic-updates in sources.list, but instead enabling bionic-updates in /etc/apt/apt.conf.d/50unattended-upgrades. Then, in my case, the upgrade did hang when installing the package snapd (the hanging command was "systemctl stop snapd.autoimport.service snapd.core-fixup.service snapd.service snapd.snap-repair.service snapd.snap-repair.service snapd.socket snapd.system-shutdown.service"). This leads to the same problems as described above.
---
ApportVersion: 2.20.9-0ubuntu7.1
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2018-06-22 (0 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
Package: unattended-upgrades 1.1ubuntu1
PackageArchitecture: all
ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17
Tags: bionic
Uname: Linux 4.15.0-20-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True |
[Impact]
* Unattended-upgrades hangs and gets killed when installing upgrades that stat/stop services on shutdown, leaving the system in a broken state
[Test Case]
* Install an updated bionic system:
$ lxc launch ubuntu:18.04 uu-shutdown-test
# apt update
...
* When testing the fixed version, install upgrade u-u at this point checking that u-u.service is set up before and is wanted by shutdown.target:
# systemd-analyze dot | grep unatt
"unattended-upgrades.service"->"-.mount" [color="green"];
"unattended-upgrades.service"->"system.slice" [color="green"];
"unattended-upgrades.service"->"network.target" [color="green"];
"unattended-upgrades.service"->"systemd-journald.socket" [color="green"];
"unattended-upgrades.service"->"local-fs.target" [color="green"];
"unattended-upgrades.service"->"-.mount" [color="black"];
"unattended-upgrades.service"->"system.slice" [color="black"];
"shutdown.target"->"unattended-upgrades.service" [color="green"];
"shutdown.target"->"unattended-upgrades.service" [color="grey66"];
Color legend: black = Requires
dark blue = Requisite
dark grey = Wants
red = Conflicts
green = After
* Configure u-u to run on shutdown and install -updates:
# echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
# echo 'Unattended-Upgrade::Allowed-Origins:: "${distro_id}:${distro_codename}-updates";' > /etc/apt/apt.conf.d/51unattended-upgrades-updates-too
* Downgrade snapd:
# apt install snapd=2.32.5+18.04
* # reboot
* With not fixed u-u observe the upgrade process being stuck:
# pstree| grep unatt
`-unattended-upgr---unattended-upgr-+-unattended-upgr---dpkg---snapd.prerm---systemctl
`-{unattended-upgr}
* With fixed u-u observe snapd update taking place and system rebooting after a few seconds with all updates installed
* Since this fix is partially reverting the fix for LP: #1654600 please test LP: #1654600 as well to avoid regressions.
[Regression Potential]
* As part of the fix manual changes were made to postinst to properly transition from coupling u-u.service with multi-user.target to coupling it with shutdown.target again which can make u-u started during normal boot when there is a bug in the implementation.
on-testing the SRU.
* Due to relationship changes between u-u.service, other services and targets u-u may fail to run on shutdown in case of an unexpected regression.
[Original Bug Text]
When using unattended-upgrades with "InstallOnShutdown" on Bionic, the package installation on various packages hangs until the systemd ShutdownTimeout (30min) is expired and systemd kills all processes and powers off/reboots the system.
This leaves packages in an unconfigured, broken state. At least sometimes this cannot be fixed with a "dpkg --configure -a", but instead requires the user to manually reinstall the package that caused the hang.
This appears to be a deadlock, because the hanging commands are always "systemctl stop ..." or "systemctl restart ...", etc.. If I understand this correctly, those systemctl commands block because systemd tries to shutdown the system and tries to satisfy all dependencies for the shutdown targets before those systemctl commands could get executed, which creates a deadlock.
Steps to reproduce:
- Install 18.04
- activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
- disable bionic-updates in /etc/apt/sources.list (more on that later)
- execute "unattended-upgrade --download-only"
- reboot the system
-> The upgrade on shutdown hangs when configuring the apport package. The hanging command is "systemctl stop apport-forward.socket". The system hangs until the systemd ShutdownTimeout expires and systemd forcefully reboots the system.
After the system is rebooted the apport package is in "iUR" state, and needs to be reinstalled to fix this.
I disabled the bionic-updates pocket in sources.list, because in the default configuration unattended-updates does not use bionic-updates, and seems to have skipped installation of apport from bionic-security (supposedly because an already newer version of apport was in bionic-updates). If my understanding of why apport initially did not get installed is correct, then this would be another problem, because it would mean that unattended-upgrades potentially does not install all available security updates when bionic-updates is enabled in sources.list (which is the default).
The problem can also be reproduced without disabling bionic-updates in sources.list, but instead enabling bionic-updates in /etc/apt/apt.conf.d/50unattended-upgrades. Then, in my case, the upgrade did hang when installing the package snapd (the hanging command was "systemctl stop snapd.autoimport.service snapd.core-fixup.service snapd.service snapd.snap-repair.service snapd.snap-repair.service snapd.socket snapd.system-shutdown.service"). This leads to the same problems as described above.
---
ApportVersion: 2.20.9-0ubuntu7.1
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2018-06-22 (0 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
Package: unattended-upgrades 1.1ubuntu1
PackageArchitecture: all
ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17
Tags: bionic
Uname: Linux 4.15.0-20-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True |
|
| 2018-08-28 15:50:32 |
Launchpad Janitor |
unattended-upgrades (Ubuntu): status |
In Progress |
Fix Released |
|
| 2018-10-12 20:55:29 |
Balint Reczey |
init-system-helpers (Ubuntu): assignee |
Balint Reczey (rbalint) |
|
|
| 2018-10-14 10:55:10 |
Balint Reczey |
bug task added |
|
apt (Ubuntu) |
|
| 2018-10-14 20:14:52 |
Balint Reczey |
apt (Ubuntu): status |
New |
Won't Fix |
|
| 2018-10-14 20:15:05 |
Balint Reczey |
apt (Ubuntu): status |
Won't Fix |
Invalid |
|
| 2018-11-01 23:07:22 |
Eric Desrochers |
nominated for series |
|
Ubuntu Bionic |
|
| 2018-11-01 23:07:22 |
Eric Desrochers |
bug task added |
|
apt (Ubuntu Bionic) |
|
| 2018-11-01 23:07:22 |
Eric Desrochers |
bug task added |
|
unattended-upgrades (Ubuntu Bionic) |
|
| 2018-11-01 23:07:22 |
Eric Desrochers |
bug task added |
|
init-system-helpers (Ubuntu Bionic) |
|
| 2018-11-01 23:07:22 |
Eric Desrochers |
bug task added |
|
snapd (Ubuntu Bionic) |
|
| 2018-11-01 23:07:30 |
Eric Desrochers |
unattended-upgrades (Ubuntu Bionic): status |
New |
Confirmed |
|
| 2018-11-02 18:47:34 |
Eric Desrochers |
unattended-upgrades (Ubuntu Bionic): status |
Confirmed |
In Progress |
|
| 2018-11-02 18:47:38 |
Eric Desrochers |
unattended-upgrades (Ubuntu Bionic): importance |
Undecided |
Medium |
|
| 2018-11-02 18:47:40 |
Eric Desrochers |
unattended-upgrades (Ubuntu Bionic): assignee |
|
Eric Desrochers (slashd) |
|
| 2018-11-06 12:35:48 |
Eric Desrochers |
unattended-upgrades (Ubuntu Bionic): assignee |
Eric Desrochers (slashd) |
|
|
| 2018-11-06 12:35:58 |
Eric Desrochers |
unattended-upgrades (Ubuntu Bionic): assignee |
|
Balint Reczey (rbalint) |
|
| 2018-11-06 13:06:56 |
Eric Desrochers |
nominated for series |
|
Ubuntu Xenial |
|
| 2018-11-06 13:06:56 |
Eric Desrochers |
bug task added |
|
apt (Ubuntu Xenial) |
|
| 2018-11-06 13:06:56 |
Eric Desrochers |
bug task added |
|
unattended-upgrades (Ubuntu Xenial) |
|
| 2018-11-06 13:06:56 |
Eric Desrochers |
bug task added |
|
init-system-helpers (Ubuntu Xenial) |
|
| 2018-11-06 13:06:56 |
Eric Desrochers |
bug task added |
|
snapd (Ubuntu Xenial) |
|
| 2018-11-06 13:07:27 |
Eric Desrochers |
unattended-upgrades (Ubuntu Xenial): importance |
Undecided |
Medium |
|
| 2018-11-06 13:07:27 |
Eric Desrochers |
unattended-upgrades (Ubuntu Xenial): status |
New |
In Progress |
|
| 2018-11-06 13:07:27 |
Eric Desrochers |
unattended-upgrades (Ubuntu Xenial): assignee |
|
Balint Reczey (rbalint) |
|
| 2018-11-14 14:25:32 |
Launchpad Janitor |
snapd (Ubuntu): status |
New |
Confirmed |
|
| 2018-11-14 14:25:32 |
Launchpad Janitor |
apt (Ubuntu Xenial): status |
New |
Confirmed |
|
| 2018-11-14 14:25:32 |
Launchpad Janitor |
init-system-helpers (Ubuntu Xenial): status |
New |
Confirmed |
|
| 2018-11-14 14:25:32 |
Launchpad Janitor |
snapd (Ubuntu Xenial): status |
New |
Confirmed |
|
| 2018-11-14 14:25:32 |
Launchpad Janitor |
apt (Ubuntu Bionic): status |
New |
Confirmed |
|
| 2018-11-14 14:25:32 |
Launchpad Janitor |
init-system-helpers (Ubuntu Bionic): status |
New |
Confirmed |
|
| 2018-11-14 14:25:32 |
Launchpad Janitor |
snapd (Ubuntu Bionic): status |
New |
Confirmed |
|
| 2018-11-24 23:43:59 |
Mathew Hodson |
bug |
|
|
added subscriber Mathew Hodson |
| 2018-11-26 17:19:16 |
Łukasz Zemczak |
unattended-upgrades (Ubuntu Cosmic): status |
New |
Fix Committed |
|
| 2018-11-26 17:19:19 |
Łukasz Zemczak |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2018-11-26 17:19:23 |
Łukasz Zemczak |
bug |
|
|
added subscriber SRU Verification |
| 2018-11-26 17:19:26 |
Łukasz Zemczak |
tags |
apport-collected bionic id-5b3568d38c766fbe4a6b1710 rls-cc-incoming |
apport-collected bionic id-5b3568d38c766fbe4a6b1710 rls-cc-incoming verification-needed verification-needed-cosmic |
|
| 2018-11-26 17:24:11 |
Łukasz Zemczak |
unattended-upgrades (Ubuntu Bionic): status |
In Progress |
Fix Committed |
|
| 2018-11-26 17:24:16 |
Łukasz Zemczak |
tags |
apport-collected bionic id-5b3568d38c766fbe4a6b1710 rls-cc-incoming verification-needed verification-needed-cosmic |
apport-collected bionic id-5b3568d38c766fbe4a6b1710 rls-cc-incoming verification-needed verification-needed-bionic verification-needed-cosmic |
|
| 2018-11-28 13:45:45 |
Balint Reczey |
description |
[Impact]
* Unattended-upgrades hangs and gets killed when installing upgrades that stat/stop services on shutdown, leaving the system in a broken state
[Test Case]
* Install an updated bionic system:
$ lxc launch ubuntu:18.04 uu-shutdown-test
# apt update
...
* When testing the fixed version, install upgrade u-u at this point checking that u-u.service is set up before and is wanted by shutdown.target:
# systemd-analyze dot | grep unatt
"unattended-upgrades.service"->"-.mount" [color="green"];
"unattended-upgrades.service"->"system.slice" [color="green"];
"unattended-upgrades.service"->"network.target" [color="green"];
"unattended-upgrades.service"->"systemd-journald.socket" [color="green"];
"unattended-upgrades.service"->"local-fs.target" [color="green"];
"unattended-upgrades.service"->"-.mount" [color="black"];
"unattended-upgrades.service"->"system.slice" [color="black"];
"shutdown.target"->"unattended-upgrades.service" [color="green"];
"shutdown.target"->"unattended-upgrades.service" [color="grey66"];
Color legend: black = Requires
dark blue = Requisite
dark grey = Wants
red = Conflicts
green = After
* Configure u-u to run on shutdown and install -updates:
# echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
# echo 'Unattended-Upgrade::Allowed-Origins:: "${distro_id}:${distro_codename}-updates";' > /etc/apt/apt.conf.d/51unattended-upgrades-updates-too
* Downgrade snapd:
# apt install snapd=2.32.5+18.04
* # reboot
* With not fixed u-u observe the upgrade process being stuck:
# pstree| grep unatt
`-unattended-upgr---unattended-upgr-+-unattended-upgr---dpkg---snapd.prerm---systemctl
`-{unattended-upgr}
* With fixed u-u observe snapd update taking place and system rebooting after a few seconds with all updates installed
* Since this fix is partially reverting the fix for LP: #1654600 please test LP: #1654600 as well to avoid regressions.
[Regression Potential]
* As part of the fix manual changes were made to postinst to properly transition from coupling u-u.service with multi-user.target to coupling it with shutdown.target again which can make u-u started during normal boot when there is a bug in the implementation.
on-testing the SRU.
* Due to relationship changes between u-u.service, other services and targets u-u may fail to run on shutdown in case of an unexpected regression.
[Original Bug Text]
When using unattended-upgrades with "InstallOnShutdown" on Bionic, the package installation on various packages hangs until the systemd ShutdownTimeout (30min) is expired and systemd kills all processes and powers off/reboots the system.
This leaves packages in an unconfigured, broken state. At least sometimes this cannot be fixed with a "dpkg --configure -a", but instead requires the user to manually reinstall the package that caused the hang.
This appears to be a deadlock, because the hanging commands are always "systemctl stop ..." or "systemctl restart ...", etc.. If I understand this correctly, those systemctl commands block because systemd tries to shutdown the system and tries to satisfy all dependencies for the shutdown targets before those systemctl commands could get executed, which creates a deadlock.
Steps to reproduce:
- Install 18.04
- activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
- disable bionic-updates in /etc/apt/sources.list (more on that later)
- execute "unattended-upgrade --download-only"
- reboot the system
-> The upgrade on shutdown hangs when configuring the apport package. The hanging command is "systemctl stop apport-forward.socket". The system hangs until the systemd ShutdownTimeout expires and systemd forcefully reboots the system.
After the system is rebooted the apport package is in "iUR" state, and needs to be reinstalled to fix this.
I disabled the bionic-updates pocket in sources.list, because in the default configuration unattended-updates does not use bionic-updates, and seems to have skipped installation of apport from bionic-security (supposedly because an already newer version of apport was in bionic-updates). If my understanding of why apport initially did not get installed is correct, then this would be another problem, because it would mean that unattended-upgrades potentially does not install all available security updates when bionic-updates is enabled in sources.list (which is the default).
The problem can also be reproduced without disabling bionic-updates in sources.list, but instead enabling bionic-updates in /etc/apt/apt.conf.d/50unattended-upgrades. Then, in my case, the upgrade did hang when installing the package snapd (the hanging command was "systemctl stop snapd.autoimport.service snapd.core-fixup.service snapd.service snapd.snap-repair.service snapd.snap-repair.service snapd.socket snapd.system-shutdown.service"). This leads to the same problems as described above.
---
ApportVersion: 2.20.9-0ubuntu7.1
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2018-06-22 (0 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
Package: unattended-upgrades 1.1ubuntu1
PackageArchitecture: all
ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17
Tags: bionic
Uname: Linux 4.15.0-20-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True |
[Impact]
* Unattended-upgrades hangs and gets killed when installing upgrades that stat/stop services on shutdown, leaving the system in a broken state
[Test Case]
* Install an updated bionic system:
$ lxc launch ubuntu:18.04 uu-shutdown-test
# apt update
...
* When testing the fixed version, install upgrade u-u at this point checking that u-u.service is set up before and is wanted by shutdown.target:
# systemd-analyze dot | grep unatt
"unattended-upgrades.service"->"-.mount" [color="green"];
"unattended-upgrades.service"->"system.slice" [color="green"];
"unattended-upgrades.service"->"network.target" [color="green"];
"unattended-upgrades.service"->"systemd-journald.socket" [color="green"];
"unattended-upgrades.service"->"local-fs.target" [color="green"];
"unattended-upgrades.service"->"-.mount" [color="black"];
"unattended-upgrades.service"->"system.slice" [color="black"];
"shutdown.target"->"unattended-upgrades.service" [color="green"];
"shutdown.target"->"unattended-upgrades.service" [color="grey66"];
Color legend: black = Requires
dark blue = Requisite
dark grey = Wants
red = Conflicts
green = After
* Configure u-u to run on shutdown and install -updates:
# echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
# echo 'Unattended-Upgrade::Allowed-Origins:: "${distro_id}:${distro_codename}-updates";' > /etc/apt/apt.conf.d/51unattended-upgrades-updates-too
* Downgrade snapd:
# apt install snapd=2.32.5+18.04
* Dowload packages for u-u:
# unattended-upgrade --download-only
* Reboot using logind to let inhibitors hold up shutdown:
# dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false
* With not fixed u-u observe the upgrade process being stuck:
# pstree| grep unatt
`-unattended-upgr---unattended-upgr-+-unattended-upgr---dpkg---snapd.prerm---systemctl
`-{unattended-upgr}
* With fixed u-u observe snapd update taking place and system rebooting after a few seconds with all updates installed
* Since this fix is partially reverting the fix for LP: #1654600 please test LP: #1654600 as well to avoid regressions.
[Regression Potential]
* As part of the fix manual changes were made to postinst to properly transition from coupling u-u.service with multi-user.target to coupling it with shutdown.target again which can make u-u started during normal boot when there is a bug in the implementation.
on-testing the SRU.
* Due to relationship changes between u-u.service, other services and targets u-u may fail to run on shutdown in case of an unexpected regression.
[Original Bug Text]
When using unattended-upgrades with "InstallOnShutdown" on Bionic, the package installation on various packages hangs until the systemd ShutdownTimeout (30min) is expired and systemd kills all processes and powers off/reboots the system.
This leaves packages in an unconfigured, broken state. At least sometimes this cannot be fixed with a "dpkg --configure -a", but instead requires the user to manually reinstall the package that caused the hang.
This appears to be a deadlock, because the hanging commands are always "systemctl stop ..." or "systemctl restart ...", etc.. If I understand this correctly, those systemctl commands block because systemd tries to shutdown the system and tries to satisfy all dependencies for the shutdown targets before those systemctl commands could get executed, which creates a deadlock.
Steps to reproduce:
- Install 18.04
- activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
- disable bionic-updates in /etc/apt/sources.list (more on that later)
- execute "unattended-upgrade --download-only"
- reboot the system
-> The upgrade on shutdown hangs when configuring the apport package. The hanging command is "systemctl stop apport-forward.socket". The system hangs until the systemd ShutdownTimeout expires and systemd forcefully reboots the system.
After the system is rebooted the apport package is in "iUR" state, and needs to be reinstalled to fix this.
I disabled the bionic-updates pocket in sources.list, because in the default configuration unattended-updates does not use bionic-updates, and seems to have skipped installation of apport from bionic-security (supposedly because an already newer version of apport was in bionic-updates). If my understanding of why apport initially did not get installed is correct, then this would be another problem, because it would mean that unattended-upgrades potentially does not install all available security updates when bionic-updates is enabled in sources.list (which is the default).
The problem can also be reproduced without disabling bionic-updates in sources.list, but instead enabling bionic-updates in /etc/apt/apt.conf.d/50unattended-upgrades. Then, in my case, the upgrade did hang when installing the package snapd (the hanging command was "systemctl stop snapd.autoimport.service snapd.core-fixup.service snapd.service snapd.snap-repair.service snapd.snap-repair.service snapd.socket snapd.system-shutdown.service"). This leads to the same problems as described above.
---
ApportVersion: 2.20.9-0ubuntu7.1
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2018-06-22 (0 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
Package: unattended-upgrades 1.1ubuntu1
PackageArchitecture: all
ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17
Tags: bionic
Uname: Linux 4.15.0-20-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True |
|
| 2018-11-28 16:14:34 |
Balint Reczey |
tags |
apport-collected bionic id-5b3568d38c766fbe4a6b1710 rls-cc-incoming verification-needed verification-needed-bionic verification-needed-cosmic |
apport-collected bionic id-5b3568d38c766fbe4a6b1710 rls-cc-incoming verification-done-bionic verification-needed verification-needed-cosmic |
|
| 2018-11-28 16:55:08 |
Balint Reczey |
tags |
apport-collected bionic id-5b3568d38c766fbe4a6b1710 rls-cc-incoming verification-done-bionic verification-needed verification-needed-cosmic |
apport-collected bionic id-5b3568d38c766fbe4a6b1710 rls-cc-incoming verification-done verification-done-bionic verification-done-cosmic |
|
| 2018-12-03 15:13:09 |
Launchpad Janitor |
unattended-upgrades (Ubuntu Cosmic): status |
Fix Committed |
Fix Released |
|
| 2018-12-03 15:13:16 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2018-12-03 15:20:08 |
Launchpad Janitor |
unattended-upgrades (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2018-12-03 19:19:12 |
Brian Murray |
unattended-upgrades (Ubuntu Xenial): status |
In Progress |
Fix Committed |
|
| 2018-12-03 19:19:18 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2018-12-03 19:19:23 |
Brian Murray |
tags |
apport-collected bionic id-5b3568d38c766fbe4a6b1710 rls-cc-incoming verification-done verification-done-bionic verification-done-cosmic |
apport-collected bionic id-5b3568d38c766fbe4a6b1710 rls-cc-incoming verification-done-bionic verification-done-cosmic verification-needed verification-needed-xenial |
|
| 2018-12-13 14:49:01 |
Balint Reczey |
description |
[Impact]
* Unattended-upgrades hangs and gets killed when installing upgrades that stat/stop services on shutdown, leaving the system in a broken state
[Test Case]
* Install an updated bionic system:
$ lxc launch ubuntu:18.04 uu-shutdown-test
# apt update
...
* When testing the fixed version, install upgrade u-u at this point checking that u-u.service is set up before and is wanted by shutdown.target:
# systemd-analyze dot | grep unatt
"unattended-upgrades.service"->"-.mount" [color="green"];
"unattended-upgrades.service"->"system.slice" [color="green"];
"unattended-upgrades.service"->"network.target" [color="green"];
"unattended-upgrades.service"->"systemd-journald.socket" [color="green"];
"unattended-upgrades.service"->"local-fs.target" [color="green"];
"unattended-upgrades.service"->"-.mount" [color="black"];
"unattended-upgrades.service"->"system.slice" [color="black"];
"shutdown.target"->"unattended-upgrades.service" [color="green"];
"shutdown.target"->"unattended-upgrades.service" [color="grey66"];
Color legend: black = Requires
dark blue = Requisite
dark grey = Wants
red = Conflicts
green = After
* Configure u-u to run on shutdown and install -updates:
# echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
# echo 'Unattended-Upgrade::Allowed-Origins:: "${distro_id}:${distro_codename}-updates";' > /etc/apt/apt.conf.d/51unattended-upgrades-updates-too
* Downgrade snapd:
# apt install snapd=2.32.5+18.04
* Dowload packages for u-u:
# unattended-upgrade --download-only
* Reboot using logind to let inhibitors hold up shutdown:
# dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false
* With not fixed u-u observe the upgrade process being stuck:
# pstree| grep unatt
`-unattended-upgr---unattended-upgr-+-unattended-upgr---dpkg---snapd.prerm---systemctl
`-{unattended-upgr}
* With fixed u-u observe snapd update taking place and system rebooting after a few seconds with all updates installed
* Since this fix is partially reverting the fix for LP: #1654600 please test LP: #1654600 as well to avoid regressions.
[Regression Potential]
* As part of the fix manual changes were made to postinst to properly transition from coupling u-u.service with multi-user.target to coupling it with shutdown.target again which can make u-u started during normal boot when there is a bug in the implementation.
on-testing the SRU.
* Due to relationship changes between u-u.service, other services and targets u-u may fail to run on shutdown in case of an unexpected regression.
[Original Bug Text]
When using unattended-upgrades with "InstallOnShutdown" on Bionic, the package installation on various packages hangs until the systemd ShutdownTimeout (30min) is expired and systemd kills all processes and powers off/reboots the system.
This leaves packages in an unconfigured, broken state. At least sometimes this cannot be fixed with a "dpkg --configure -a", but instead requires the user to manually reinstall the package that caused the hang.
This appears to be a deadlock, because the hanging commands are always "systemctl stop ..." or "systemctl restart ...", etc.. If I understand this correctly, those systemctl commands block because systemd tries to shutdown the system and tries to satisfy all dependencies for the shutdown targets before those systemctl commands could get executed, which creates a deadlock.
Steps to reproduce:
- Install 18.04
- activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
- disable bionic-updates in /etc/apt/sources.list (more on that later)
- execute "unattended-upgrade --download-only"
- reboot the system
-> The upgrade on shutdown hangs when configuring the apport package. The hanging command is "systemctl stop apport-forward.socket". The system hangs until the systemd ShutdownTimeout expires and systemd forcefully reboots the system.
After the system is rebooted the apport package is in "iUR" state, and needs to be reinstalled to fix this.
I disabled the bionic-updates pocket in sources.list, because in the default configuration unattended-updates does not use bionic-updates, and seems to have skipped installation of apport from bionic-security (supposedly because an already newer version of apport was in bionic-updates). If my understanding of why apport initially did not get installed is correct, then this would be another problem, because it would mean that unattended-upgrades potentially does not install all available security updates when bionic-updates is enabled in sources.list (which is the default).
The problem can also be reproduced without disabling bionic-updates in sources.list, but instead enabling bionic-updates in /etc/apt/apt.conf.d/50unattended-upgrades. Then, in my case, the upgrade did hang when installing the package snapd (the hanging command was "systemctl stop snapd.autoimport.service snapd.core-fixup.service snapd.service snapd.snap-repair.service snapd.snap-repair.service snapd.socket snapd.system-shutdown.service"). This leads to the same problems as described above.
---
ApportVersion: 2.20.9-0ubuntu7.1
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2018-06-22 (0 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
Package: unattended-upgrades 1.1ubuntu1
PackageArchitecture: all
ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17
Tags: bionic
Uname: Linux 4.15.0-20-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True |
[Impact]
* Unattended-upgrades hangs and gets killed when installing upgrades that stat/stop services on shutdown, leaving the system in a broken state
[Test Case]
* Install an updated bionic system:
$ lxc launch ubuntu:18.04 uu-shutdown-test
# apt update
...
* When testing the fixed version, install upgrade u-u at this point checking that u-u.service is set up before and is wanted by shutdown.target:
# systemd-analyze dot | grep unatt
"unattended-upgrades.service"->"-.mount" [color="green"];
"unattended-upgrades.service"->"system.slice" [color="green"];
"unattended-upgrades.service"->"network.target" [color="green"];
"unattended-upgrades.service"->"systemd-journald.socket" [color="green"];
"unattended-upgrades.service"->"local-fs.target" [color="green"];
"unattended-upgrades.service"->"-.mount" [color="black"];
"unattended-upgrades.service"->"system.slice" [color="black"];
"shutdown.target"->"unattended-upgrades.service" [color="green"];
"shutdown.target"->"unattended-upgrades.service" [color="grey66"];
Color legend: black = Requires
dark blue = Requisite
dark grey = Wants
red = Conflicts
green = After
* Configure u-u to run on shutdown and install -updates:
# echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
# echo 'Unattended-Upgrade::Allowed-Origins:: "${distro_id}:${distro_codename}-updates";' > /etc/apt/apt.conf.d/51unattended-upgrades-updates-too
* Downgrade snapd:
# apt install snapd=2.32.5+18.04
* Dowload packages for u-u:
# unattended-upgrade --download-only
* Reboot using logind to let inhibitors hold up shutdown:
# dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false
* With not fixed u-u observe the upgrade process being stuck:
# pstree| grep unatt
`-unattended-upgr---unattended-upgr-+-unattended-upgr---dpkg---snapd.prerm---systemctl
`-{unattended-upgr}
* With fixed u-u observe snapd update taking place and system rebooting after a few seconds with all updates installed
* Since this fix is partially reverting the fix for LP: #1654600 please test LP: #1654600 as well to avoid regressions.
[Regression Potential]
* As part of the fix manual changes were made to postinst to properly transition from coupling u-u.service with multi-user.target to coupling it with shutdown.target again which can make u-u started during normal boot when there is a bug in the implementation.
on-testing the SRU.
* Due to relationship changes between u-u.service, other services and targets u-u may fail to run on shutdown in case of an unexpected regression.
* There is a rarely occurring new crash caused by the (second) fix tracked in LP: #1806487.
[Original Bug Text]
When using unattended-upgrades with "InstallOnShutdown" on Bionic, the package installation on various packages hangs until the systemd ShutdownTimeout (30min) is expired and systemd kills all processes and powers off/reboots the system.
This leaves packages in an unconfigured, broken state. At least sometimes this cannot be fixed with a "dpkg --configure -a", but instead requires the user to manually reinstall the package that caused the hang.
This appears to be a deadlock, because the hanging commands are always "systemctl stop ..." or "systemctl restart ...", etc.. If I understand this correctly, those systemctl commands block because systemd tries to shutdown the system and tries to satisfy all dependencies for the shutdown targets before those systemctl commands could get executed, which creates a deadlock.
Steps to reproduce:
- Install 18.04
- activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
- disable bionic-updates in /etc/apt/sources.list (more on that later)
- execute "unattended-upgrade --download-only"
- reboot the system
-> The upgrade on shutdown hangs when configuring the apport package. The hanging command is "systemctl stop apport-forward.socket". The system hangs until the systemd ShutdownTimeout expires and systemd forcefully reboots the system.
After the system is rebooted the apport package is in "iUR" state, and needs to be reinstalled to fix this.
I disabled the bionic-updates pocket in sources.list, because in the default configuration unattended-updates does not use bionic-updates, and seems to have skipped installation of apport from bionic-security (supposedly because an already newer version of apport was in bionic-updates). If my understanding of why apport initially did not get installed is correct, then this would be another problem, because it would mean that unattended-upgrades potentially does not install all available security updates when bionic-updates is enabled in sources.list (which is the default).
The problem can also be reproduced without disabling bionic-updates in sources.list, but instead enabling bionic-updates in /etc/apt/apt.conf.d/50unattended-upgrades. Then, in my case, the upgrade did hang when installing the package snapd (the hanging command was "systemctl stop snapd.autoimport.service snapd.core-fixup.service snapd.service snapd.snap-repair.service snapd.snap-repair.service snapd.socket snapd.system-shutdown.service"). This leads to the same problems as described above.
---
ApportVersion: 2.20.9-0ubuntu7.1
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2018-06-22 (0 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
Package: unattended-upgrades 1.1ubuntu1
PackageArchitecture: all
ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17
Tags: bionic
Uname: Linux 4.15.0-20-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True |
|
| 2018-12-28 20:53:18 |
Mathew Hodson |
bug task deleted |
apt (Ubuntu) |
|
|
| 2018-12-28 20:53:26 |
Mathew Hodson |
bug task deleted |
apt (Ubuntu Xenial) |
|
|
| 2018-12-28 20:53:35 |
Mathew Hodson |
bug task deleted |
apt (Ubuntu Bionic) |
|
|
| 2018-12-28 21:05:12 |
Mathew Hodson |
unattended-upgrades (Ubuntu Cosmic): importance |
Undecided |
Medium |
|
| 2019-01-24 18:43:57 |
André Jutisz |
attachment added |
|
unattended-upgrades.log https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1778219/+attachment/5232300/+files/unattended-upgrades.log |
|
| 2019-04-01 02:25:04 |
Mathew Hodson |
bug task deleted |
init-system-helpers (Ubuntu) |
|
|
| 2019-04-01 02:25:19 |
Mathew Hodson |
bug task deleted |
init-system-helpers (Ubuntu Xenial) |
|
|
| 2019-04-01 02:25:26 |
Mathew Hodson |
bug task deleted |
init-system-helpers (Ubuntu Bionic) |
|
|
| 2019-04-11 11:35:48 |
Balint Reczey |
tags |
apport-collected bionic id-5b3568d38c766fbe4a6b1710 rls-cc-incoming verification-done-bionic verification-done-cosmic verification-needed verification-needed-xenial |
apport-collected bionic id-5b3568d38c766fbe4a6b1710 rls-cc-incoming verification-done verification-done-bionic verification-done-cosmic verification-done-xenial |
|
| 2019-04-12 06:29:09 |
Balint Reczey |
description |
[Impact]
* Unattended-upgrades hangs and gets killed when installing upgrades that stat/stop services on shutdown, leaving the system in a broken state
[Test Case]
* Install an updated bionic system:
$ lxc launch ubuntu:18.04 uu-shutdown-test
# apt update
...
* When testing the fixed version, install upgrade u-u at this point checking that u-u.service is set up before and is wanted by shutdown.target:
# systemd-analyze dot | grep unatt
"unattended-upgrades.service"->"-.mount" [color="green"];
"unattended-upgrades.service"->"system.slice" [color="green"];
"unattended-upgrades.service"->"network.target" [color="green"];
"unattended-upgrades.service"->"systemd-journald.socket" [color="green"];
"unattended-upgrades.service"->"local-fs.target" [color="green"];
"unattended-upgrades.service"->"-.mount" [color="black"];
"unattended-upgrades.service"->"system.slice" [color="black"];
"shutdown.target"->"unattended-upgrades.service" [color="green"];
"shutdown.target"->"unattended-upgrades.service" [color="grey66"];
Color legend: black = Requires
dark blue = Requisite
dark grey = Wants
red = Conflicts
green = After
* Configure u-u to run on shutdown and install -updates:
# echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
# echo 'Unattended-Upgrade::Allowed-Origins:: "${distro_id}:${distro_codename}-updates";' > /etc/apt/apt.conf.d/51unattended-upgrades-updates-too
* Downgrade snapd:
# apt install snapd=2.32.5+18.04
* Dowload packages for u-u:
# unattended-upgrade --download-only
* Reboot using logind to let inhibitors hold up shutdown:
# dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false
* With not fixed u-u observe the upgrade process being stuck:
# pstree| grep unatt
`-unattended-upgr---unattended-upgr-+-unattended-upgr---dpkg---snapd.prerm---systemctl
`-{unattended-upgr}
* With fixed u-u observe snapd update taking place and system rebooting after a few seconds with all updates installed
* Since this fix is partially reverting the fix for LP: #1654600 please test LP: #1654600 as well to avoid regressions.
[Regression Potential]
* As part of the fix manual changes were made to postinst to properly transition from coupling u-u.service with multi-user.target to coupling it with shutdown.target again which can make u-u started during normal boot when there is a bug in the implementation.
on-testing the SRU.
* Due to relationship changes between u-u.service, other services and targets u-u may fail to run on shutdown in case of an unexpected regression.
* There is a rarely occurring new crash caused by the (second) fix tracked in LP: #1806487.
[Original Bug Text]
When using unattended-upgrades with "InstallOnShutdown" on Bionic, the package installation on various packages hangs until the systemd ShutdownTimeout (30min) is expired and systemd kills all processes and powers off/reboots the system.
This leaves packages in an unconfigured, broken state. At least sometimes this cannot be fixed with a "dpkg --configure -a", but instead requires the user to manually reinstall the package that caused the hang.
This appears to be a deadlock, because the hanging commands are always "systemctl stop ..." or "systemctl restart ...", etc.. If I understand this correctly, those systemctl commands block because systemd tries to shutdown the system and tries to satisfy all dependencies for the shutdown targets before those systemctl commands could get executed, which creates a deadlock.
Steps to reproduce:
- Install 18.04
- activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
- disable bionic-updates in /etc/apt/sources.list (more on that later)
- execute "unattended-upgrade --download-only"
- reboot the system
-> The upgrade on shutdown hangs when configuring the apport package. The hanging command is "systemctl stop apport-forward.socket". The system hangs until the systemd ShutdownTimeout expires and systemd forcefully reboots the system.
After the system is rebooted the apport package is in "iUR" state, and needs to be reinstalled to fix this.
I disabled the bionic-updates pocket in sources.list, because in the default configuration unattended-updates does not use bionic-updates, and seems to have skipped installation of apport from bionic-security (supposedly because an already newer version of apport was in bionic-updates). If my understanding of why apport initially did not get installed is correct, then this would be another problem, because it would mean that unattended-upgrades potentially does not install all available security updates when bionic-updates is enabled in sources.list (which is the default).
The problem can also be reproduced without disabling bionic-updates in sources.list, but instead enabling bionic-updates in /etc/apt/apt.conf.d/50unattended-upgrades. Then, in my case, the upgrade did hang when installing the package snapd (the hanging command was "systemctl stop snapd.autoimport.service snapd.core-fixup.service snapd.service snapd.snap-repair.service snapd.snap-repair.service snapd.socket snapd.system-shutdown.service"). This leads to the same problems as described above.
---
ApportVersion: 2.20.9-0ubuntu7.1
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2018-06-22 (0 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
Package: unattended-upgrades 1.1ubuntu1
PackageArchitecture: all
ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17
Tags: bionic
Uname: Linux 4.15.0-20-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True |
[Impact]
* Unattended-upgrades hangs and gets killed when installing upgrades that stat/stop services on shutdown, leaving the system in a broken state
[Test Case]
* Install an updated bionic system:
$ lxc launch ubuntu:18.04 uu-shutdown-test
# apt update
...
* When testing the fixed version, install upgrade u-u at this point checking that u-u.service is set up before and is wanted by shutdown.target:
# systemd-analyze dot | grep unatt
"unattended-upgrades.service"->"-.mount" [color="green"];
"unattended-upgrades.service"->"system.slice" [color="green"];
"unattended-upgrades.service"->"network.target" [color="green"];
"unattended-upgrades.service"->"systemd-journald.socket" [color="green"];
"unattended-upgrades.service"->"local-fs.target" [color="green"];
"unattended-upgrades.service"->"-.mount" [color="black"];
"unattended-upgrades.service"->"system.slice" [color="black"];
"shutdown.target"->"unattended-upgrades.service" [color="green"];
"shutdown.target"->"unattended-upgrades.service" [color="grey66"];
Color legend: black = Requires
dark blue = Requisite
dark grey = Wants
red = Conflicts
green = After
* Configure u-u to run on shutdown and install -updates:
# echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
# echo 'Unattended-Upgrade::Allowed-Origins:: "${distro_id}:${distro_codename}-updates";' > /etc/apt/apt.conf.d/51unattended-upgrades-updates-too
* Downgrade snapd:
# apt install snapd=2.32.5+18.04
* Dowload packages for u-u:
# unattended-upgrade --download-only
* Reboot using logind to let inhibitors hold up shutdown:
# dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 "org.freedesktop.login1.Manager.Reboot" boolean:false
* With not fixed u-u observe the upgrade process being stuck:
# pstree| grep unatt
`-unattended-upgr---unattended-upgr-+-unattended-upgr---dpkg---snapd.prerm---systemctl
`-{unattended-upgr}
* With fixed u-u observe snapd update taking place and system rebooting after a few seconds with all updates installed
[Regression Potential]
* As part of the fix manual changes were made to postinst to properly transition from coupling u-u.service with multi-user.target to coupling it with shutdown.target again which can make u-u started during normal boot when there is a bug in the implementation.
on-testing the SRU.
* Due to relationship changes between u-u.service, other services and targets u-u may fail to run on shutdown in case of an unexpected regression.
* There is a rarely occurring new crash caused by the (second) fix tracked in LP: #1806487.
[Original Bug Text]
When using unattended-upgrades with "InstallOnShutdown" on Bionic, the package installation on various packages hangs until the systemd ShutdownTimeout (30min) is expired and systemd kills all processes and powers off/reboots the system.
This leaves packages in an unconfigured, broken state. At least sometimes this cannot be fixed with a "dpkg --configure -a", but instead requires the user to manually reinstall the package that caused the hang.
This appears to be a deadlock, because the hanging commands are always "systemctl stop ..." or "systemctl restart ...", etc.. If I understand this correctly, those systemctl commands block because systemd tries to shutdown the system and tries to satisfy all dependencies for the shutdown targets before those systemctl commands could get executed, which creates a deadlock.
Steps to reproduce:
- Install 18.04
- activate "InstallOnShutdown" in /etc/apt/apt.conf.d/50unattended-upgrades
- disable bionic-updates in /etc/apt/sources.list (more on that later)
- execute "unattended-upgrade --download-only"
- reboot the system
-> The upgrade on shutdown hangs when configuring the apport package. The hanging command is "systemctl stop apport-forward.socket". The system hangs until the systemd ShutdownTimeout expires and systemd forcefully reboots the system.
After the system is rebooted the apport package is in "iUR" state, and needs to be reinstalled to fix this.
I disabled the bionic-updates pocket in sources.list, because in the default configuration unattended-updates does not use bionic-updates, and seems to have skipped installation of apport from bionic-security (supposedly because an already newer version of apport was in bionic-updates). If my understanding of why apport initially did not get installed is correct, then this would be another problem, because it would mean that unattended-upgrades potentially does not install all available security updates when bionic-updates is enabled in sources.list (which is the default).
The problem can also be reproduced without disabling bionic-updates in sources.list, but instead enabling bionic-updates in /etc/apt/apt.conf.d/50unattended-upgrades. Then, in my case, the upgrade did hang when installing the package snapd (the hanging command was "systemctl stop snapd.autoimport.service snapd.core-fixup.service snapd.service snapd.snap-repair.service snapd.snap-repair.service snapd.socket snapd.system-shutdown.service"). This leads to the same problems as described above.
---
ApportVersion: 2.20.9-0ubuntu7.1
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2018-06-22 (0 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
Package: unattended-upgrades 1.1ubuntu1
PackageArchitecture: all
ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17
Tags: bionic
Uname: Linux 4.15.0-20-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True |
|
| 2019-04-25 10:50:00 |
Launchpad Janitor |
unattended-upgrades (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2019-04-26 03:26:38 |
Mathew Hodson |
bug task deleted |
snapd (Ubuntu Bionic) |
|
|
| 2019-04-26 03:26:46 |
Mathew Hodson |
bug task deleted |
snapd (Ubuntu Xenial) |
|
|
| 2019-04-26 03:26:53 |
Mathew Hodson |
bug task deleted |
snapd (Ubuntu) |
|
|
| 2019-04-26 03:27:00 |
Mathew Hodson |
removed subscriber Mathew Hodson |
|
|
|
