Systemd update installation hangs in unattended-upgrades InstallOnShutdown mode

Bug #1803391 reported by Balint Reczey on 2018-11-14
264
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Status tracked in Disco
Xenial
High
Unassigned
Bionic
High
Unassigned
Cosmic
High
Unassigned
Disco
High
Unassigned

Bug Description

[Impact]

 * Installation of latest systemd update in -security hangs with current versions of unattended-upgrades in supported releases. The u-u-side fix is tracked in LP: #1778219.

[Regression Potential]

 * The daemons, shipped in deb:systemd, are not attempted to be restarted because despite package installation the system is in the middle of shutting down. This means that currently running daemons may be helding up open files on the filesystem, however all process are being stopped and killed as part of shutdown. Hence the worst possible regression from this, is an unclean shutdown, but even that shouldn't happen with this update.

[Test Case]

Reproduction:

rbalint@yogi:~$ lxc launch ubuntu:18.04 uu-systemd-onshutdown
Creating uu-systemd-onshutdown
Starting uu-systemd-onshutdown
rbalint@yogi:~$ lxc shell uu-systemd-onshutdown
mesg: ttyname failed: No such device
root@uu-systemd-onshutdown:~# apt update -qq
23 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@uu-systemd-onshutdown:~# echo 'Unattended-Upgrade::InstallOnShutdown "true";' > /etc/apt/apt.conf.d/51unattended-upgrades-on-shutdown
root@uu-systemd-onshutdown:~# apt list --upgradable
Listing... Done
apport/bionic-updates 2.20.9-0ubuntu7.5 all [upgradable from: 2.20.9-0ubuntu7.4]
gettext-base/bionic-updates,bionic-security 0.19.8.1-6ubuntu0.1 amd64 [upgradable from: 0.19.8.1-6]
kmod/bionic-updates 24-1ubuntu3.1 amd64 [upgradable from: 24-1ubuntu3]
libglib2.0-0/bionic-updates 2.56.3-0ubuntu0.18.04.1 amd64 [upgradable from: 2.56.2-0ubuntu0.18.04.2]
libglib2.0-data/bionic-updates 2.56.3-0ubuntu0.18.04.1 all [upgradable from: 2.56.2-0ubuntu0.18.04.2]
libkmod2/bionic-updates 24-1ubuntu3.1 amd64 [upgradable from: 24-1ubuntu3]
libmspack0/bionic-updates,bionic-security 0.6-3ubuntu0.2 amd64 [upgradable from: 0.6-3ubuntu0.1]
libnss-systemd/bionic-updates,bionic-security 237-3ubuntu10.6 amd64 [upgradable from: 237-3ubuntu10.3]
libpam-systemd/bionic-updates,bionic-security 237-3ubuntu10.6 amd64 [upgradable from: 237-3ubuntu10.3]
libsystemd0/bionic-updates,bionic-security 237-3ubuntu10.6 amd64 [upgradable from: 237-3ubuntu10.3]
libudev1/bionic-updates,bionic-security 237-3ubuntu10.6 amd64 [upgradable from: 237-3ubuntu10.3]
lxd/bionic-updates 3.0.2-0ubuntu1~18.04.1 amd64 [upgradable from: 3.0.1-0ubuntu1~18.04.1]
lxd-client/bionic-updates 3.0.2-0ubuntu1~18.04.1 amd64 [upgradable from: 3.0.1-0ubuntu1~18.04.1]
openssh-client/bionic-updates,bionic-security 1:7.6p1-4ubuntu0.1 amd64 [upgradable from: 1:7.6p1-4]
openssh-server/bionic-updates,bionic-security 1:7.6p1-4ubuntu0.1 amd64 [upgradable from: 1:7.6p1-4]
openssh-sftp-server/bionic-updates,bionic-security 1:7.6p1-4ubuntu0.1 amd64 [upgradable from: 1:7.6p1-4]
python3-apport/bionic-updates 2.20.9-0ubuntu7.5 all [upgradable from: 2.20.9-0ubuntu7.4]
python3-distupgrade/bionic-updates 1:18.04.28 all [upgradable from: 1:18.04.27]
python3-problem-report/bionic-updates 2.20.9-0ubuntu7.5 all [upgradable from: 2.20.9-0ubuntu7.4]
systemd/bionic-updates,bionic-security 237-3ubuntu10.6 amd64 [upgradable from: 237-3ubuntu10.3]
systemd-sysv/bionic-updates,bionic-security 237-3ubuntu10.6 amd64 [upgradable from: 237-3ubuntu10.3]
ubuntu-release-upgrader-core/bionic-updates 1:18.04.28 all [upgradable from: 1:18.04.27]
udev/bionic-updates,bionic-security 237-3ubuntu10.6 amd64 [upgradable from: 237-3ubuntu10.3]
root@uu-systemd-onshutdown:~# reboot

Session terminated, terminating shell...Terminated
root@uu-systemd-
rbalint@yogi:~$
rbalint@yogi:~$ lxc shell uu-systemd-onshutdown
mesg: ttyname failed: No such device
root@uu-systemd-onshutdown:~# tail /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
Preparing to unpack .../libsystemd0_237-3ubuntu10.6_amd64.deb ...
Unpacking libsystemd0:amd64 (237-3ubuntu10.6) over (237-3ubuntu10.3) ...
Setting up libsystemd0:amd64 (237-3ubuntu10.6) ...
Processing triggers for ureadahead (0.100.0-20) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
Setting up systemd (237-3ubuntu10.6) ...
Failed to try-restart systemd-networkd.service: Transaction is destructive.
See system logs and 'systemctl status systemd-networkd.service' for details.
Failed to try-restart systemd-resolved.service: Transaction is destructive.
See system logs and 'systemctl status systemd-resolved.service' for details.
root@uu-systemd-onshutdown:~# ps -ef | cat
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 15:34 ? 00:00:00 /lib/systemd/systemd --system --deserialize 22
root 53 1 0 15:34 ? 00:00:00 /lib/systemd/systemd-journald
systemd+ 153 1 0 15:34 ? 00:00:00 /lib/systemd/systemd-networkd
systemd+ 154 1 0 15:34 ? 00:00:00 /lib/systemd/systemd-resolved
message+ 194 1 0 15:34 ? 00:00:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root 572 1 0 15:35 ? 00:00:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown
root 586 572 0 15:35 ? 00:00:08 /usr/bin/python3 /usr/bin/unattended-upgrade
root 829 0 0 15:35 ? 00:00:00 su -l
root 830 829 0 15:35 ? 00:00:00 -su
root 1244 1 0 15:35 ? 00:00:00 /lib/systemd/systemd-udevd
root 1918 1 0 15:35 ? 00:00:00 /usr/sbin/sshd -D
root 1952 586 0 15:35 ? 00:00:00 /usr/bin/python3 /usr/bin/unattended-upgrade
root 2111 1952 0 15:35 pts/0 00:00:00 /usr/bin/dpkg --status-fd 10 --configure --pending
root 2125 2111 0 15:35 pts/0 00:00:00 /bin/sh /var/lib/dpkg/info/systemd.postinst configure 237-3ubuntu10.3
root 2165 2125 0 15:35 pts/0 00:00:00 systemctl try-restart systemd-journald.service
root 2170 0 0 15:36 ? 00:00:00 su -l
root 2171 2170 0 15:36 ? 00:00:00 -su
root 2197 2171 0 15:36 ? 00:00:00 ps -ef
root 2198 2171 0 15:36 ? 00:00:00 cat
root@uu-systemd-onshutdown:~# systemctl list-jobs
JOB UNIT TYPE STATE
323 reboot.target start waiting
405 basic.target stop waiting
359 sysinit.target stop waiting
433 cloud-init.service stop waiting
422 user.slice stop waiting
388 snapd.socket stop waiting
428 swap.target stop waiting
332 shutdown.target start waiting
412 slices.target stop waiting
415 systemd-networkd.service stop waiting
338 syslog.socket stop waiting
399 acpid.socket stop waiting
377 uuidd.socket stop waiting
416 systemd-networkd-wait-online.service stop waiting
373 acpid.path stop waiting
432 local-fs.target stop waiting
437 cryptsetup.target stop waiting
341 lxd.socket stop waiting
491 systemd-journald.service restart waiting
340 sockets.target stop waiting
330 var-lib-lxcfs.mount stop waiting
430 cloud-init-local.service stop waiting
453 paths.target stop waiting
402 iscsid.socket stop waiting
439 local-fs-pre.target stop waiting
392 dbus.socket stop waiting
440 systemd-resolved.service stop waiting
353 systemd-ask-password-console.path stop waiting
324 systemd-reboot.service start waiting
498 systemd-journal-flush.service restart waiting
325 final.target start waiting
414 systemd-sysctl.service stop waiting
331 run-user-0.mount stop waiting
435 network.target stop waiting
457 systemd-tmpfiles-setup.service stop waiting
458 systemd-ask-password-wall.path stop waiting
329 umount.target start waiting
369 unattended-upgrades.service stop running
434 network-pre.target stop waiting
436 systemd-tmpfiles-setup-dev.service stop waiting
456 systemd-update-utmp.service stop waiting
389 apport-forward.socket stop waiting
497 systemd-journald-audit.socket start waiting

43 jobs listed.
root@uu-systemd-onshutdown:~#

CVE References

Dimitri John Ledkov (xnox) wrote :

that's really really really bad!

_systemctl try-restart systemd-journald.service || true

_must_ not hang, it should either succeed or fail. that is the whole point of try-restart....

I wonder if `--no-block` would help here, but that's also a bandaid. Imho systemctl/systemd itself shouldn't be able to hang systems like that and like should reject this job submission if it will be impossible to complete.

Changed in systemd (Ubuntu):
importance: Undecided → High
Balint Reczey (rbalint) wrote :

systemctl(1) does not say that try-restart must not hang and --no-block in maintainer scripts would make maintainer scripts continue and execute actions in highly unpredictable states.

tags: added: patch
Dimitri John Ledkov (xnox) wrote :

@ Security team please consider uploading the attached debdiff as 237-3ubuntu10.7 into security pocket to resolve hangs on shutdown when applying updates on shutdown.

Changed in systemd (Ubuntu Disco):
status: New → Fix Committed
Changed in systemd (Ubuntu Cosmic):
status: New → In Progress
Changed in systemd (Ubuntu Bionic):
status: New → In Progress
information type: Public → Public Security
description: updated

Hello Balint, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 237-3ubuntu10.9

---------------
systemd (237-3ubuntu10.9) bionic-security; urgency=medium

  [ Chris Coulson ]
  * SECURITY UPDATE: symlink mishandling in systemd-tmpfiles
    - debian/patches/CVE-2018-6954_2.patch: backport the remaining patches to
      resolve this completely
    - CVE-2018-6954

  [ Balint Reczey ]
  * Fix LP: #1803391 - Skip daemon-reexec and try-restarts during shutdown
    - update debian/systemd.postinst

 -- Chris Coulson <email address hidden> Thu, 15 Nov 2018 20:45:11 +0000

Changed in systemd (Ubuntu Bionic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 239-7ubuntu10.4

---------------
systemd (239-7ubuntu10.4) cosmic-security; urgency=medium

  [ Chris Coulson ]
  * SECURITY UPDATE: symlink mishandling in systemd-tmpfiles
    - debian/patches/CVE-2018-6954_2.patch: backport the remaining patches to
      resolve this completely
    - CVE-2018-6954

  [ Balint Reczey ]
  * Fix LP: #1803391 - Skip daemon-reexec and try-restarts during shutdown
    - update debian/systemd.postinst

 -- Chris Coulson <email address hidden> Thu, 15 Nov 2018 20:42:32 +0000

Changed in systemd (Ubuntu Cosmic):
status: In Progress → Fix Released
Balint Reczey (rbalint) wrote :

The upgrade on Xenial fails in udev.postint, the attached patch should fix that. (Under test)

Balint Reczey (rbalint) wrote :

Should better be this patch.

Balint Reczey (rbalint) wrote :

That last patch works for me on Xenial and fixes the upgrade.

Changed in systemd (Ubuntu Xenial):
status: New → Confirmed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 229-4ubuntu21.10

---------------
systemd (229-4ubuntu21.10) xenial-security; urgency=medium

  [ Chris Coulson ]
  * Revert the fixes for CVE-2018-6954 for causing a regression when running
    in a container on old kernels (LP: #1804847)
    - update debian/patches/series

  [ Balint Reczey ]
  * Fix LP: #1803391 - Don't always trigger systemctl stop of udev service
    and sockets
    - update debian/udev.postinst

 -- Chris Coulson <email address hidden> Tue, 27 Nov 2018 11:10:48 +0000

Changed in systemd (Ubuntu Xenial):
status: Confirmed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 239-7ubuntu14

---------------
systemd (239-7ubuntu14) disco; urgency=medium

  * Fix compat with new meson.
    File: debian/patches/meson-rename-Ddebug-to-Ddebug-extra.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3b764ec1b76768a8c40635019fa5a8acb81b223e

 -- Dimitri John Ledkov <email address hidden> Thu, 29 Nov 2018 16:53:00 +0000

Changed in systemd (Ubuntu Disco):
status: Fix Committed → Fix Released
Changed in systemd (Ubuntu Xenial):
importance: Undecided → High
Changed in systemd (Ubuntu Bionic):
importance: Undecided → High
Changed in systemd (Ubuntu Cosmic):
importance: Undecided → High
To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers