systemd unit leaves a process running for the system lifetime but it only does anything at shutdown

[Steve Langasek]

The /lib/systemd/system/unattended-upgrades.service unit leaves a process, /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal, running from system start until shutdown. This increases the minimum memory footprint of every host, VM, or container where unattended-upgrades is installed. The motivation for having this systemd unit is clear in terms of the practicalities of implementing what this unit needs to do, but we should try to find some way to achieve this which does not rely on increasing the low water mark of system memory.

[Dave Jones (waveform)]

As requested, here's what I managed to dig up on possible alternatives
to running unattended-upgrades-shutdown persistently:

There appear to be several methods by which systemd can be told to run
something at or near shutdown, but one has to be exceptionally careful
in the selection to cover all edge cases.

Some possibilities can be trivially discarded. For instance, the
systemd-shutdown (8) logic runs too late, potentially after the root
file-system has been remounted read-only.

Other possibilities have extremely subtle side-effects. For example, one
tempting possibility is to reverse the current logic of the
unattended-upgrades.service, which currently looks like this:

  [Service]
  ExecStart=/usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
  KillMode=process
  TimeoutStopSec=1800

And instead use a "fake" service that systemd pretends is running, then
activate the shutdown script when systemd goes to stop the service, like
so:

  [Service]
  ExecStart=/bin/true
  RemainAfterExit=yes
  ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
  KillMode=process
  TimeoutStopSec=1800

Note the lack of --wait-for-signal (because we want the script to run
immediately when systemd goes to "stop" the fake service).

However, tempting though this possibility is, LP: #1778219 (and associated
commits, particularly [4]) describes why this doesn't work in practice. If a
package upgrade is occurring during shutdown, and that package attempts to
start or restart a systemd service, the attempt will hang. This is particularly
an issue with the "InstallOnShutdown" behaviour (not the default) which expects
to fully complete the upgrade before shutdown (as opposed to the default
behaviour which attempts to stop the upgrade after the next transaction
completes).

Side note: The KillMode setting appears to be an attempt to prevent systemd
from killing a subordinate apt process in the event it deems
unattended-upgrades to have timed out. This seems … risky to me. The timeout
duration is, so far as I can tell, picked fairly arbitrarily and what happens in the
event that unattended-upgrades times out, but leaves subordinate apt/dpkg
processes lying around is unclear (does systemd wait for them indefinitely,
does it kill them after another timeout, does it shutdown/reset the machine
with them still running?). There's some more context in LP: #1690980 regard
this.

Ultimately, the solution appears to have been having a persistent process
(u-u-shutdown) watch for the logind PrepareForShutdown event which occurs well
before systemd actually starts the shutdown. Unfortunately (as far as I've been
able to tell?) this requires an active process to watch for it.

The following bugs contain a great deal of detail and further context to this
issue:

* LP: #1778219 -- u-u hanging on shutdown
* LP: #1654600 -- u-u hanging when /var is separate file-system
* LP: #1618900 -- u-u failing to upgrade kernels due to missing mount (+others)
* LP: #1690980 -- context regarding u-u's shutdown inhibit, and killmode

There are also some useful test cases found in these bugs that should be used
in ensuring certain scenarios aren't broken. The f...