[Xenial/0.90] Systemd dependencies issues when used in "Shutdown mode"

[Guillaume Penin]

[Impact]
There are some race conditions in unattended-upgrades when it is used in shutdown mode. It is possible that the network has already been brought down or for a local file system, like /var/, to be unmounted before the unattended-upgrades script is run. This then causes the updates not to be installed thereby making systems less secure - if security updates are enabled.

[Test Case]
Because this is a race condition, I haven't actually been able to recreate the bug. However, the fix is straightforward and testing for regressions should be sufficient.

1) On a freshly installed system.
2) Modify /etc/apt/apt.conf.d/50unattended-upgrades so that
Unattended-Upgrade::InstallOnShutdown "true"; is not commented out.
3) Reboot and watch updates being installed
4) Verify updates were installed by reviewing /var/log/unattended-upgrades/unattended-upgrades.log
5) Uncomment -updates in 50unattended-upgrades

Repeats steps 3 and 4 again. There isn't any need to run unattended-upgrades in *not* shutdown mode because we have only modified the systemd service file.

[Regression Potential]
There is little chance of a regression, but because of the change (and this feature now working better) its possible the shutdown process will take longer and surprise users.

Original Description
--------------------

Using unattended-upgrades 0.90 in "Shutdown mode" on Ubuntu Xenial, we encounter the following systemd dependencies issues :
- The network is often down when unattended-upgrades is running, so packages can not be downloaded (can be mitigated by using APT::Periodic::Download-Upgradeable-Packages "1";) :
    => ERROR An error occurred: 'Could not resolve host: ****.fr'
    => ERROR The URI 'https://****.fr:33000/ubuntu-security/pool/main/libi/libidn/libidn11_1.32-3ubuntu1.1_amd64.deb' failed to download, aborting
- Important mountpoints like /boot are unmounted before unattended-upgrades is running, so newer kernels can not be installed properly (ramdisk and grub configuration can not be generated)

[Brian Murray (brian-murray)]

The service possibly needs a relationship with the network service.

[Unit]
Description=Unattended Upgrades Shutdown
DefaultDependencies=no
Before=shutdown.target reboot.target halt.target
Documentation=man:unattended-upgrade(8)

[Service]
Type=oneshot
ExecStart=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
TimeoutStartSec=900

[Install]
WantedBy=shutdown.target

[Martin Pitt (pitti)]

If a unit needs to get stopped before the network gets stopped, you shold add "After=network.target". (See man systemd.special).

[Martin Pitt (pitti)]

Oh, actually not -- that would be for a service which gets *stopped* during shutdown (i. e. the usual way). If you want to *start* on shutdown, you need to be Before=network.target.

[Brian Murray (brian-murray)]

Before=local-fs.target would be needed for the mountpoints like /boot.

[Brian Murray (brian-murray)]

I've created an upstream pull request with these fixes.

https://github.com/mvo5/unattended-upgrades/pull/38

[Brian Murray (brian-murray)]

This should be fixed in Zesty now:

unattended-upgrades (0.93.1) unstable; urgency=medium

  [ Brian Murray ]
  * Create logfile_dpkg if it doesn't exist
  * Make sure the statedir exists too
  * ensure network and local-fs are available in shutdown
...
-- Michael Vogt <email address hidden> Sun, 11 Dec 2016 11:31:26 +0100

[Brian Murray (brian-murray)]

I want to write a test case for this SRU, where did you see the error messages that appear in the bug description? Were they in /var/log/unattended-upgrades/unattended-upgrades.log?

[Brian Murray (brian-murray)]

Sorry, I read the code and see that it is producing those messages.

unattended-upgrade: print(_("An error occurred: '%s'") % item.error_text)
unattended-upgrade: logging.error(_("An error occurred: '%s'"), item.error_text)
Binary file unattended-upgrade.pyc matches

[Timo Aaltonen (tjaalton)]

Hello Guillaume, or anyone else affected,

Accepted unattended-upgrades into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/0.92ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Timo Aaltonen (tjaalton)]

Hello Guillaume, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/0.90ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

verification-done for yakkety:

2016-12-20 10:41:41,532 INFO Starting unattended upgrades script
2016-12-20 10:41:41,532 INFO Allowed origins are: ['o=Ubuntu,a=yakkety', 'o=Ubuntu,a=yakkety-security']
2016-12-20 10:43:35,014 INFO Packages that will be upgraded: apport apport-gtk apt apt-transport-https apt-utils bind9-host curl dbus dbus-user-session dbus-x11 dnsutils firefox firefox-locale-en ghostscript ghostscript-x gstreamer1.0-plugins-good gstreamer1.0-pulseaudio imagemagick imagemagick-6.q16 imagemagick-common libapt-inst2.0 libapt-pkg5.0 libbind9-140 libcurl3 libcurl3-gnutls libdbus-1-3 libdns-export162 libdns162 libgd3 libgs9 libgs9-common libgstreamer-plugins-good1.0-0 libisc-export160 libisc160 libisccc140 libisccfg140 liblwres141 libmagickcore-6.q16-2 libmagickcore-6.q16-2-extra libmagickwand-6.q16-2 liboxideqt-qmlplugin liboxideqtcore0 liboxideqtquick0 libsmbclient libwbclient0 linux-generic linux-headers-generic linux-image-generic linux-libc-dev oxideqt-codecs python3-apport python3-cryptography python3-problem-report samba-libs tar thunderbird thunderbird-gnome-support thunderbird-locale-en thunderbird-locale-en-us tzdata vim vim-common vim-runtime vim-tiny
2016-12-20 10:43:35,014 INFO Writing dpkg log to '/var/log/unattended-upgrades/unattended-upgrades-dpkg.log'
2016-12-20 10:45:35,240 INFO All upgrades installed
2016-12-20 10:47:15,408 INFO Initial blacklisted packages:
2016-12-20 10:47:15,409 INFO Initial whitelisted packages:
2016-12-20 10:47:15,409 INFO Starting unattended upgrades script
2016-12-20 10:47:15,409 INFO Allowed origins are: ['o=Ubuntu,a=yakkety', 'o=Ubuntu,a=yakkety-security', 'o=Ubuntu,a=yakkety-updates']
2016-12-20 10:48:57,310 INFO Packages that will be upgraded: cups-browsed cups-filters cups-filters-core-drivers deja-dup file-roller gnome-software gnome-software-common ifupdown im-config libc-bin libc-dev-bin libc6 libc6-dbg libc6-dev libcupsfilters1 libfcitx-config4 libfcitx-gclient0 libfcitx-utils0 libfontembed1 libgail-common libgail18 libgspell-1-1 libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libmetacity0 libnautilus-extension1a libnss-resolve libpam-systemd libpulse-mainloop-glib0 libpulse0 libpulsedsp libsystemd0 libudev1 light-themes locales metacity metacity-common multiarch-support nautilus nautilus-data pulseaudio pulseaudio-module-bluetooth pulseaudio-utils python3-distupgrade qml-module-ubuntu-web qtdeclarative5-ubuntu-web-plugin snapd suru-icon-theme systemd systemd-sysv ubuntu-artwork ubuntu-mobile-icons ubuntu-mono ubuntu-release-upgrader-core ubuntu-release-upgrader-gtk ubuntu-software udev webapp-container webbrowser-app xmir xserver-common xserver-xorg-core
2016-12-20 10:48:57,310 INFO Writing dpkg log to '/var/log/unattended-upgrades/unattended-upgrades-dpkg.log'
2016-12-20 10:50:02,211 INFO All upgrades installed

bdmurray@clean-yakkety-amd64:~$ apt-cache policy unattended-upgrades
unattended-upgrades:
  Installed: 0.92ubuntu1.2
  Candidate: 0.92ubuntu1.2
  Version table:
 *** 0.92ubuntu1.2 100
        100 /var/lib/dpkg/status

[Brian Murray (brian-murray)]

2016-12-20 11:01:59,325 INFO Starting unattended upgrades script
2016-12-20 11:01:59,326 INFO Allowed origins are: ['o=Ubuntu,a=xenial', 'o=Ubuntu,a=xenial-security']
2016-12-20 11:04:43,551 INFO Packages that will be upgraded: apport apport-gtk apt apt-transport-https apt-utils bind9-host cpp-5 curl dbus dbus-x11 dnsutils firefox firefox-locale-en g++-5 gcc-5 gcc-5-base ghostscript ghostscript-x gir1.2-gst-plugins-base-1.0 gstreamer1.0-alsa gstreamer1.0-plugins-base gstreamer1.0-plugins-base-apps gstreamer1.0-plugins-good gstreamer1.0-pulseaudio gstreamer1.0-x imagemagick imagemagick-6.q16 imagemagick-common libapt-inst2.0 libapt-pkg5.0 libasan2 libatomic1 libbind9-140 libcc1-0 libcilkrts5 libcurl3 libcurl3-gnutls libdbus-1-3 libdns-export162 libdns162 libgcc-5-dev libgd3 libgomp1 libgs9 libgs9-common libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0 libisc-export160 libisc160 libisccc140 libisccfg140 libitm1 liblsan0 liblwres141 libmagickcore-6.q16-2 libmagickcore-6.q16-2-extra libmagickwand-6.q16-2 libmpx0 liboxideqt-qmlplugin liboxideqtcore0 liboxideqtquick0 libpython2.7 libpython2.7-minimal libpython2.7-stdlib libpython3.5 libpython3.5-minimal libpython3.5-stdlib libquadmath0 libsmbclient libstdc++-5-dev libstdc++6 libtracker-sparql-1.0-0 libtsan0 libubsan0 libwbclient0 linux-firmware linux-generic linux-headers-generic linux-image-generic linux-libc-dev oxideqt-codecs python2.7 python2.7-minimal python3-apport python3-cryptography python3-problem-report python3.5 python3.5-minimal samba-libs tar thunderbird thunderbird-gnome-support thunderbird-locale-en thunderbird-locale-en-us tzdata vim vim-common vim-runtime vim-tiny
2016-12-20 11:04:43,552 INFO Writing dpkg log to '/var/log/unattended-upgrades/unattended-upgrades-dpkg.log'
2016-12-20 11:06:59,150 INFO All upgrades installed
2016-12-20 11:27:12,868 INFO Starting unattended upgrades script
2016-12-20 11:27:12,868 INFO Allowed origins are: ['o=Ubuntu,a=xenial', 'o=Ubuntu,a=xenial-security', 'o=Ubuntu,a=xenial-updates']
2016-12-20 11:29:45,097 INFO Packages that will be upgraded: accountsservice apparmor base-files bind9-host deja-dup distro-info-data dnsutils gir1.2-dbusmenu-glib-0.4 gnome-calculator gnome-software gnome-software-common ifupdown im-config init init-system-helpers initramfs-tools initramfs-tools-bin initramfs-tools-core isc-dhcp-client isc-dhcp-common kbd language-pack-en language-pack-gnome-en libaccountsservice0 libapparmor-perl libapparmor1 libappstream-glib8 libbind9-140 libc-bin libc-dev-bin libc6 libc6-dbg libc6-dev libdbusmenu-glib4 libdbusmenu-gtk3-4 libdbusmenu-gtk4 libdns-export162 libdns162 libfcitx-config4 libfcitx-gclient0 libfcitx-utils0 libisc-export160 libisc160 libisccc140 libisccfg140 liblightdm-gobject-1-0 liblwres141 libmetacity-private3a libnm-glib-vpn1 libnm-glib4 libnm-util2 libnm0 libpam-systemd libprocps4 libpulse-mainloop-glib0 libpulse0 libpulsedsp libqt5core5a libqt5dbus5 libqt5gui5 libqt5network5 libqt5opengl5 libqt5printsupport5 libqt5sql5 libqt5sql5-sqlite libqt5test5 libqt5widgets5 libqt5xml5 libsystemd0 libudev1 libwhoopsie0 light-themes lightdm linux-firmware locales metacity metacity-common multiarch-support network-manage...

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for unattended-upgrades has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package unattended-upgrades - 0.90ubuntu0.3

---------------
unattended-upgrades (0.90ubuntu0.3) xenial-proposed; urgency=medium

  * Ensure that network and local filesystems are available when running
    unattended-upgrades during shutdown mode. (LP: #1618900)

 -- Brian Murray <email address hidden> Wed, 14 Dec 2016 13:38:32 -0800

[Launchpad Janitor (janitor)]

This bug was fixed in the package unattended-upgrades - 0.92ubuntu1.2

---------------
unattended-upgrades (0.92ubuntu1.2) yakkety; urgency=medium

  * Ensure that network and local filesystems are available when running
    unattended-upgrades during shutdown mode. (LP: #1618900)

 -- Brian Murray <email address hidden> Wed, 14 Dec 2016 13:31:31 -0800