unattended-upgrade-shutdown hangs when /var is a separate filesystem

Bug #1654600 reported by Erik Mouw on 2017-01-06
176
This bug affects 31 people
Affects Status Importance Assigned to Milestone
unattended-upgrades (Debian)
New
Unknown
unattended-upgrades (Ubuntu)
High
Unassigned
Xenial
High
Louis Bouchard
Yakkety
High
Louis Bouchard
Zesty
High
Louis Bouchard

Bug Description

[SRU justification]
This fix is needed to make sure that the system does not hang on shutdown when /var is a speparate file system

[Impact]
System can hang up to 10 minutes if not fixed.

[Fix]
Change the systemd unit's ExecStart to an ExecStop so the unit is correctly sequenced.
Change WantedBy to multi-user.target. This requires working around Debian Bug #797108 which causes the new unit not to be enabled.
Remove the unneeded override_dh_isntallinit
Add Default-Start levels to the SysV initscript
Improve DEP8 testing

[Test Case]
In a VM with /var separated from the / file system, shutdown the VM. It will hang for 10 minutes

[Regression]
Upgrade has been tested on Xenial, Yakkety, Zesty. do-release-upgrade has been tested from Trusty to Xenial. All behave as expected.

A change of behavior may occur now that the systemd unit is correctly enabled, which would make functionalities like InstallOnShutdown to work as expected whereas it could have been broken previously. This cannot be considered as a regression as it is expected behavior.

Shutdown may be slowed down as it now correctly depends on /var and /boot to be available so the unit will run earlier than previously.

[Original description of the problem]
The systemd unit file unattended-upgrades.service is used to stop a running unattended-upgrade
process during shutdown. This unit file is running together with all filesystem
unmount services.

The unattended-upgrades service checks if the lockfile for unattended-upgrade
(in /var/run) exists, and if it does, there is an unattended-upgrade in progress
and the service will wait until it finishes (and therefore automatically wait at
shutdown).

However, if /var is a separate filesystem, it will get unmounted even though /var/run
is a tmpfs that's still mounted on top of the /var/run directory in the /var filesystem.
The unattended-upgrade script will fail to find lockfile, sleeps for 5 seconds, and
tries to check the lockfile again. After 10 minutes (the default timeout), it will finally
exit and the system will continue shutdown.

The problem is the error handling in /usr/share/unattended-upgrades/unattended-upgrade-shutdown
where it tries to lock itself:

    while True:
        res = apt_pkg.get_lock(options.lock_file)
        logging.debug("get_lock returned %i" % res)
        # exit here if there is no lock
        if res > 0:
            logging.debug("lock not taken")
            break
        lock_was_taken = True

The function apt_pkg.get_lock() either returns a file descriptor, or -1 on an error.
File descriptors are just C file descriptors, so they are always positive integers.
The code should check the result to be negative, not positive. I have attached a patch
to reverse the logic.

Additional information:

1)
Description: Ubuntu 16.04.1 LTS
Release: 16.04

2)
unattended-upgrades:
  Installed: 0.90ubuntu0.3
  Candidate: 0.90ubuntu0.3
  Version table:
 *** 0.90ubuntu0.3 500
        500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://nl.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages
        100 /var/lib/dpkg/status
     0.90 500
        500 http://nl.archive.ubuntu.com/ubuntu xenial/main amd64 Packages
        500 http://nl.archive.ubuntu.com/ubuntu xenial/main i386 Packages
3)
Fast reboot
4)
Very slow reboot (after a 10 minutes timeout)

Erik Mouw (erikzbhj) wrote :
Erik Mouw (erikzbhj) wrote :

I have also tried to let the systemd unit file for this service WantedBy the umount.target.
According to the systemd documentation that should make it run before the filesystems
are unmounted, but apparently that doesn't work. systemctl list-dependencies still shows
that the unattended-upgrade.service reverse depends on the shutdown.target.

tags: added: patch
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in unattended-upgrades (Ubuntu):
status: New → Confirmed
BSA (b-s-a) wrote :

I have same problem. I need to kill unattended-upgrades service manually to shutdown machine. This problem comes in 2017 year.

System is Ubuntu 16.04 x86_64.

Erik Mouw (erikzbhj) wrote :

I have narrowed down. This bug was introduced by the fix in #1618900.
That tries to fix the problem of the the unattended-upgrades.service
by letting it depend on network.target and local-fs.target.
However, that doesn't change any of the dependencies, it only makes
the race window larger.

After some experiments, the correct dependencies in the unit file appear to be:

Before=remote-fs.target local-fs.target

Which make it behave just like the SysV init script.

This change together with the patch I attached to this bug report
together should fix the issue and also fix #1618900 .

Regards,
Erik

Erik Mouw (erikzbhj) wrote :

FWIW, the init script /etc/init.d/unattended-upgrades should be
removed from systemd systems (Xenial, Yaketty).

Changed in unattended-upgrades (Ubuntu):
importance: Undecided → High
George Moutsopoulos (gmoutso) wrote :

I get this sometimes on shutdown but not always.

Erik, until the fix reaches users, should I:
systemctl disable unattended-upgrades.service
?

Louis Bouchard (louis) on 2017-01-26
Changed in unattended-upgrades (Ubuntu):
assignee: nobody → Louis Bouchard (louis-bouchard)
Changed in unattended-upgrades (Ubuntu Xenial):
assignee: nobody → Louis Bouchard (louis-bouchard)
Changed in unattended-upgrades (Ubuntu Yakkety):
assignee: nobody → Louis Bouchard (louis-bouchard)
Changed in unattended-upgrades (Ubuntu Xenial):
importance: Undecided → High
Changed in unattended-upgrades (Ubuntu Yakkety):
importance: Undecided → High
Changed in unattended-upgrades (Ubuntu Xenial):
status: New → Confirmed
Louis Bouchard (louis) wrote :

Hello,

This logic is correct. apt_pkg.get_lock(options.lock_file) is trying to acquire a lockfile (/var/run/unattended-upgrades.lock) which is only possible if no unattended-upgrade is already running. If it can acquire the lock, it means that NO unattended-upgrade is running, hence shutdown may proceed.

 while True:
        res = apt_pkg.get_lock(options.lock_file)
        logging.debug("get_lock returned %i" % res)
        # exit here if there is no lock
        if res > 0:
            logging.debug("lock not taken")
            break
        lock_was_taken = True

The problem is that apt_pkg.get_lock(options.lock_file) will also return -1 if the path to the lock file doesn't exist, which is the case when /var is unmounted, hence /var/run is no longer present (/var/run is a symlink to /run).

I'm working on identifying the proper systemd order to make sure that /var/run is still accessible.

Download full text (5.0 KiB)

You're right, I thought it was the lockfile for the shutdown helper only.

Somewhere else in this thread I already mentioned that a dependency on the local_fs and remote_fs targets fixes the systemd order:

Before=remote-fs.target local-fs.target

That makes sure the service is run before all file systems are unmounted. If you add the shutdown.target to that rule, systemd somehow thinks it should be run just before shutdown which is obviously too late. I'm not sure that is a systemd bug or a systemd feature, but either way it doesn't do what we want.

Regards,

Erik

--
Erik Mouw

> Op 26 jan. 2017 om 18:00 heeft Louis Bouchard <email address hidden> het volgende geschreven:
>
> Hello,
>
> This logic is correct. apt_pkg.get_lock(options.lock_file) is trying to
> acquire a lockfile (/var/run/unattended-upgrades.lock) which is only
> possible if no unattended-upgrade is already running. If it can acquire
> the lock, it means that NO unattended-upgrade is running, hence shutdown
> may proceed.
>
> while True:
> res = apt_pkg.get_lock(options.lock_file)
> logging.debug("get_lock returned %i" % res)
> # exit here if there is no lock
> if res > 0:
> logging.debug("lock not taken")
> break
> lock_was_taken = True
>
> The problem is that apt_pkg.get_lock(options.lock_file) will also return
> -1 if the path to the lock file doesn't exist, which is the case when
> /var is unmounted, hence /var/run is no longer present (/var/run is a
> symlink to /run).
>
> I'm working on identifying the proper systemd order to make sure that
> /var/run is still accessible.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1654600
>
> Title:
> unattended-upgrade-shutdown hangs when /var is a separate filesystem
>
> Status in unattended-upgrades package in Ubuntu:
> Confirmed
> Status in unattended-upgrades source package in Xenial:
> Confirmed
> Status in unattended-upgrades source package in Yakkety:
> New
>
> Bug description:
> The systemd unit file unattended-upgrades.service is used to stop a running unattended-upgrade
> process during shutdown. This unit file is running together with all filesystem
> unmount services.
>
> The unattended-upgrades service checks if the lockfile for unattended-upgrade
> (in /var/run) exists, and if it does, there is an unattended-upgrade in progress
> and the service will wait until it finishes (and therefore automatically wait at
> shutdown).
>
> However, if /var is a separate filesystem, it will get unmoun...

Read more...

Louis Bouchard (louis) wrote :

Ok, here is how I understand the situation.

According to the systemd documentation about [Unit] Before= :

"Given two units with any ordering dependency between them, if one unit is shut down and the other is started up, the shutdown is ordered before the start-up. It doesn't matter if the ordering dependency is After= or Before=."

To me this means that network.service & local-fs.service will be shutdown _BEFORE_ unattended-upgrades-shutdown runs, hence /var var will be unmounted when it runs.

My current solution is to turn the unattended-upgrades.service ExecStart= into an ExecStop= so the unit will run as a shutdown instead of a start when the system shuts down :

[Unit]
Description=Unattended Upgrades Shutdown
DefaultDependencies=no
Before=shutdown.target reboot.target halt.target network.target local-fs.target
Documentation=man:unattended-upgrade(8)

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown --debug
TimeoutStopSec=900

[Install]
WantedBy=shutdown.target

Preliminary tests seem to run fine but I want to get confirmation on such a change by someone more expert with systemd that I am.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in unattended-upgrades (Ubuntu Yakkety):
status: New → Confirmed
Changed in unattended-upgrades (Debian):
status: Unknown → New
Nils Meyer (nils-nm) wrote :

Is there a good reason not to simply change the path to the lock file from /var/run to /run?

jftempo (jfb-tempo-consulting) wrote :

As a workaround I created a symlink to /run inside the mounted /var :
mount -o bind / /mnt
ln -s /run /mnt/var/
umount /mnt

LeaseWeb (leaseweb) wrote :

Bug #1661611 seems to be identical

Louis Bouchard (louis) on 2017-03-02
Changed in unattended-upgrades (Ubuntu):
status: Confirmed → In Progress
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package unattended-upgrades - 0.93.1ubuntu2

---------------
unattended-upgrades (0.93.1ubuntu2) zesty; urgency=medium

  * The systemd unit needs to be an ExecStop since it is is activated on
    shutdown. Otherwise, it will get scheduled after completion of
    the local-fs.target. In the case where /var is a separate
    filesystem, unattended-upgrade-shutdown will hang until timeout
    since /var/run is expected but no longer there (LP: #1654600)

 -- Louis Bouchard <email address hidden> Thu, 02 Mar 2017 16:55:26 +0100

Changed in unattended-upgrades (Ubuntu):
status: In Progress → Fix Released
kay (kay-diam) wrote :
Download full text (5.7 KiB)

Tested the fix. It works. Thanks!

quick fix:

sed -i "s#ExecStart=#RemainAfterExit=yes\nExecStop=#;" /lib/systemd/system/unattended-upgrades.service

But now it seems that it doesn't run it on shutdown. I used the following debug service:

ExecStop=/bin/bash -c 'echo -e "\nhello######################\n" > /dev/ttyS0'

And it doesn't print message in serial console. But when you start and stop this unit manually - it prints.

[ OK ] Stopped target Graphical Interface.
         Stopping Accounts Service...
         Stopping User Manager for UID 1000...
         Stopping Session 1 of user ubuntu.
         Stopping ACPI event daemon...
[ OK ] Stopped target Cloud-init target.
[ OK ] Stopped Execute cloud user/final scripts.
[ OK ] Stopped target Multi-User System.
         Stopping Deferred execution scheduler...
         Stopping LXD - container startup/shutdown...
         Stopping D-Bus System Message Bus...
         Stopping LSB: daemon to balance interrupts for SMP systems...
         Stopping LSB: Set the CPU Frequency Scaling governor to "ondemand"...
         Stopping LSB: MD monitoring daemon...
         Stopping Regular background program processing daemon...
         Stopping FUSE filesystem for LXC...
         Stopping OpenBSD Secure Shell server...
         Stopping LSB: Record successful boot for GRUB...
[ OK ] Stopped target Timers.
[ OK ] Stopped Timer to automatically refresh installed snaps.
[ OK ] Stopped Daily apt activities.
[ OK ] Stopped Daily Cleanup of Temporary Directories.
[ OK ] Stopped target Login Prompts.
         Stopping Getty on tty1...
         Stopping Serial Getty on ttyS0...
[ OK ] Stopped Apply the settings specified in cloud-config.
[ OK ] Stopped target Cloud-config availability.
         Stopping Snappy daemon...
         Stopping System Logging Service...
[ OK ] Stopped target System Time Synchronized.
         Stopping LSB: automatic crash report generation...
[ OK ] Closed Load/Save RF Kill Switch Status /dev/rfkill Watch.
         Stopping Authenticate and Authorize Users to Run Privileged Tasks...
[ OK ] Unmounted /var/lib/lxcfs.
[ OK ] Stopped System Logging Service.
[ OK ] Stopped Deferred execution scheduler.
[ OK ] Stopped OpenBSD Secure Shell server.
[ OK ] Stopped Accounts Service.
[ OK ] Stopped Snappy daemon.
[ OK ] Stopped ACPI event daemon.
[ OK ] Stopped Authenticate and Authorize Users to Run Privileged Tasks.
[ OK ] Stopped Serial Getty on ttyS0.
[ OK ] Stopped Regular background program processing daemon.
[ OK ] Stopped Getty on tty1.
[ OK ] Stopped User Manager for UID 1000.
[ OK ] Stopped Session 1 of user ubuntu.
[ OK ] Stopped D-Bus System Message Bus.
[ OK ] Stopped FUSE filesystem for LXC.
[ OK ] Stopped LXD - container startup/shutdown.
[ OK ] Stopped LSB: MD monitoring daemon.
[ OK ] Stopped LSB: Record successful boot for GRUB.
[ OK ] Stopped LSB: daemon to balance interrupts for SMP systems.
[ OK ] Stopped LSB: automatic crash report generation.
[ OK ] Stopped LSB: Set the CPU Frequency Scaling governor to "ondemand".
[ OK ] Stopped User Manager for UID 1000.
[ OK ] Removed slice User Slice of ubuntu.
   ...

Read more...

Louis Bouchard (louis) on 2017-03-06
description: updated
Changed in unattended-upgrades (Ubuntu Xenial):
status: Confirmed → In Progress
Changed in unattended-upgrades (Ubuntu Yakkety):
status: Confirmed → In Progress
SergeiFranco (sergei-franco) wrote :

I have tested this and it does NOT work.

I fixed the issue in couple of different ways:

One way is completely remove the unit (who in right mind does install updates during reboot on production servers?). I will be actively doing that on all machines I encounter.

Another way is this:
sed -i 's#/var/run/#/run/#g' /usr/share/unattended-upgrades/unattended-upgrade-shutdown
So the problem with that sed is that the logging part will fail. So not really a solution.

Systemd has proven to be inconsistent pile of crap so far...

I could not make systemd to run this unit before local-fs.target. It appears that the documentation is misleading...

Tom H (tomh0665) wrote :

Before=shutdown.target reboot.target halt.target network.target local-fs.target

should be

Before=shutdown.target reboot.target halt.target network.target
After=local-fs.target

because, AIUI, at shutdown, "Before" means "After" and "After" means "Before".

So unattended-upgrades.service's ExecStop'll start before local-fs.target, so before "/var" is unmounted and "/var/log" and "/var/run" are gone.

[Why is "/var/run" used? This is a native Ubuntu package; are there any versions of Ubuntu where "/var/run" isn't a symlink to "/run"?]

Louis Bouchard (louis) wrote :

Hello,

I am curious to hear about the context of your tests : my Zesty system with a separated /var does show the problem prior to upgrade to unattended-upgrades-0.93.1ubuntu2 and no longer hangs when the new package is applied. There might be situations that I have not tested where the issue could still exists.

Your suggestion to replace /var/run by /var is incorrect : /var/run/unattended-upgrades.lock is placed by the unattended-upgrades script; Looking in /run for a lock file that has been put in /var/run is simply inadequate.

While replacing /var/run by /run across the board is a valid proposition, it requires more inspection than simply replacing its occurrences in one script.

Hello Erik, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/0.90ubuntu0.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in unattended-upgrades (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in unattended-upgrades (Ubuntu Yakkety):
status: In Progress → Fix Committed
Chris J Arges (arges) wrote :

Hello Erik, or anyone else affected,

Accepted unattended-upgrades into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/0.92ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

kay (kay-diam) wrote :

@arges,

I already noticed that upgrade procedure doesn't start at all with the new change. But actual shutdown/reboot hang has disappeared.

Louis Bouchard (louis) wrote :

@kay-diam

I don't understand your statement. The unit clearly get started. This is from your log :

[ OK ] Stopped target Local File Systems.
[ OK ] Started Unattended Upgrades Shutdown. <<<<<<<
         Unmounting /run/user/1000...
         Unmounting /tmp...
[ OK ] Stopped Apply Kernel Variables.

kay (kay-diam) wrote :

@louis-bouchard

"Started Unattended Upgrades Shutdown." means that the unit was started. Since the patch assumes that we don't use "ExecStart" this means nothing.

Upgrade procedure should be executed on "ExecStop". But it is not happening. I tried to set debug info inside the "ExecStop" instead of executing "unattended-upgrade-shutdown", but it doesn't work. I can provide you with the detailed step-by-step instructions on how to 100% reproduce the issue, but it will require KVM host, and running my scripts which will deploy VM with LVM.

Basic steps:

* follow https://github.com/kayrus/deploy-vm docs
* git clone https://github.com/kayrus/deploy-vm
* cd deploy-vm
* ./deploy_vms_cluster.sh -o ubuntu -c xenial
* follow https://github.com/kayrus/rescue-initramfs docs
* and reinstall VM from initramfs using "deploy_os_lvm.sh" script

Then your Xenial VM will stuck at reboot

kay (kay-diam) wrote :

BTW, KVM allows you to use serial console you can use it for debugging. But you have explicitly specify the order of defined "console" kernel parameters, i.e.:

this will specify VGA console as default output:
console=ttyS0 console=tty1

this will specify serial console as default output:
console=tty1 console=ttyS0

Louis Bouchard (louis) wrote :

@kay-diam:

I get what you mean. I also use KVM to debug the issue.

I'm going to switch to verification-failed to this one does not get released and push the investigation further as there is another issue that side-tracked me :

If Unattended-Upgrade::InstallOnShutdown is set to "true", it will still fail as the network will no longer be available when it gets to access the archive so this must also be fixed.

...Louis

tags: added: verification-failed
removed: verification-needed
kay (kay-diam) wrote :

@louis-bouchard

I'd love to help you, but I'm in rush on my current job.

SergeiFranco (sergei-franco) wrote :

Hello,

Yes, I am fully aware that my /var/run->/run replacement in that script is a hack.

To be clear I am not testing in Zesty but Ubuntu 16.04.2 LTS Xenial.

The setup is following:
/dev/sda1 on / type xfs (rw,relatime,attr2,inode64,noquota)
/dev/sdc on /var type xfs (rw,relatime,attr2,inode64,noquota)

Every single machine we (my and my colleague) tested with the updated unit still hangs.

As var as I can tell they never reboot.

I have posted a question regarding this issue to systemd mailing list, and one of the responses was the following:

"""
> [Unit]
> Description=Unattended Upgrades Shutdown
> DefaultDependencies=no
> Before=shutdown.target reboot.target halt.target network.target
> local-fs.target

Well, you order if before local-fs.target, which means it is stopped
after local-fs.target. You get exactly what you ask for. If this is
wrong, fix unit definition, but we do not really know what is
appropriate for this service.

> Documentation=man:unattended-upgrade(8)
>
> [Service]
> Type=oneshot
> RemainAfterExit=yes
> ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown --debug
> TimeoutStopSec=900
>
> [Install]
> WantedBy=shutdown.target
>
> It appears no one who is involved in this fix understands systemd (which is
> very worrying!).
>
> How does one would fix this unit so it is ran before the file systems get
> unmounted?
>

Order it

After=local-fs.target
"""
"Obviously" according to systemd mailing list if I want to things to run Before I need to use After!

So I believe the confusion is with what exactly Before and After mean in systemd units.

Today I will be testing various combinations with After/Before to see if I can make sense of it.

SergeiFranco (sergei-franco) wrote :

Hello!

I have changed Before to After and it is working!!!!
Here is what the unit looks like now:

[Unit]
Description=Unattended Upgrades Shutdown
DefaultDependencies=no
After=shutdown.target reboot.target halt.target network.target local-fs.target
Documentation=man:unattended-upgrade(8)

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
TimeoutStopSec=900

[Install]
WantedBy=shutdown.target

So the proper fix would be changing the Service to ExecStop= and Unit to After=

Louis Bouchard (louis) wrote :

@sergei-franco:

From what I can tell, it works for you because changing the unit the way you did simply disable execution of the unit. When running with such a configuration either on Xenial or Zesty, I do not see any mention of "Unattended Upgrades Shutdown" in the console output.

When limiting to "After=network.target local-fs.target", I see the unit being started (which is basically a NOOP) but not stopping which is what we want.

I'm continuing my tests

Louis Bouchard (louis) on 2017-03-09
Changed in unattended-upgrades (Ubuntu):
status: Fix Released → In Progress
Changed in unattended-upgrades (Ubuntu Xenial):
status: Fix Committed → Triaged
Changed in unattended-upgrades (Ubuntu Yakkety):
status: Fix Committed → Triaged
Louis Bouchard (louis) wrote :

Here is a recap of my day of work :

The unattended-upgrades.service unit never runs the ExecStop because it is not enabled as we see here :

# systemctl status unattended-upgrades.service
● unattended-upgrades.service - Unattended Upgrades Shutdown
   Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled; vendor preset: enabled)
   Active: inactive (dead) <<<<<<<<<<<<<<<
     Docs: man:unattended-upgrade(8)

This is caused by a missing "Default-Start" header in /etc/init.d/unattended-upgrades. Trying to enable the unit leads to :

# systemctl enable unattended-upgrades.service
Synchronizing state of unattended-upgrades.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable unattended-upgrades
update-rc.d: error: unattended-upgrades Default-Start contains no runlevels, aborting.

I'm currently working on fixing all this in the packaging so we get a correctly working unit after the upgrade of the package.

I have also reworked the unit :

[Unit]
Description=Unattended Upgrades Shutdown
DefaultDependencies=no
After=network.target local-fs.target
RequiresMountsFor=/var/log /var/run
Documentation=man:unattended-upgrade(8)

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
TimeoutStopSec=900

[Install]
WantedBy=multi-user.target

With this configuration, the unit runs the ExecStop as expected. I also tested using Unattended-Upgrade::InstallOnShutdown "true" and it works as expected, which is to block the shutdown for upgrade with the network being available.

I should have that available for testing tomorrow.

kay (kay-diam) wrote :

Still doesn't work for me.

Louis Bouchard (louis) wrote :
Download full text (3.8 KiB)

Here is a recap of my work and current status :

#1) when a system has a separated /var, the unit will hang since it is looking
for /var/run to be present and it has been unmounted.

#2) when using Unattended-Upgrade::InstallOnShutdown "true"; the upgrade never
completes as the query to the online archive fails since the network is no
longer available.

#3) it is impossible to enable the unattended-upgrades.service unit. Here
is an example :

> $ systemctl status unattended-upgrades.service
> ● unattended-upgrades.service - Unattended Upgrades Shutdown
> Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled; vendor preset: enabled)
> Active: inactive (dead)
> Docs: man:unattended-upgrade(8)

According to the doc[2], during shutdown, even if Before= or After= is used, the
unit being started will only start after the Shutdown of its dependencies. Prior
to the recent addition of local-fs.target network.target, the unit ran quickly
enough for the /var to be still mounted. But even without these dependencies,
InstallOnShutdown would fail with the following :

> 2017-03-10 13:40:42,803 INFO Starting unattended upgrades script
> 2017-03-10 13:40:42,803 INFO Allowed origins are: ['o=Ubuntu,a=zesty', 'o=Ubuntu,a=zesty-security']
> 2017-03-10 13:41:40,554 ERROR An error occurred: 'Cannot initiate the connection to 192.168.200.3:8000 (192.168.200.3). - connect (101: Network is unreachable)'
> 2017-03-10 13:41:40,555 ERROR The URI 'http://fr.archive.ubuntu.com/ubuntu/pool/main/i/init-system-helpers/init-system-helpers_1.47_all.deb' failed to download, aborting

When trying to switch the unit to an ExecStop=, we find that the Stop never
runs. This is caused by the fact that the unit is disabled (#3). Trying to
enable the unit leads to :

> # systemctl enable unattended-upgrades
> Synchronizing state of unattended-upgrades.service with SysV service script with /lib/systemd/systemd-sysv-install.
> Executing: /lib/systemd/systemd-sysv-install enable unattended-upgrades
> update-rc.d: error: unattended-upgrades Default-Start contains no runlevels, aborting.

Adding runlevels 2 3 4 5 fixes this then the unit can be enabled. So we get
to a unit that looks like this :

> [Unit]
> Description=Unattended Upgrades Shutdown
> DefaultDependencies=no
> After=network.target local-fs.target
> RequiresMountsFor=/var/run /var/log
> Documentation=man:unattended-upgrade(8)
>
> [Service]
> Type=oneshot
> RemainAfterExit=yes
> ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
> TimeoutStopSec=900
>
> [Install]
> WantedBy=multi-user.target

Before= is replaced by After= as, during shutdown otherwise the unit does not
run. RequiresMountsFor= is added as both /var/log and /var/run are needed in
order to run correctly.

RemainAfterExit=yes is added so the unit appears as started. There is no longer
a requirement to have an ExecStart present.

WantedBy is switched to multi-user.target as on the way up, we do nothing and we
are no longer depending on anything related to shutdown.

Now this only works IF /var is a separate FS. The reason for that is the
presence of DefaultDependencies=no. I don't think that it is required but was
the...

Read more...

Geoff Short (geoff-short) wrote :

The Unit file in #33 works for me, ie stops the 10 minute hang, when unattended-upgrades is not running.

- Xenial with /var as a separate FS, InstallOnShutdown="false"

I will try to create a test case where unattended-upgrades is running when shutdown is called.

Geoff Short (geoff-short) wrote :

Follow-up to #34, on a system where unattended-upgrades happens to be running when shutdown is called, the shutdown does not wait, therefore any running dpkg is killed and the package left in an inconsistent state.

 - Xenial with single partition, InstallOnShutdown not defined.

This may be a separate issue, but I'll investigate further before raising it as such.

Louis Bouchard (louis) wrote :

A quick update on this issue. I have a solution ready to fix this. Unfortunately it requires changing the WantedBy= statement, which triggers a deb-systemd-helper bug : https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=797108

This has the effect of leaving the service disabled after upgrade. I'm working on fixing that.

Louis Bouchard (louis) on 2017-04-04
tags: added: sts-sru-needed
Louis Bouchard (louis) wrote :

Targetting Zesty as the fix will come after release so as a SRU.

Louis Bouchard (louis) wrote :

Brief update to tell everyone that I have a fix ready for all stable releases. I will update it to Artful soon and the SRU will follow soon.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package unattended-upgrades - 0.93.1ubuntu3

---------------
unattended-upgrades (0.93.1ubuntu3) artful; urgency=medium

  * Complete the solution for the unattended-upgrades.service unit not
    correctly working (LP: #1654600):
    - d/rules : Remove the override_dh_installinit. The stop option is no longer
      available so the command falls back to default. This is the normal
      behavior so the override is not required
    - d/unattended-upgrades.init : Add Default-Start runlevels, otherwise the
      unattended-upgrades.service unit cannot be enabled on boot
    - d/postinst : Cleanup the stop symlinks created by the wrong
      override_dh_installinit. Without that, the systemd unit cannot be
      enabled correctly.
      Force disable the service before deb-systemd-helper runs so the old
      symlink is not left dangling (workaround for Debian Bug #797108).
      Force enable and start of the systemd unit to work around Debian Bug #797108
      which fails to enable systemd units correctly when WantedBy= statement
      is changed which is the case here.
    - d/unattended-upgrades.service : Fix the service so it runs correctly on
      shutdown :
        Remove DefaultDependencies=no : Breaks normal shutdown dependencies
        Set After= to network.target and local-fs.target. Since our service is
        now ExecStop, it will run before network and local-fs become unavailable.
        Add RequiresMountsFor=/var/log /var/run /var/lib /boot : Necessary if
        /var is a separate file system. Set WantedBy= to multi-user.target
    - Add DEP8 tests to verify the following :
      Verify that the unattended-upgrades.service unit is enabled and started.
      Verify that InstallOnShutdown works when configured.

 -- Louis Bouchard <email address hidden> Mon, 24 Apr 2017 14:07:19 +0200

Changed in unattended-upgrades (Ubuntu):
status: In Progress → Fix Released
Louis Bouchard (louis) on 2017-04-25
description: updated
Louis Bouchard (louis) on 2017-04-25
Changed in unattended-upgrades (Ubuntu Yakkety):
status: Triaged → In Progress
Changed in unattended-upgrades (Ubuntu Xenial):
status: Triaged → In Progress
Changed in unattended-upgrades (Ubuntu):
assignee: Louis Bouchard (louis) → nobody
Brian Murray (brian-murray) wrote :

Hello Erik, or anyone else affected,

Accepted unattended-upgrades into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/0.93.1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in unattended-upgrades (Ubuntu Zesty):
status: In Progress → Fix Committed
tags: removed: verification-failed
tags: added: verification-needed
Brian Murray (brian-murray) wrote :

Hello Erik, or anyone else affected,

Accepted unattended-upgrades into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/0.92ubuntu1.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in unattended-upgrades (Ubuntu Yakkety):
status: In Progress → Fix Committed
Brian Murray (brian-murray) wrote :

Hello Erik, or anyone else affected,

Accepted unattended-upgrades into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/unattended-upgrades/0.90ubuntu0.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in unattended-upgrades (Ubuntu Xenial):
status: In Progress → Fix Committed
Louis Bouchard (louis) on 2017-04-28
tags: added: verification-done
removed: sts-sru-needed verification-needed
Louis Bouchard (louis) wrote :

Test performed for verification-done for :
 - Xenial
 - Yakkety
 - Zesty
1) Installed unattended-upgrades from -proposed
2) Verified the status of the unattended-upgrades service. All displayed :

# systemctl status unattended-upgrades
● unattended-upgrades.service - Unattended Upgrades Shutdown
   Loaded: loaded (/lib/systemd/system/unattended-upgrades.service; enabled; vendor preset: enabled)
   Active: active (exited) since Fri 2017-04-28 14:02:35 CEST; 47s ago
     Docs: man:unattended-upgrade(8)

3) Enabled Unattended-Upgrade::InstallOnShutdown "true"; in /etc/apt/apt.conf.d/50unattended-upgrades and rebooted. All releases applied the upgradeable packages upon reboot (some had too many and the timeout ran out before end of update)

4) Ran apt -y dist-upgrade on all releases. Rebooted. systems rebooted normally.

5) Enabled trusty-proposed on a Trusty system and ran do-release-upgrade to go to Xenial. After completion of the full upgrade, unattended-upgrades service was active (as seen in #2)

6) Enabled zesty-proposed on a system with /var as a separate file system. Upgraded unattended-upgrades. After upgrade, unattended-upgrades service was active (as seen in #2)

7) Verified that the workaround for Debian Bug #809669 left the upgraded system in an adequate configuration :

cat /var/lib/systemd/deb-systemd-helper-enabled/unattended-upgrades.service.dsh-also
/etc/systemd/system/multi-user.target.wants/unattended-upgrades.service

# find /etc/systemd | grep unatt
/etc/systemd/system/multi-user.target.wants/unattended-upgrades.service

Tapan Maulik (tmaulik) wrote :

This resolved the issue.
Version of the package testes: unattended-upgrades 0.90ubuntu0.5
Ubuntu version - 16.04 xenial
verification-done

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.