apt/unattended-upgrades stalls shutdown

Bug #1661611 reported by Jogarem on 2017-02-03
74
This bug affects 12 people
Affects Status Importance Assigned to Milestone
unattended-upgrades (Ubuntu)
High
Unassigned

Bug Description

When unattended-upgrades is installed, 9 out of 10 shutdowns/reboots hang while "starting unattended upgrades shutdown". This hang stalls the shutdown process for 5-10 mins.

If I disable unnattended-upgrades via the /etc/apt/apt.conf.d/20auto-upgrades and/or 50unattended-upgrades, the problems occurs.

If I terminate the service before shutdown/reboot (sudo service unattended-upgrades stop) the problem still occurs.

If I remove the package (sudo apt remove unattended-upgrades) the problem no longer occurs.

This occurs on a freshly installed version of Ubuntu Server 16.04.1 (both unattended-upgrades installed via install GUI or manual install of unattended-upgrades)

Both Kern.log & syslog do not show the shutdown process (I believe because the filesystems have already unmounted)

Original report: http://askubuntu.com/questions/878630/apt-unattended-upgrades-stalls-shutdown

Jogarem (jogi) wrote :

Solution:

 - fix /usr/share/unattended-upgrades/unattended-upgrades-shutdown to expect "false" instead of "False"

patching /usr/share/unattended-upgrades/unattended-upgrade-shutdown:

--- /tmp/unattended-upgrade-shutdown 2017-02-03 14:53:03.238103238 +0100
+++ /tmp/unattended-upgrade-shutdown_fix 2017-02-03 14:53:17.685589001 +0100
@@ -117,7 +117,7 @@
     # run it
     p = None
     apt_pkg.init_config()
- if apt_pkg.config.find_b("Unattended-Upgrade::InstallOnShutdown", False):
+ if apt_pkg.config.find_b("Unattended-Upgrade::InstallOnShutdown", false):
         env = copy.copy(os.environ)
         env["UNATTENDED_UPGRADES_FORCE_INSTALL_ON_SHUTDOWN"] = "1"
         logging.debug("starting unattended-upgrades in shutdown mode")

The above solves the issue for me.

description: updated
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in unattended-upgrades (Ubuntu):
status: New → Confirmed
Mirzet Kadic (caracal-enl) wrote :

I'am effected by the same bug and here is a analysis made by someone else:
http://askubuntu.com/questions/878630/apt-unattended-upgrades-stalls-shutdown

The suggested patch works well. I hope it won't take too long to merge it.

tags: added: xenial
Changed in unattended-upgrades (Ubuntu):
importance: Undecided → High
status: Confirmed → Triaged
Brian Murray (brian-murray) wrote :

Could you provide the configuration line "Unattended-Upgrade::InstallOnShutdown" from /etc/apt/apt.conf.d/20auto-upgrades and 50unattended-upgrades? Thanks in advance!

Changed in unattended-upgrades (Ubuntu):
status: Triaged → Incomplete
Brian Murray (brian-murray) wrote :

apt_pkg.config.find_b() returns a boolean not a string.

"find_b(key[, default=False]) → bool Return the boolean value stored at key, or the value given by the bool object default if the requested option is not set."

From apt.alioth.debian.org/python-apt-doc/library/apt_pkg.html#apt_pkg.config.find_b

So it's not a bug in unattended-upgrade-shutdown as checking for False is correct.

Brian Murray (brian-murray) wrote :

Actually, the False part is setting the default value for the key if it doesn't exit.

Jogarem (jogi) wrote :

Well ok that was my first thought,too (boolean) so my first tests were misleading then. I tested a little bit more and find something maybe more useful.

When using Ubuntu 16.04.2 fresh installed and I set in /etc/apt/apt.conf.d/50unattended-upgrades:

Unattended-Upgrade::InstallOnShutdown "true";

it seems to have no effect. Upgrades are there but they do not get installed. I get no plymouth message on shutdown.
Same when set to "True".

When I change /usr/share/unattended-upgrades/unattended-upgrade-shutdown:

(line 120 again)

    # check if we need to run unattended-upgrades on shutdown and if so,
    # run it
    p = None
    apt_pkg.init_config()
before << if apt_pkg.config.find_b("Unattended-Upgrade::InstallOnShutdown", False):
after >> if apt_pkg.config.find_b("Unattended-Upgrade::InstallOnShutdown", False) == True:
        env = copy.copy(os.environ)
        env["UNATTENDED_UPGRADES_FORCE_INSTALL_ON_SHUTDOWN"] = "1"

Then it seems to work like expected.

Now when i set:

"Unattended-Upgrade::InstallOnShutdown "true";

in /etc/apt/apt.conf.d/50unattended-upgrades it starts upgrading on shutdown (I see the plymouth message "Unattended Upgrades..."). When I set it to "false" or just outcomment the whole line it do nothing on shutdown. So exact the behavior I expect.

Well I'm not a python expert but it seems to me that the <if True> thing works not as it should or maybe has changed in python3 some how? It may affect all other statements as well then (like MailOnlyOnError etc) but I haven't tested them yet...

With the above patch it works for me as it should. At least for the shutdown setting.

Jogarem (jogi) wrote :

hm no. sometimes it works sometimes not.. :(

SergeiFranco (sergei-franco) wrote :

Regarding presence of "Unattended-Upgrade::InstallOnShutdown" in /etc/apt/apt.conf.d/20auto-upgrades and /etc/apt/apt.conf.d/50unattended-upgrades:

It does not exist in /etc/apt/apt.conf.d/20auto-upgrades
and it is commented out in /etc/apt/apt.conf.d/50unattended-upgrades ,

Eg: //Unattended-Upgrade::InstallOnShutdown "true";

Changed in unattended-upgrades (Ubuntu):
status: Incomplete → Confirmed
kay (kay-diam) wrote :

It doesn't relate to "/usr/share/unattended-upgrades/unattended-upgrade-shutdown" file at all. When you "fix" it - it just fails because of the invalid python syntax. And shutdown just works.

The real problem is systemd dependencies. Here is the log of the shutdown process:
Unattended upgrades should be started before filesystem unmount, but it starts after unmount:

[ OK ] Unmounted /home.
[ OK ] Stopped File System Check on /dev/mapper/VG-home.
[ OK ] Unmounted /var.
[ OK ] Stopped File System Check on /dev/mapper/VG-var.
[ OK ] Unmounted /boot.
[ OK ] Stopped File System Check on /dev/mapper/VG-boot.
[ OK ] Removed slice system-systemd\x2dfsck.slice.
[ OK ] Stopped Login to default iSCSI targets.
         Stopping iSCSI initiator daemon (iscsid)...
[ OK ] Stopped iSCSI initiator daemon (iscsid).
[ OK ] Stopped target Network is Online.
[ OK ] Stopped target Network.
[ OK ] Stopped Initial cloud-init job (metadata service crawler).
         Stopping Raise network interfaces...
[ OK ] Stopped Raise network interfaces.
[ OK ] Stopped target Local File Systems.
         Unmounting /run/user/1000...
         Starting Unattended Upgrades Shutdown...
[ OK ] Stopped Apply Kernel Variables.
[ OK ] Stopped target Network (Pre).
[ OK ] Stopped Initial cloud-init job (pre-networking).
[ OK ] Stopped Load Kernel Modules.
[ OK ] Unmounted /run/user/1000.
[ OK ] Reached target Unmount All Filesystems.
[ OK ] Stopped target Local File Systems (Pre).
[ OK ] Stopped Create Static Device Nodes in /dev.
         Stopping Monitoring of LVM2 mirrors... dmeventd or progress polling...
[ OK ] Stopped Remount Root and Kernel File Systems.
[ OK ] Stopped Monitoring of LVM2 mirrors,...ng dmeventd or progress polling.
         Stopping LVM2 metadata daemon...
[ OK ] Stopped LVM2 metadata daemon.

And it is reproducible with LVM only. Have to spend some time on investigation.

SergeiFranco (sergei-franco) wrote :

I can confirm this bug also happens machines without LVM volumes.

Louis Bouchard (louis) wrote :

This is a duplicate of another bug I'm working on and is caused by a systemd ordering issue.

baas (pindapower) wrote :

@Louis Bouchard (louis-bouchard) do you have a link to the other bug? I'm interested to follow the status off this bug.

Herbert (herbert-fischer) wrote :

Up +100

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers