Reboot hangs forever because /bin/systemctl --force reboot fails with "Failed to execute operation: Connection timed out"

[Marius Gedminas]

This morning unattended-upgrades had a reboot scheduled for 6:30 AM. The reboot process succeeded in turning off all network access but failed to actually reboot. As far as I can tell, this is because

    Sep 20 06:50:44 fridge systemd[1]: Starting Reboot...
    Sep 20 06:51:09 fridge systemctl[3886]: Failed to execute operation: Connection timed out
    Sep 20 06:53:54 fridge systemd[1]: systemd-reboot.service: Main process exited, code=exited, status=1/FAILURE
    Sep 20 06:53:54 fridge systemd[1]: Failed to start Reboot.
    Sep 20 06:53:54 fridge systemd[1]: systemd-reboot.service: Unit entered failed state.
    Sep 20 06:53:54 fridge systemd[1]: systemd-reboot.service: Failed with result 'exit-code'.

(Don't ask me why it took 20 minutes to get to this point. Most everything was already shut down by 06:35.)

After this the journal shows a few other random-looking service messages

    Sep 20 07:13:44 fridge systemd[1]: Stopping LVM2 metadata daemon...
    Sep 20 07:13:44 fridge systemd[1]: Stopped LVM2 metadata daemon.
    Sep 20 07:51:48 fridge systemd[1]: Started LVM2 metadata daemon.
    Sep 20 07:51:48 fridge systemd[1]: Stopped target Shutdown.

and the end of a cron session that initiated the unattended-upgrade shutdown -r 06:30 call, presumably

    Sep 20 07:52:54 fridge CRON[1176]: pam_unix(cron:session): session closed for user root

and then the machine sat idle waiting for keyboard input, which I provided in the form of Alt+SysRq+S,U,B a few hours later.

This has never happened before on this server, which is running Ubuntu 16.04 LTS since November 2016. I haven't tried to reproduce this (yet).

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: systemd 229-4ubuntu19
ProcVersionSignature: Ubuntu 4.4.0-96.119-generic 4.4.83
Uname: Linux 4.4.0-96-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.10
Architecture: amd64
Date: Wed Sep 20 11:10:34 2017
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-96-generic root=/dev/mapper/hostname-root ro nomdmonddf nomdmonisw
SourcePackage: systemd
UpgradeStatus: Upgraded to xenial on 2016-11-07 (316 days ago)
dmi.bios.date: 10/02/2007
dmi.bios.vendor: Intel Corp.
dmi.bios.version: DPP3510J.86A.0293.2007.1002.1519
dmi.board.asset.tag: Base Board Asset Tag
dmi.board.name: DG33TL
dmi.board.vendor: Intel Corporation
dmi.board.version: AAD89517-803
dmi.chassis.type: 2
dmi.modalias: dmi:bvnIntelCorp.:bvrDPP3510J.86A.0293.2007.1002.1519:bd10/02/2007:svn:pn:pvr:rvnIntelCorporation:rnDG33TL:rvrAAD89517-803:cvn:ct2:cvr:

[Marius Gedminas (mgedmin)]

Here's the full journal log of the shutdown process.

[Dan Streetman (ddstreet)]

please reopen if this is still an issue