timeout on restart or shutdown with LUKS root

Bug #1554795 reported by Mason Loring Bliss on 2016-03-08
56
This bug affects 10 people
Affects Status Importance Assigned to Milestone
initramfs-tools (Ubuntu)
High
Unassigned
systemd (Ubuntu)
High
Unassigned

Bug Description

Using the server install ISO, it's possible to specify root on LUKS and variations thereof - for instance, root on LUKS on MD-RAID, root on LVM on LUKS on MD-RAID, and so forth. The installer does the right thing and initramfs-tools does everything necessary to support booting this sort of thing.

However, systemd gives a 90-second timeout on restart or shutdown, presumably because it cannot dispose of the things beneath root.

It's wholly unclear to me where the 90-second timeout is specified, should I wish to shorten it to reboot without the futile delay, but more to the point, there seems to be infrastructure for handling this kind of situation that doesn't exist in Ubuntu at present.

I was pointed at this:

    https://www.freedesktop.org/wiki/Software/systemd/InitrdInterface/

However, Ubuntu seems not to have anything in its initramfs-tools to facilitate "shutdown-initrd" functionality.

I haven't tested this, but I suspect this problem will exist for folks running root on MD-RAID without the LUKS as well. Either way, a relatively common vanilla install will force 90-second timeouts on users, which is unfortunate.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: systemd 229-2ubuntu1 [modified: usr/share/dbus-1/system-services/org.freedesktop.systemd1.service]
ProcVersionSignature: Ubuntu 4.4.0-11.26-generic 4.4.4
Uname: Linux 4.4.0-11-generic x86_64
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl nvidia_uvm nvidia_modeset nvidia
ApportVersion: 2.20-0ubuntu3
Architecture: amd64
Date: Tue Mar 8 18:06:45 2016
InstallationDate: Installed on 2016-02-24 (13 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Alpha amd64 (20160219)
Lsusb:
 Bus 002 Device 002: ID 1058:0820 Western Digital Technologies, Inc. My Passport Ultra (WDBMWV, WDBZFP)
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 003: ID 046d:c408 Logitech, Inc. Marble Mouse (4-button)
 Bus 001 Device 002: ID 046d:c318 Logitech, Inc. Illuminated Keyboard
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: Gigabyte Technology Co., Ltd. To be filled by O.E.M.
ProcEnviron:
 TERM=xterm-color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-11-generic.efi.signed root=/dev/mapper/hostname-root ro net.ifnames=0 biosdevname=0
SourcePackage: systemd
SystemdDelta:
 [EXTENDED] /lib/systemd/system/systemd-timesyncd.service → /lib/systemd/system/systemd-timesyncd.service.d/disable-with-time-daemon.conf
 [EXTENDED] /lib/systemd/system/rc-local.service → /lib/systemd/system/rc-local.service.d/debian.conf

 2 overridden configuration files found.
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/04/2015
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: F1
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: X150M-PRO ECC-CF
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: To Be Filled By O.E.M.
dmi.chassis.version: To Be Filled By O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrF1:bd12/04/2015:svnGigabyteTechnologyCo.,Ltd.:pnTobefilledbyO.E.M.:pvrTobefilledbyO.E.M.:rvnGigabyteTechnologyCo.,Ltd.:rnX150M-PROECC-CF:rvrx.x:cvnToBeFilledByO.E.M.:ct3:cvrToBeFilledByO.E.M.:
dmi.product.name: To be filled by O.E.M.
dmi.product.version: To be filled by O.E.M.
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

Mason Loring Bliss (y-mason) wrote :
Mason Loring Bliss (y-mason) wrote :

I just managed to time it right such that I caught the error on my screen, and in doing to noticed a glaring typo that's likely indicative of the overall code quality of the related software.

I'd be grateful for debugging tips. I hope Canonical's not going to ship software that punishes LUKS users with these 90-second delays each reboot or shutdown. I'd settle for having a way to shorten the timeout, but systemd seems hopelessly opaque and I haven't found where this is set as yet.

Anyway, thanks in advance for fixing this annoyance.

Mason Loring Bliss (y-mason) wrote :

Assuming this isn't fixed any time soon, I'm running with this:

mason@ogre /home/mason$ cat /etc/systemd/system.conf.d/expletive.conf
# required singleton - high ceremony
[Manager]
#DefaultTimeoutStartSec=15s
DefaultTimeoutStopSec=15s

Note that while "Manager" is evidently the only possible section, declaring it is required, even for what would
otherwise be presumed to be config snippets in a system.conf.d/ directory.

To clarify previous notes:

1. I don't blame Canonical for systemd.
2. md1_crypt is swap and EXT4 / → LVM → LUKS → MD-RAID1
3. luksroot1 and luksroot2 are /home and /usr/src → ZFS mirror on two LUKS

Mason Loring Bliss (y-mason) wrote :

I just confirmed that I see this on a laptop as well. Conveniently, the laptop install is not using MD-RAID or ZFS - it's using LVM on LUKS on a single disk, so it's a simpler case.

On Sun, Mar 20, 2016 at 08:09:56AM -0000, Mason Loring Bliss wrote:
> I just managed to time it right such that I caught the error on my
> screen, and in doing to noticed a glaring typo that's likely indicative
> of the overall code quality of the related software.

The typo looks like an easy fix:

https://github.com/systemd/systemd/blob/da9a4daa08f84db68f7620d2e926690542c31689/src/core/job.c#L648

Thanks

Mason Loring Bliss (y-mason) wrote :

In addition to the typo, there may be an issue where the new setting I poked in is used, but the hardcoded default is still displayed on-screen. I've not been able to capture this on camera as yet, and I have yet to set things up such that all the generated messages are stored for later perusal, if that's in fact possible.

The lack of a shutdown-initrd seems to be the more critical issue, in any event. The typos and possible erroneously displayed data on-screen are cosmetic.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in initramfs-tools (Ubuntu):
status: New → Confirmed
Changed in systemd (Ubuntu):
status: New → Confirmed
Mason Loring Bliss (y-mason) wrote :

I'm going to open a seperate ticket for the typo/cosmetic issues noted, so that this ticket can focus on the core. I'll note the ticket number here once I've created it, which I'll do on the other side of my commute.

Changed in systemd (Ubuntu):
importance: Undecided → High
Changed in initramfs-tools (Ubuntu):
importance: Undecided → High
Ian (vallamost) wrote :

Are there any workarounds for this for already installed Ubuntu installs?

I have a laptop user that is having 3+ minute shutdowns for this LUKS timeout issue.

Ian (vallamost) wrote :

Reboots now hang indefinitely with the laptop user. Any update on this?

amir (amirziler) on 2016-11-23
Changed in systemd (Ubuntu):
status: Confirmed → Fix Released
Changed in initramfs-tools (Ubuntu):
status: Confirmed → Fix Committed
Mason Loring Bliss (y-mason) wrote :

I'm curious about these two status changes:

Changed in systemd (Ubuntu):
status: Confirmed → Fix Released
Changed in initramfs-tools (Ubuntu):
status: Confirmed → Fix Committed

Would it be possible to have pointers to the commit(s) that fixed this?

Thanks!

Seth Arnold (seth-arnold) wrote :

I'm resetting these to Confirmed since it looks to me like Amir may have accidentally set them to incorrect statuses.

Thanks

Changed in initramfs-tools (Ubuntu):
status: Fix Committed → Confirmed
Changed in systemd (Ubuntu):
status: Fix Released → Confirmed
Joshua Kugler (jkugler) wrote :

I have this problem on a system, and it *never* finishes stopping. It prints out a bunch of "timeout" messages about stopping various disks, then hangs forever stopping the crypt disk. I'll try to get an image of it.

Ivan Kozik (ludios) wrote :

In my case the LUKS stuff was unrelated and I was experiencing shutdown hangs because I had daemonizing processes started by /etc/rc.local: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1656237

I managed to discover that by pre-killing a bunch of processes before shutting down, which eventually narrowed down the cause.

Jan Vollendorf (u-jan) wrote :

I had the same issue, too...
Lots of messages saying "Timed out stoppping ...".
The system never rebooted.

Here is what fixed the problem for me:

Firstly, I activated systemd's debug shell using 'systemctl enable debug-shell.service'.
Then I initiated a reboot and when the problem occured, I switched to debug shell (alt+F9 [/F10 ?] ).
Using 'systemctl list-jobs', I figured out what jobs were running while the system hung.

One of the jobs was an "unattended-upgrdes" job.
This job held one ore more files on my root filesystem open.
When I killed that process, the system finished its shutdown and rebooted.

After having removed the package by 'apt-get --purge remove unattended-upgrades' the problem didn't arise again.

I am using cron-apt on my servers - therefore I do not necessarily need unattended-upgrades.

Hopefully this helps as workaround and to find the real cause of the problem.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers