systemd does not unlock dm-crypt password

[Gunter Ohrner]

Since upgrading from version 14.10 to Kubuntu 15.04, my system does not boot normally any more.

Maybe it's only a stupid oversight on my part, but not being a systemd pro, I'm currently at a loss of how to debug this any further.

Hints and pointers to appropriate documentation are welcome!

Kubuntu 15.04 quickly boots to a specific point where it seems to try to enable a dm-crypt partition which also carries /home, beside other data, and simply stalls.

I'm not asked to enter my crytpdisk password and I also cannot enter it blindly without being asked.

You can see the point where it stalls in the attached "screenshot" (photo of my screen).

I can then press Ctrl+Alt+Del to cleanly restart the system but I could not find a way to get beyond this point with systemd.

Using the alternative boot option with upstart works flawlessly. (Although booting with this boot option is really slow and it takes a while to reach the graphical login screen - it basically looks as if the system is waiting for some name resolving timeouts during boot, but that's a different issue, if've not debugged it further and maybe it's just gone once systemd works to boot my machine.)

Further info about my setup:

$ cat /etc/crypttab
crypt /dev/mapper/main-cryptstore none luks,noearly,discard

$ mount | egrep crypt
/dev/mapper/crypt on /mnt/crypt type ext4 (rw,relatime,data=ordered)
/dev/mapper/crypt on /home type ext4 (rw,relatime,data=ordered)
/dev/mapper/crypt on /var/lib/mysql type ext4 (rw,relatime,data=ordered)

Here the first entry is the actual mount point while the other mount points are bind mounts:

$ cat /etc/fstab | egrep crypt
/dev/mapper/crypt /mnt/crypt ext4 defaults 0 2
/mnt/crypt/home /home none bind 0 0
/mnt/crypt/var_lib_mysql /var/lib/mysql none bind 0 0

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: systemd 219-7ubuntu4
ProcVersionSignature: Ubuntu 3.19.0-16.16-generic 3.19.3
Uname: Linux 3.19.0-16-generic x86_64
ApportVersion: 2.17.2-0ubuntu1
Architecture: amd64
CurrentDesktop: KDE
Date: Mon May 11 20:28:18 2015
MachineType: Sony Corporation SVS13A3W9ES
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.19.0-16-generic root=/dev/mapper/main-root ro quiet init=/sbin/upstart
SourcePackage: systemd
SystemdDelta:
 [EXTENDED] /lib/systemd/system/systemd-timesyncd.service -> /lib/systemd/system/systemd-timesyncd.service.d/disable-with-time-daemon.conf

 1 overridden configuration files found.
UpgradeStatus: Upgraded to vivid on 2015-04-28 (13 days ago)
dmi.bios.date: 03/13/2013
dmi.bios.vendor: Insyde Corp.
dmi.bios.version: R1021C8
dmi.board.asset.tag: N/A
dmi.board.name: VAIO
dmi.board.vendor: Sony Corporation
dmi.board.version: N/A
dmi.chassis.asset.tag: N/A
dmi.chassis.type: 10
dmi.chassis.vendor: Sony Corporation
dmi.chassis.version: N/A
dmi.modalias: dmi:bvnInsydeCorp.:bvrR1021C8:bd03/13/2013:svnSonyCorporation:pnSVS13A3W9ES:pvrC60BRBTW:rvnSonyCorporation:rnVAIO:rvrN/A:cvnSonyCorporation:ct10:cvrN/A:
dmi.product.name: SVS13A3W9ES
dmi.product.version: C60BRBTW
dmi.sys.vendor: Sony Corporation

[Martin Pitt (pitti)]

> crypt /dev/mapper/main-cryptstore none luks,noearly,discard

This looks really curious -- the source device is "none", so how is that finding out where the underlying encrypted device is? This should usually be something like UUID=12345, or /dev/sdb3 or whatever.

Can you please boot with the systemd once, then again with upstart, and attach /var/log/syslog here? Thanks!

[Gunter Ohrner (gohrner)]

"none" is the key file:

crypttab(5):

"""The third field, key file, describes the file to use as a key for decrypting the data of the source device. Note that the entire key file will be used as the passphrase; the passphrase must not be followed by a newline character."""

However I'll try booting with the different init systems and attach the syslog results soon, hopefully this evening.

[Gunter Ohrner (gohrner)]

When I boot using systemd, nothing is logged at all - maybe booting stalls even before rsyslogd is started?

If I boot using upstart, the syslog is as follows - note the massive delay before activation of "org.freedesktop.UDisks2" which causes the boot process to take quite long - Kubuntu 14.10 had a *much* faster boot process without such a delay.

[Anders Hall (a.hall)]

I have a very similar issue. Upstart works but systemd does not. Lucs encrypted logical volume on a work computer. It has worked flawless with boot for many versions of Ubuntu (even before upstart I think). I can produce the syslog with two boots tomorrow. Also have taken a picture of the actual error message during boot with systemd, which is attached. Upstart presents the password prompt, systemd does not.

[Anders Hall (a.hall)]

Some setup info as previous post:

sda5_crypt UUID=acdea251-1e21-4612-afce-4101e364d6c6 none luks
cryptswap1 /dev/dm-1 /dev/urandom swap,cipher=aes-cbc-essiv:sha256

mount -l with upstart

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=6130708k,nr_inodes=1532677,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1229448k,mode=755)
/dev/mapper/lvm1-sysroot on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
none on /sys/fs/cgroup type tmpfs (rw,relatime,size=4k,mode=755)
none on /sys/fs/fuse/connections type fusectl (rw,relatime)
none on /sys/kernel/debug type debugfs (rw,relatime)
none on /sys/kernel/security type securityfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,noatime)
none on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
none on /run/shm type tmpfs (rw,nosuid,nodev,relatime)
none on /run/user type tmpfs (rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755)
none on /sys/fs/pstore type pstore (rw,relatime)
cgmfs on /run/cgmanager/fs type tmpfs (rw,relatime,size=100k,mode=755)
/dev/mapper/lvm1-home on /home type ext4 (rw,relatime,data=ordered)
/dev/sda1 on /boot type ext4 (rw,relatime,data=ordered)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
rpc_pipefs on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/user/104 type tmpfs (rw,nosuid,nodev,relatime,size=1229448k,mode=700,uid=104,gid=111)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=1229448k,mode=700,uid=1000,gid=1000)
/home/user1/.Private on /home/user1 type ecryptfs (rw,nosuid,nodev,relatime,ecryptfs_fnek_sig=9609cabaf4e75b54,ecryptfs_sig=825adaf5c17ffed8,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_unlink_sigs)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)

[Anders Hall (a.hall)]

Requested syslog attached.

[Anders Hall (a.hall)]

Requested syslog attached (for real).

[Martin Pitt (pitti)]

Gunter,

sorry about the "none" comment #2, that was obviously bogus, I mis-looked.

Can you please follow /usr/share/doc/systemd/README.Debian to boot with a debug shell, then switch to it once it stalls, run "journalctl -ab > /root/journal.txt", reboot, and attach /root/journal.txt here?

[Martin Pitt (pitti)]

@Anders: There is no syslog for a systemd boot. Can you please do the same exercise as in comment #9, attach the journal, and additioally /etc/crypttab and /etc/fstab? Thanks!

[Gunter Ohrner (gohrner)]

I fetched the debug log using the debug console, however it's contents do not look suspicious to me...

systemd seems to try to start the cryptsetup service, the only question probably is where the password prompt does end up...

[Martin Pitt (pitti)]

Indeed, the log doesn't even show an attempt to unlock cryptstore. In that debug shell, can you please do

  tar cf /root/generator.tar /run/systemd/generator*

and attach /root/generator.tar here?

[Gunter Ohrner (gohrner)]

HTH

[Anders Hall (a.hall)]

@Martin,

Sorry for late reply. I spilled juice in my computer and it fried. Never gets old :-) I have a new computer so I cant do proper debugging anymore. I will check a friends computer since I think he has the exact same problem. On Wednesday next week that is (is in transit until then).

..will test the method in /usr/share/doc/systemd/README.Debian.

Btw, could the problem be that I had two passwords. One for the full disk and one for the home folder? So the setup tried to use the same password for the disk as the home folder? I had a keyboard key that was included in the password once that broke and had to change the disk/boot password (was a pain to fix).

[Gunter Ohrner (gohrner)]

BTW, is there anything else needed from my side or anything else I should or can do to provide any required information?

I'm still having this issue and I fear my system might be rendered unusable if the next Kubuntu update possibly removes upstart support completely and with it the fallback boot option that's currently saving me...

[Anders Hall (a.hall)]

Don't think i can verify this on the other computer as stated. It seems to be a different problem.

[Launchpad Janitor (janitor)]

[Expired for systemd (Ubuntu) because there has been no activity for 60 days.]

[Gunter Ohrner (gohrner)]

Unfortunately, this bug is still current - I submitted all requested information, there has been no activity in response, but the problem still persists.

[Gunter Ohrner (gohrner)]

I just noticed that I can enter the password and unlock the disk if I boot with splash / plymouth enabled.

I normally do not use the "splash" boot option to boot up in text mode, and the cryptsetup unit file or something does not seem to be able to cope with that and does not query for the password.

Adding "splash" to the boot options made a password prompt appear during systemd boot, but of course also enabled a graphical boot up process...

[Bob Ziuchkovski (bob-ziuchkovski)]

I can confirm the exact same behavior: if /etc/crypttab has entries for non-root devices, then the only way to unlock with a password is to force plymouth to run with the "splash" boot option.

I've been trying to test luks-encrypted data volumes (non-root) on ubuntu-server and have hit this problem for 15.04, 15.10, and 16.04 daily...basically ever since the switch to systemd. Worse, adding "splash" doesn't actually resolve the problem on ubuntu-server...perhaps due to a broken/incomplete plymouth install?

I ended up testing ubuntu-desktop 15.10 on a whim with "splash" and that was the first time I was able to enter the password to unlock the devices.

It seems like this is a problem with the systemd-ask-password-console service. I've tried to test the theory by creating an extremely simple "oneshot" service script that requires systemd-ask-password-console and runs "/bin/systemd-ask-password --no-tty 'Test'". I've tried all manner of systemd service options and I can't for the life of me get systemd-ask-password-console to prompt for a password.

[Bob Ziuchkovski (bob-ziuchkovski)]

I tried another test in an attempt to get /bin/systemd-tty-ask-password-agent to prompt for a password. I ensured no other systemd password agents were running (ps auxww | grep ask), then launched 'sudo /bin/systemd-tty-ask-password-agent --watch --console' in one terminal and launched 'sudo /bin/systemd-ask-password --no-tty --timeout=30 Test' in another terminal. The terminal with systemd-tty-ask-password-agent running never prompts for anything and after 30 seconds, the systemd-ask-password process times-out. It seems like there's some sort of bug that prevents systemd-tty-ask-password-agent from doing its job.

In an attempt to be thorough, I repeated the test after running 'sudo rm /lib/systemd/system/systemd-ask-password-*' and rebooting. I did this to ensure none of the agents were spawning via systemd path events and interfering with the direct test.

Finally, I also repeated the test one more time via 'sudo strace -f /bin/systemd-tty-ask-password-agent --watch --console' and captured the strace output. That's attached as 'tty-agent.out'. It hangs at that final line indefinitely: 'read(6,'

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[James Johnston (mail-codenest)]

Unfortunately this is pretty easy to reproduce. I installed Ubuntu Server 15.10, set up an additional partition and formatted it with LUKS, and added to crypttab. That is all it takes to make the boot process hang while it asks for a password.

It appears that if the partition is unlocked in the initramfs, then it works. The initramfs cryptsetup is ok. But if initramfs doesn't unlock it - i.e. it's not the boot drive - then it's up to systemd to do so, which doesn't work.

Oddly enough, if I reboot then I can momentarily see the password prompt just before the system reboots. Of course by then it's too late. :(