Remote unlocking not possible if plymouth is active (Bug or Feature?)

[hal2100]

Binary package hint: cryptsetup

If plymouth is active, it is no longer possible in an easy way to remotely unlock the disc(s).
Which means that with a standard Ubuntu setup the README.remote is wrong or incomplete.

Reason: Plymouth is "stealing" the password prompt because the cryptroot script checks if plymouth is active:

if [ -z "$cryptkeyscript" ]; then
   cryptkey="Unlocking the disk $cryptsource ($crypttarget)\nEnter passphrase: "
   if [ -x /bin/plymouth ] && plymouth --ping; then
    cryptkeyscript="plymouth ask-for-password --prompt"
    cryptkey=$(echo -e "$cryptkey")
   else
    cryptkeyscript="/lib/cryptsetup/askpass"
   fi
  fi

but only askpass has a feature which is checking for a file with a password in it.

Because I am not so good in writing startup fixes, I am proposing this as a bug.
Possible solutions:
1. Include a new script, which doesn't use plymouth at all.
2. Use command line switches to use askpass instead of plymouth.
3. Patch plymouth, e.g. to include a "pass-as-password" option, which is passing the password along to a running plymouth(d?).

My knowledge about the inner workings of the startup process is limited, I would prefer solution no. 3.
Any suggestions?

[Steve Langasek (vorlon)]

You're correct that the documentation is now out of date. However, plymouth is only the /default/ interface used by cryptsetup; you can probably get the same effect as before by using a 'keyscript=' option in /etc/crypttab. Though plymouth is still going to control the console, so you'll have to do some further tuning if you want to be able to unlock using either local console or a remote unlock command.

[hal2100 (hal-from-2001)]

Sorry, but I think we have a misunderstanding here.
We should aim for a solution which enables both ways. With the introduction of plymouth also a bug was introduced, because the old behavior was canceled without notice. The solution with the keyscript is not feasible, as far as I understand, because this would disable plymouth completely, which is not the goal.

From my point of view this could be easily fixed if someone who knows the startup procedure very well could just post the necessary steps to disable plymouth just for this "one" startup when remote unlocking is desired. My problem is I do not know the side effects if I just do a:
$killall plymouthd
$killall plymouth
$/bin/sh /path/to/cryptroot
...

Perhaps there is a way to stop plymouth without the killall, but I simply do not know the cleanest ways.
I think this bug can be fixed without a single line of code, just with a better documentation. I would write one, but I do not know the consequences...

[Steve Langasek (vorlon)]

On Mon, Jun 21, 2010 at 05:34:54PM -0000, hal2100 wrote:
> We should aim for a solution which enables both ways.

I agree that's the optimum, but that's not possible today, so I'm suggesting
a workaround that would work for anyone who exclusively unlocks their system
remotely.

> With the introduction of plymouth also a bug was introduced

I'm not disputing that, but this is not nearly so easy to fix as you
suppose, and we're certainly not going to revert the plymouth support which
fixed many other, much higher-impact bugs.

> I think this bug can be fixed without a single line of code

Sorry, but you're mistaken. The cryptsetup "askpass" helper doesn't support
plymouth, instead cryptsetup uses the plymouth client directly, and that's
currently mutually exclusive with remote unlocks without writing some
significant code.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[hal2100 (hal-from-2001)]

>I'm not disputing that, but this is not nearly so easy to fix as you
>suppose, and we're certainly not going to revert the plymouth support which
>fixed many other, much higher-impact bugs.

This is the point I don't understand. Why is a "killall plymouth*" bad?
This would allow for an easy solution, kill plymouth after remote log in and use the askpass "fallback" as it is currently implemented. Leave everything as it is if I am sitting in front of the machine using plymouth to unlock.
I do not want to disable plymouth completely, just for the remote sessions.

An more complicated solution would be to introduce a "--accept-pass SECRET" option to plymouth to inject the password from a second console (the remote login) into the currently waiting plymouth(d?) process which asks for the password.
Alternatively plymouth could check for the same file askpass is checking, but I would prefer the first solution.

But this would definitely mean some more lines of code, I agree.

[DW (derwolf111)]

if I comment the lines in /usr/share/initramfs-tools/scripts/local-top/cryptroot like this:

if [ -z "$cryptkeyscript" ]; then
   cryptkey="Unlocking the disk $cryptsource ($crypttarget)\nEnter passphrase: "
   #if [ -x /bin/plymouth ] && plymouth --ping; then
   # cryptkeyscript="plymouth ask-for-password --prompt"
   # cryptkey=$(echo -e "$cryptkey")
   #else
    cryptkeyscript="/lib/cryptsetup/askpass"
   #fi
  fi

then remote unlocking works again

[ingo (ingo-steiner)]

Check this bug report how to de-install plymouth:
https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/556372

[daniel_L (daniel-das-grauen)]

@DW / #5:
I cannot confirm this, I still can't unlock my disk via ssh when commenting out the mentioned lines. Can you pleasy explain how exactly you unlock the disk?

[DW (derwolf111)]

daniel:
checkout readme.debian in /usr/share/doc/cryptsetup or http://wiki.ubuntuusers.de/Verschl%C3%BCsseltes_System_via_SSH_freischalten

[daniel_L (daniel-das-grauen)]

@DW:
That's actually the guide I already followed without success. Anyways, I found the "solution": I had to update the initramfs image in order to get this working; that's a step the guide is missing (maybe because it's more or less obvious). Nevertheless, the guide is written as if it provides all neccessary steps so I just didn't get the idea - until now.

Thanks anyway.

[164747 (jacquet-david)]

@daniel_L
From yout comment, I understand that you have uncomment lines suggested by DW in #5, and fot remote ssh unlocking working after running "update-initramfs -u". Thats nice! However can you still unlock the machine by classical mean, keyboard on site? Or do you have unlock it remote?

[Ryan Lackey (ryan.lackey)]

I've also applied DW's fix (thanks!) and think maybe this should be the default for now until there is a better fix in place.

[daniel_L (daniel-das-grauen)]

@164747
Unfortunately, with this fix it's not possible to unlock the system with a keyboard on site.

[hal2100 (hal-from-2001)]

@daniel_L
If we find a way to unlock the computer on site as well, this "bug" could be closed at last....

[Timmeey (zoggerfish)]

I found a solution in ubuntuusers Forum

http://forum.ubuntuusers.de/topic/script-verschluesseltes-system-via-ssh-freisch/#post-2797575

this guy wrote a little script. It worked for me (10.10)now I'm able to unlock locally AND over ssh

just copy this script into /usr/share/initramfs-tools/hooks/
make it executable
(chmod +x) and
update-initramfs -u

reboot.... now ure up and running

[Richard Hansen (rhansen)]

@Steve Langasek: What about adding a read-from-fifo capability to 'plymouth ask-for-password' like /lib/cryptsetup/askpass has?

[Stephen Warren (srwarren)]

A solution to making this work remotely is:

1) SSH in
2) Manually run cryptsetup; basically run ps, find the cryptsetup command-line, remote the key-file option, run it, type the password into cryptsetup
3) kill cryptsept process, kill plymouth process

It just worked for me:-)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Marco Paganini (paganini-launchpad)]

@a7x that would be my preferred solution as well. I looked at the source and it's unfortunately a client/server based system so it's not a 10-minute fix for someone not used to the source code of plymouth (like me.)

[AllenS (thehousecat)]

The following solution is simple and works for me. My configuration is the default encrypted LVM setup created by the Precise alternative installer.

ssh to the remote machine and run the following commands:

kill $(pidof plymouthd)
# Wait a few seconds for '/scripts/local-top/cryptroot' to timeout and retry using 'askpass' instead of plymouth
echo -ne password > /lib/cryptsetup/passfifo

[Davidos (davidos-di)]

Have a look at probably simpler solution provided here:
https://bugs.launchpad.net/cryptsetup/+bug/556372

The solution is to install plymouth-dummy package and remove the standard one. Then everything works fine as earlier.
This is fully based on https://bugs.launchpad.net/cryptsetup/+bug/556372/comments/78
###
Oliver Joos (oliver-joos) wrote on 2012-03-25: #78

Ok then, here is the ready-to-install plymouth-dummy_1.0_all.deb! It's an empty package but allows to remove plymouth while keeping the packages that have wrong dependencies to plymouth.

To workaround this (WontFix??) bug and remove plymouth just execute 2 commands in a Terminal:
  sudo dpkg -i plymouth-dummy_1.0_all.deb
  sudo apt-get remove plymouth

Advanced users can build plymouth-dummy_1.0_all.deb themselves: see comment 16
Many thanks to Christian Kujau!
###

I've tested it on 12.04 LTS

[cunnilinux (cunnilinux)]

with installed plymouth-dummy and removed original plymouth, i could not unlock my partitions vial local keyboard.
seems like this works for remote unlocking via ssh, but still does not fixes all issues.
tested on raring as for 2013-02-17.

[Giancarlo Razzolini (grazzolini)]

@cunnilinux

I've tried on quantal (12.10) server and with the plymouth-dummy package I could unlock using BOTH the passfifo method and using a keyboard on site, attached to the machine. Did you followed the correct instructions?:

1) dpkg -i plymouth-dummy_1.0_all.deb
2) apt-get remove plymouth
3) update-initramfs -u

reboot then done

[Richard Hansen (rhansen)]

Attached is a script I wrote to make it easier to unlock systems from remote. It works on precise and quantal even with the original plymouth. I haven't tried the script on raring yet.

[Giancarlo Razzolini (grazzolini)]

It appears that plymouth-dummy doesn't work on raring (13.04). I've installed it and when I try to remove the plymouth package, it tries to remove a lot of dependencies and if I remove it, the apt gets broken. If I reinstall it though, it works again, but remote unlocking using the passfifo method doesn't work, only using the script from #23 or one of the other methods (that kill the cryptroot process and wait many seconds). There is any deadline for this bug being permanently solved? Remote unlocking a luks or dm-crypt root container should be easy to use, perhaps even an option on an ubuntu server installation, since there are a lot of VPS hosting providers that offer this kind of installation.

[Felix Eckhofer (eckhofer)]

On Raring, something like this works:

  cat /conf/conf.d/cryptroot
  /sbin/cryptsetup luksOpen <source> <target>
  kill $(pidof plymouth)

Where <source> and <target> have to be replaced by the respective parameters from the output of the first command.

What is the proposed way to fix this issue? It should be rather easy to create a script which uses the functions from local-top/cryptroot to parse options from /proc/cmdline and /conf/conf.d/cryptroot, then calls /sbin/cryptsetup as above and on success kills the plymouth password prompt. Calling this script from a wrapper which waits for input on /lib/cryptsetup/passfifo seems similary simple. Would a patch be accepted if it did that?

[Giancarlo Razzolini (grazzolini)]

How is the status on this? Now on ubuntu 14.04 the previous hacks to remote unlock the disk do not work anymore. This bug already is more than 4 years old.

[Richard Hansen (rhansen)]

> Now on ubuntu 14.04 the previous hacks to remote unlock the disk do
> not work anymore.

The script I added in comment #23 above still works for me on trusty
(14.04).

(I would prefer a better solution to this bug, however.)

Direct link to the script:
https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/595648/+attachment/3662930/+files/unlock-cryptroot

[Pierre BLONDEAU (pierre-blondeau)]

I have the same problem.

I propose this patch, which preserve the three features:
 - Used askpass and its fifo in all cases
 - Used plymouth if available
 - Used another program if the cryptkeyscript variable is specified by user.

Regards

[Pierre BLONDEAU (pierre-blondeau)]

Oops, Replace Used by Use. Regards

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "cryptroot.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Nathaniel Homier (mechamechanism)]

I'm a server admin, I manage a mail server and DNS server. The server is encrypted for valid reasons. I need to have remote boot via SSH. This bug needs to be made a higher priority! I have daily local access to the sever, but what happens when I'm on vacation and I need to reboot the encrypted server. Right now I have to trust a third party to boot the server and of course they have to know the password, but what happens when their out of town and I'm out of town, hmmm?

The workarounds in the comments are not the proper Ubuntu/Debian way and risk breaking with every update that comes down the line. I simply don't have the time to test every update to make sure the scripts posted in the comments don't break.

Would giving money to the Ubuntu programmers solve this or does Ubuntu not have a $$$ bounty program. I couldn't afford much, maybe $50.

[AllenS (thehousecat)]

Hi Nathaniel - have you tried my suggestion in https://bugs.launchpad.net/ubuntu/+source/cryptsetup/+bug/595648/comments/19 ? i.e. just kill pymouth and then echo the password to passfifo as normal

This still works for me in 14.04. I haven't needed to do any of the other suggested patches or installing non standard packages. Hopefully it works for you too.

[Nathaniel Homier (mechamechanism)]

Hi AllenS. I want to try it. But unfortunately when I went to install dropbear the version was to old to support ecdsa ssh key. I will have to wait for a new version to be uploaded or find a ppa. If I can get the latest dropbear than I would like to try your trick.

Thanks AllenS.

[Pierre BLONDEAU (pierre-blondeau)]

Hi,

If it's a server, you can "solve" the bug by remove the Plymouth package.

Regards

[Richard Hansen (rhansen)]

The /scripts/local-top/cryptroot script in the initramfs only unlocks the root and resume filesystems. All other encrypted filesystems are unlocked by /etc/init/cryptdisks-udev.conf and /etc/init/cryptdisks.conf after the real / has been mounted.

This design is problematic for remote unlocking: If one of those non-root non-resume encrypted filesystems is essential to booting the system (it has the 'bootwait' option in /etc/fstab), then the initramfs will go away before the filesystem is unlocked (because the root filesystem is mounted), but sshd won't start because it's waiting for another essential filesystem to be unlocked. Thus, there's no way to remotely access the system and unlock the remaining filesystem(s).

Before this bug can be considered fixed, /usr/share/initramfs-tools/hooks/cryptroot will have to be edited to include all 'bootwait' filesystems in the /conf/conf.d/cryptroot config file it produces in the initramfs.

As a temporary workaround, users can add non-root non-resume 'bootwait' filesystems to /etc/initramfs-tools/conf.d/resume as if they were resume devices, though they must be listed BEFORE the real resume device. (/usr/share/initramfs-tools/hooks/cryptroot can handle multiple RESUME=* lines, and the initramfs init script ignores all RESUME=* lines but the last.)

[Richard Hansen (rhansen)]

I'm attaching a newer version of my unlock-cryptroot script. It now takes less time to unlock the remote filesystem(s), it uses a more robust method to unlock, and if you have multiple encrypted filesystems with the same password it will only ask for the password once (it tries the previously-entered password before prompting again).

[Gordon (kmputerguy)]

a7x's latest script works for me on 14.04, but such an elaborate workaround really shouldn't be necessary.

[Edward (edward-doolittle)]

Would it be difficult to patch plymouth so that it would take input from /lib/cryptsetup/passfifo as well as taking input from the console?

[Richard Hansen (rhansen)]

FYI, you can get the latest version of my remote unlock script from my Git repository:

    https://github.com/rhansen/unlock-cryptroot

[Maddes (maddes.b)]

I could verify that the Debian fix works for Debian "testing" with and without plymouth splash screen.

Unfortunately the Debian fix is not yet implemented in the Ubuntu 16.04 (Xenial Xerus) Daily Build at http://cdimage.ubuntu.com/daily-live/current/ as of today.

[Jeremias Kangas (jeremias-kangas)]

Any news about this bug on Xenial?

[Maddes (maddes.b)]

No changes in Xenial, but 1.7.2 is in the stage for Yakkety.
See http://packages.ubuntu.com/cryptsetup

Latest versions of Debian, see https://packages.debian.org/cryptsetup

[aljos (aljos)]

Can someone please verify that they have successfully used a script, such as the one from rhansen above, to remote unlock a LUKS encrypted Xenial server?

I currently use this script on my Trusty server:
https://hacksr.blogspot.de/2012/05/ssh-unlock-with-fully-encrypted-ubuntu.html

Until I hear confirmation that there is a working script to get around this bug in Xenial, I will need to postpone my Trusty upgrades.

Thanks

[Maddes (maddes.b)]

Still not working correctly with Ubuntu 16.10 as the BusyBox package of Ubuntu doesn't support the parameters -eo of command `ps`.

[Maddes (maddes.b)]

Additionally the BusyBox package of Ubuntu doesn't support the parameter -z of command `grep` in the initramfs.