cryptsetup starts too early

[Xavier Gnata]

I hope that the tilte is correct.
I have encrypted a sdcard using cryptsetup.
During the boot sequence, it does not ask for the password. I fail to find an error message in the logs.

I think my config is ok because /etc/init.d/cryptdisks start does work:
* Starting remaining crypto disks...
* data (starting)
Enter passphrase to unlock the disk /dev/mmcblk0p1 (data):
(I enter the password)
key slot 0 unlocked.
Command successful.
data (started)... [ OK ]

and then, "mount -a" mounts the encrypted partition.

It looks like cryptsetup starts *before* /dev/mmcblk0p1 is created.
That would explain why cryptsetup fails to see my encrypted disk at boot time.

My config files:
crypttab:
# <target name> <source device> <key file> <options>
data /dev/mmcblk0p1 none luks

fstab:
/dev/mapper/data /home/gnata/data ext4 defaults 0 1

TEST CASE:
1. install the cryptsetup package from karmic
2. Configure a device that will be slow to be initialized by the kernel, such as a USB drive or an SD card, to be used with cryptsetup by running commands such as this:
  cryptsetup luksFormat /dev/mmcblk0p1
  cryptsetup luksOpen /dev/mmcblk0p1 slowdisk
  mkfs.ext4 /dev/mapper/slowdisk
3. Set the device up to be decrypted at boot time by adding a line such as this to /etc/crypttab:
    slowdisk /dev/mmcblk0p1 none luks
4. Set the device up to be mounted at boot time by adding a line such as this to /etc/fstab:
    /dev/mapper/slowdisk /mnt ext4 defaults,bootwait 0 0
5. reboot a few times (in single user mode, to prevent gdm from starting and killing usplash); verify that the passphrase prompt for decrypting this disk is racy, and does not always appear
6. install the cryptsetup package from karmic-proposed
7. reboot a few times (in single user mode, again); verify that the passphrase prompt now appears reliably.

REGRESSION POTENTIAL:
With this change cryptsetup will no longer be racing udev; however, cryptsetup instances *will* be racing each other. In the unlikely case that a user has CRYPTDISKS_MOUNT set to a non-empty value in /etc/default/cryptdisks, *and* has more than one volume configured in /etc/crypttab, the upstart jobs can race each other and one job can unmount the filesystems before the other job has a chance to use them. However, even in this case the cryptdisks-enable catch-all job should still take care of it, unless the second failed decrypt job then unmounts the filesystem when cryptdisks-enable needs it (a double race).

This change also increases the chance that a filesystem not marked as 'bootwait' in /etc/fstab will get in a race with gdm at boot time, leading to the passphrase prompt being masked by X, or cryptsetup fighting X for control of the console. However, that class of race is already present in karmic and documented in the release notes, so I don't think this should be a blocker for SRU.

[Federico Zenith (federico-zenith)]

I confirm the problem on a EEE 901A.
I use an encrypted SD card for my home folder, and even if /etc/crypttab is configured correctly I usually do not get a prompt at the splash screen.
However, I said "usually": sometimes I DO get the prompt as it should be, typically when starting encrypted partitions is delayed by some reason, such as a fsck of the root partition.
So if I have to guess it is really a timing issue.

[Xavier Gnata (xavier-gnata-gmail)]

> I confirm the problem on a EEE 901A.
> I use an encrypted SD card for my home folder, and even if /etc/crypttab is configured correctly I usually do not get a prompt at the splash screen.
> However, I said "usually": sometimes I DO get the prompt as it should be, typically when starting encrypted partitions is delayed by some reason, such as a fsck of the root partition.
> So if I have to guess it is really a timing issue.
>
>
The bug is still there 4days before release :(
I think a timer cannot solve this problem. A timer is never a proper fix
when we have a sync pb.
We need synchronization here. cryptsetup should not start before the
end of disks detection.
As we are switching to an async boot sequence, it may be hard...but it
is mandatory.

[Xavier Gnata (xavier-gnata-gmail)]

Still fully broken.
Unfortunately, I would have to understand how to tune the upstart
sequence to try to fix this...
Any comments?

> ** Changed in: cryptsetup (Ubuntu)
> Status: New => Confirmed
>
>

[Launchpad Janitor (janitor)]

This bug was fixed in the package cryptsetup - 2:1.1.0~rc2-1ubuntu4

---------------
cryptsetup (2:1.1.0~rc2-1ubuntu4) lucid; urgency=low

  [ Steve Langasek ]
  * debian/initramfs/cryptroot-script: if plymouth is present in the
    initramfs, use this directly, bypassing the cryptsetup askpass script;
    but keep support for these other frontends around on a transitional
    basis.
  * debian/cryptdisks.functions:
    - change interaction to use plymouth directly if present, and if not, to
      fall back to /lib/cryptsetup/askpass as before
    - wrap the call to /lib/cryptsetup/askpass with watershed, to make sure
      we only ever have one of these running at a time; otherwise multiple
      invocations could steal each other's input and/or write over each
      other's output
    - new function, crypttab_start_one_disk, to look for the named source
      device in /etc/crypttab (by device name, UUID, or label) and start it
      if configured to do so
  * debian/cryptdisks-enable.upstart: run the upstart job once for each block
    device, using the new crypttab_start_one_disk function, triggered by udev;
    this doesn't eliminate the possibility of a race with gdm when the
    decrypted volume isn't a 'bootwait' mount point (since gdm kills
    plymouth), but it does eliminate the race between udev and cryptsetup.
    LP: #454898.
  * debian/cryptdisks-enable.upstart: check that the package is installed
    and exit gracefully if it's not. LP: #435814
  * debian/cryptdisk.functions: initially create the device under a temporary
    name and rename it only at the end using 'dmsetup rename', to ensure that
    upstart/mountall doesn't see our device before it's ready to go.
    LP: #475936.

  [ Colin Watson ]
  * Add ext4 support to passdev.
 -- Steve Langasek <email address hidden> Tue, 15 Dec 2009 18:05:45 -0800

[Martin Pitt (pitti)]

Accepted cryptsetup into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Xavier Gnata (xavier-gnata-gmail)]

Something has changed but that's not that good:
It does not ask for the password during the boot.
Worst, /etc/init.d/cryptdisks restart now hangs:

* Stopping remaining crypto
disks... * data
(stopped)... [
OK ]
  * Starting remaining crypto
disks... * data
(starting)
and it look like it's look in a infinite loop....

I may have done something wrong with the "proposed" update:
udev Version: 147~-6
2.6.31-16-generic
crypsetup Version: 2:1.0.6+20090405.svn49-1ubuntu7.1

Well the other pb is that it is only a test machine. That's why I'm
going to install a Lynx on it.
Let's see if it is fine with a fresh (and allready lucid) lynx.

> Accepted cryptsetup into karmic-proposed, the package will build now and
> be available in a few hours. Please test and give feedback here. See
> https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to
> enable and use -proposed. Thank you in advance!
>
> ** Changed in: cryptsetup (Ubuntu Karmic)
> Status: In Progress => Fix Committed
>
> ** Tags added: verification-needed
>
>

[Steve Langasek (vorlon)]

Xavier,

Can you please attach your /etc/fstab and /etc/crypttab here?

[Steve Langasek (vorlon)]

Also, when this hang occurs, please check whether you have multiple 'watershed' processes running. If the boot-time cryptsetup has hung for whatever reason, 'cryptdisks restart' will also hang waiting, to avoid colliding with the one that's already running.

[Xavier Gnata (xavier-gnata-gmail)]

> Also, when this hang occurs, please check whether you have multiple
> 'watershed' processes running. If the boot-time cryptsetup has hung for
> whatever reason, 'cryptdisks restart' will also hang waiting, to avoid
> colliding with the one that's already running.
>
>

ok. I stop /etc/init.d/cryptdisks restart (looping for ever) with crtl+c
and then:
ps aux |grep watershed
root 886 0.0 0.0 1668 484 ? S 14:57 0:00
/lib/udev/watershed /lib/cryptsetup/askpass Enter passphrase to unlock
the disk /dev/mmcblk0p1 (data):
root 988 0.0 0.0 1668 464 ? S 14:57 0:00
/lib/udev/watershed /lib/cryptsetup/askpass Enter passphrase to unlock
the disk /dev/mmcblk0p1 (data):

but this message never appears during the boot process.

I kill both watershedprocesses and then the /etc/init.d/cryptdisks
restart is working fine:
  Enter passphrase to unlock the disk /dev/mmcblk0p1 (data):
key slot 0 unlocked.
Command successful.
  * data
(started)... [
OK ]

It looks like something starts during the boot but it is not a blocking
call.
It may explain why I do not see "Enter passphrase to unlock the disk
/dev/mmcblk0p1 (data):" during the boot. This question remains
unresponded and watershed is still alive waiting for an input.

fstab:
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
# / was on /dev/sda2 during installation
UUID=0fd38542-fff5-43ce-b78b-f11e0cea69a3 / ext4
errors=remount-ro 0 1
# swap was on /dev/sda1 during installation
UUID=30240ff1-54bd-412f-bab0-757b788bf87f none swap
sw 0 0

/dev/mapper/data /home/gnata/data ext4 defaults 0 1

crypttab:
# <target name> <source device> <key file> <options>
data /dev/mmcblk0p1 none luks

All these tests are performed on a new fresh karmic + dist-upgrade after
insertion of "proposed" in the sources.list.

[Steve Langasek (vorlon)]

Xavier,

If you change the options on the fstab entry to 'defaults,bootwait' (as mentioned in the Ubuntu 9.10 release notes), does this make a difference?

[Xavier Gnata (xavier-gnata-gmail)]

Well, using 'defaults,bootwait' it does work.
However, I use a set of encrypted SDcards.
If there is no card in the slot, it does not boot anymore (I only get a maintenance shell).
That's why I put:

/dev/mapper/data /home/gnata/data ext4 defaults 0 1
and not:

/dev/mapper/data /home/gnata/data ext4 defaults 0 0

in the fstab (pass=1)

Now, if I add bootwait, then it is like having pass=0.

Ok, maybe it is just a bad idea to put SDcard in the fstab? (but I think that "pass=1 in the fstab" should address this usecase).

Xaiver.

[Steve Langasek (vorlon)]

Xavier,

There are two configurations for such devices that are supported by karmic:

- configure the filesystem as 'bootwait' in /etc/fstab. This will block the boot indefinitely waiting for the device, to ensure that slow hardware does not prevent the passphrase prompt from being displayed.
- configure the filesystem as 'noauto' in /etc/fstab, and configure the crypted device as 'noauto' in /etc/crypttab. You will not be prompted for a passphrase at boot time; you will need to manually run 'cryptdisks_start' and 'mount' after you've logged in.

There simply is no reliable way to prompt for the passphrase *only* when the device is available, because unless 'bootwait' is set, there's nothing to prevent gdm starting while you're in the middle of typing in your passphrase.

[Xavier Gnata (xavier-gnata-gmail)]

Steve,

Ok I'm going to use "noauto".
I'm not quite sure there is a point to support my usecase (too complex
and too specific).
You can set this bug to "won't fix". I'm fine with that.

[Steve Langasek (vorlon)]

There's still a real, fixable bug here that we've identified, i.e., that the old upstart job could completely miss *prompting* for a device that's plugged in and hang the boot forever. So we'll continue to track that here.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cryptsetup - 2:1.0.6+20090405.svn49-1ubuntu7.2

---------------
cryptsetup (2:1.0.6+20090405.svn49-1ubuntu7.2) karmic-proposed; urgency=low

  * Depend on watershed.
  * cryptdisks.functions: do_tmp should mount under /var/run/cryptsetup for
    changing the permissions of the filesystem root, not directly on /tmp,
    since mounting on /tmp a) is racy, b) confuses mountall something fierce.
    LP: #475936.

cryptsetup (2:1.0.6+20090405.svn49-1ubuntu7.1) karmic-proposed; urgency=low

  * debian/cryptdisks.functions:
    - wrap the call to /lib/cryptsetup/askpass with watershed, to make sure
      we only ever have one of these running at a time; otherwise multiple
      invocations could steal each other's input and/or write over each
      other's output
    - new function, crypttab_start_one_disk, to look for the named source
      device in /etc/crypttab (by device name, UUID, or label) and start it
      if configured to do so
  * debian/cryptdisks-udev.upstart: new, additional upstart job run once for
    each block device, using the new crypttab_start_one_disk function,
    triggered by udev; this doesn't eliminate the possibility of a race with
    gdm when the decrypted volume isn't a 'bootwait' mount point (since gdm
    kills usplash), but it does eliminate the race between udev and
    cryptsetup. LP: #454898.
    The other cryptdisks-enable job is still needed as well, to give us the
    second pass needed to cover devices that are decrypted using keys stored
    on other encrypted disks. LP: #443980.
  * debian/cryptdisk.functions: initially create the device under a temporary
    name and rename it only at the end using 'dmsetup rename', to ensure that
    upstart/mountall doesn't see our device before it's ready to go.
    LP: #475936.
  * Make the 'start' action of the init script a no-op, this should be
    handled entirely by the upstart job now; ad remove any symlinks from
    /etc/rcS.d on upgrade. LP: #473615.
 -- Steve Langasek <email address hidden> Tue, 22 Dec 2009 23:29:32 +0000

[o6k7 (o6k7)]

I have the same problem but with regular encrypted partition. I enabled proposed repository and updated cryptsetup, but it didn't help. If I don't have bootwait in fstab boot process continues without waiting for the password. If I add bootwait for that device in fstab it asks for the password but tells something like "other devices have to be mounted" asks for the password again and hangs. I am using fresh 9.10 installed on amd64 (Lenovo T400)