cryptdisk start too late at boot process - disks ignored by zfs mount

[Hadmut Danisch]

Hi,

I have the following setup:

- System booting from encrypted SSD (luks, btrfs),
- two more hard-disks, both encrypted (luks) and with zfs (that's because the linux version of zfs does not have encrypted, I've therefore put it on two luks-encrypted disks
- encrypted zfs disks have no partition tables, i.e. luks is put directly into sda and sdb.
- luks-key for zfs disks is derived from boot partition using the key-script coming with cryptsetup

Problem:

the system starts these two encrypted disks too late, i.e. through /etc/init/cryptdisks.conf. The initramfs does not mount them early, since it mounts only root and resume partitions. /etc/init/cryptdisks-udev.conf does not seem to detect the disks.

The problem is, that this runs after /etc/init/zpool-import.conf is triggered and run, thus zfs does not find it's disks when trying to mount them at boot time.

I can easily start the zfs disks by simply running zfs import NAMEOFPOOL manually, but that's not the idea, it should be mounted automatically.

My first guess would be that /etc/init/cryptdisks-udev.conf is not run properly. Maybe that's because the encrypted device is not put in a partition table slice, but directly into /dev/sda and /dev/sdb. Maybe the

 start on block-device-added ID_FS_USAGE=crypto

is not triggered.

/sbin/blkid -o udev -p /dev/sda
ID_FS_UUID=af83410f-2b2a-4271-b7ba-1ef5ccdb1bc5
ID_FS_UUID_ENC=af83410f-2b2a-4271-b7ba-1ef5ccdb1bc5
ID_FS_VERSION=1
ID_FS_TYPE=crypto_LUKS
ID_FS_USAGE=crypto

says ID_FS_USAGE is crypto, which seems correct, however, it does not work.

regards

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: cryptsetup 2:1.6.1-1ubuntu1
ProcVersionSignature: Ubuntu 3.13.0-45.74-generic 3.13.11-ckt13
Uname: Linux 3.13.0-45-generic x86_64
NonfreeKernelModules: zfs zunicode zavl zcommon znvpair
ApportVersion: 2.14.1-0ubuntu3.6
Architecture: amd64
CurrentDesktop: XFCE
Date: Sun Feb 15 19:23:12 2015
SourcePackage: cryptsetup
UpgradeStatus: No upgrade log present (probably fresh install)
crypttab:
 sdc3_crypt UUID=cdb53b1b-58d8-4c61-baad-68e7f19b3920 none luks,discard
 sdc2_crypt UUID=b800eec1-ec70-44fd-aa17-0cc6dec90a9f sdc3_crypt luks,discard,swap,keyscript=/lib/cryptsetup/scripts/decrypt_derived
 sda_crypt UUID=af83410f-2b2a-4271-b7ba-1ef5ccdb1bc5 sdc3_crypt luks,keyscript=/lib/cryptsetup/scripts/decrypt_derived
 sdb_crypt UUID=5824d863-1bb8-4e56-92f4-7866c7878097 sdc3_crypt luks,keyscript=/lib/cryptsetup/scripts/decrypt_derived

[Steve Langasek (vorlon)]

What is /etc/init/zpool-import.conf ? This does not appear to be part of Ubuntu.

Without this file, we can't confirm that your analysis of the problem is correct.

[Hadmut Danisch (hadmut)]

It comes with the zfs kernel modules. zfs is not part of mainstream ubuntu due to license incompatibilities.

/usr/bin/add-apt-repository ppa:zfs-native/trusty

packages zfs-dkms zfsutils

Even if ignoring details of zfs: I have booted the machine with nosplash to see the boot process messages. Opening these two encrypted disks happens very late, almost at the end of boot message. Which means that - beyond the zfs problem - regular services and daemons would fail to start if they had to access these disks instead of running from the boot device.

Formerly I had my machines configured to use raid disks with dmsetup, encrypt the md-device, put a lvm inside and have several logical partitions. This always worked since the initramfs boot code always decrypts this device early, because it contains the root partition.

The problem occured when I dropped raid and lvm, and put my data partitions on completely different disks which are neither root partions, nor resume partitions. Then the cryptdisk does not put them in the cryptroot list contained in the initramfs to be opened before pivoting the root file system and entering the real init.

This is not a problem if these partitions are not used for any service or daemon. If they are just there to be used e.g. for users logging in, this is no problem, since these disks are ready when login prompts appear.

But since zfs is started similar to any daemon on the system (and not through fstab), the problem occurs.

[Steve Langasek (vorlon)]

> /usr/bin/add-apt-repository ppa:zfs-native/trusty

The zpool-import job in this package does:

  start on ( starting mountall )

And mountall is 'start on startup' - which makes zpool-import the very first job to run on the system, *including* before udev. If this job does not correctly handle the underlying disks not yet being available, then that's entirely a bug in zpool-import.

> Even if ignoring details of zfs: I have booted the machine with nosplash
> to see the boot process messages. Opening these two encrypted disks
> happens very late, almost at the end of boot message. Which means that -
> beyond the zfs problem - regular services and daemons would fail to
> start if they had to access these disks instead of running from the boot
> device.

That's not at all true. The opening of encrypted disks is, at *latest*, done in /etc/rcS.d, which is called from /etc/init/rc-sysinit.conf before switching runlevels. And if the devices being mounted are part of the core filesystem, the rc-sysinit job will be triggered by the 'failsafe-boot' event, with the 'filesystem' event happening only after the disks have been unlocked and mounted. So both jobs configured to start on 'filesystem' and jobs configured to start on 'runlevel' will trigger after /etc/init/cryptdisks.conf.

Now, if the setup of those disks /after/ being decrypted is slow, there could be a race between 'runlevel' and mounting those disks. (Since we've been triggered by the 'failsafe-boot', everything at that point is best-effort.) However, /etc/init/cryptdisks.conf is itself a failsafe which is not expected to trigger in the normal case, and you haven't provided any direct evidence that it is being triggered here. All your problems here are is explained by the fact that zpool-import is run before mountall and don't cope with newly-discovered devices.