dracut does not support booting from an encrypted ZFS volume

[Benjamin Drung]

Dracut does not support booting from an encrypted ZFS volume. Steps to reproduce:

1. In a VM install Ubuntu 24.10 with an encrypted ZFS volume
2. Install dracut afterwards: sudo apt install dracut zfs-dracut
3. Add rd.shell to the boot arguments
4. Reboot

The boot will fail:

```
dracut-pre-mount[817]: Warning: ZFS: Key /run/keystore/rpool/system.key for rpool hasn't appeared. Trying anyway.
dracut-pre-mount[863]: Key load error: Failed to open key material file: No such file or directory
[FAILED] Failed to mount sysroot.mount - /sysroot.
```

The initrd should have asked for the password, but it did not.

ProblemType: Bug
DistroRelease: Ubuntu 24.10
Package: dracut-core 102-3ubuntu2
ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1
Uname: Linux 6.8.0-31-generic x86_64
NonfreeKernelModules: zfs
ApportVersion: 2.28.1-0ubuntu4
Architecture: amd64
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
Date: Fri Jun 21 09:35:42 2024
InstallationDate: Installed on 2024-06-20 (1 days ago)
InstallationMedia: Ubuntu 24.10 "Oracular Oriole" - Daily amd64 (20240617)
ProcEnviron:
 LANG=de_DE.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm-256color
 XDG_RUNTIME_DIR=<set>
SourcePackage: dracut
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[l (tcrfvyguiop)]

https://github.com/dracut-ng/dracut-ng/pull/529

[Benjamin Drung (bdrung)]

I tried 103-1ubuntu2 with the patch from https://github.com/dracut-ng/dracut-ng/pull/529 and the result is the same.

[l (tcrfvyguiop)]

> The initrd should have asked for the password, but it did not.

If you supply the key in the initrd (e.g. with /etc/cryptab), it should work.

For the "should have asked for the password", `--add bash` is likely a workaround. See https://github.com/zbm-dev/zfsbootmenu/issues/690

Possible fix - https://github.com/dracut-ng/dracut-ng/pull/1012 . Please help testing it !

[Benjamin Drung (bdrung)]

I tested dracut 106-2ubuntu1 in a plucky VM with the same results. I attached the rdsosreport.txt.

[l (tcrfvyguiop)]

This discussion might be relevant - https://www.reddit.com/r/zfs/comments/1g16637/whats_the_design_rationale_for_the_keystore_on/ .

Perhaps someone with knowledge on how the installer configures ZFS could help out.

Which component suppose to populate /run/keystore/rpool/system.key ?

Why expecting a password prompt when dracut seems to be configured to use a key ?

What happens with `keylocation=prompt` ?

[Serem (seremkjerkegor)]

Hmmh so it wasnt ill intent? Im still learning

On Sat, 8 Mar 2025, 02:05 l, <email address hidden> wrote:

> This discussion might be relevant -
>
> https://www.reddit.com/r/zfs/comments/1g16637/whats_the_design_rationale_for_the_keystore_on/
> .
>
> Perhaps someone with knowledge on how the installer configures ZFS could
> help out.
>
> Which component suppose to populate /run/keystore/rpool/system.key ?
>
> Why expecting a password prompt when dracut seems to be configured to
> use a key ?
>
> What happens with `keylocation=prompt` ?
>
> --
> You received this bug notification because you are subscribed to Ubuntu
> ubuntu-25.04.
> Matching subscriptions: sjeremkjerkegor
> https://bugs.launchpad.net/bugs/2070066
>
> Title:
> dracut does not support booting from an encrypted ZFS volume
>
> Status in dracut package in Ubuntu:
> Triaged
> Status in zfs-linux package in Ubuntu:
> Confirmed
>
> Bug description:
> Dracut does not support booting from an encrypted ZFS volume. Steps to
> reproduce:
>
> 1. In a VM install Ubuntu 24.10 with an encrypted ZFS volume
> 2. Install dracut afterwards: sudo apt install dracut zfs-dracut
> 3. Add rd.shell to the boot arguments
> 4. Reboot
>
> The boot will fail:
>
> ```
> dracut-pre-mount[817]: Warning: ZFS: Key /run/keystore/rpool/system.key
> for rpool hasn't appeared. Trying anyway.
> dracut-pre-mount[863]: Key load error: Failed to open key material file:
> No such file or directory
> [FAILED] Failed to mount sysroot.mount - /sysroot.
> ```
>
> The initrd should have asked for the password, but it did not.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 24.10
> Package: dracut-core 102-3ubuntu2
> ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1
> Uname: Linux 6.8.0-31-generic x86_64
> NonfreeKernelModules: zfs
> ApportVersion: 2.28.1-0ubuntu4
> Architecture: amd64
> CasperMD5CheckResult: pass
> CurrentDesktop: ubuntu:GNOME
> Date: Fri Jun 21 09:35:42 2024
> InstallationDate: Installed on 2024-06-20 (1 days ago)
> InstallationMedia: Ubuntu 24.10 "Oracular Oriole" - Daily amd64
> (20240617)
> ProcEnviron:
> LANG=de_DE.UTF-8
> PATH=(custom, no user)
> SHELL=/bin/bash
> TERM=xterm-256color
> XDG_RUNTIME_DIR=<set>
> SourcePackage: dracut
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/dracut/+bug/2070066/+subscriptions
>
>

[Serem (seremkjerkegor)]

I was gonna do more android focused exploits via wifi or bluetooth cause im
very fresh and dumb

On Sat, 8 Mar 2025, 07:19 Serem Kjerkegor, <email address hidden> wrote:

> Hmmh so it wasnt ill intent? Im still learning
>
> On Sat, 8 Mar 2025, 02:05 l, <email address hidden> wrote:
>
>> This discussion might be relevant -
>>
>> https://www.reddit.com/r/zfs/comments/1g16637/whats_the_design_rationale_for_the_keystore_on/
>> .
>>
>> Perhaps someone with knowledge on how the installer configures ZFS could
>> help out.
>>
>> Which component suppose to populate /run/keystore/rpool/system.key ?
>>
>> Why expecting a password prompt when dracut seems to be configured to
>> use a key ?
>>
>> What happens with `keylocation=prompt` ?
>>
>> --
>> You received this bug notification because you are subscribed to Ubuntu
>> ubuntu-25.04.
>> Matching subscriptions: sjeremkjerkegor
>> https://bugs.launchpad.net/bugs/2070066
>>
>> Title:
>> dracut does not support booting from an encrypted ZFS volume
>>
>> Status in dracut package in Ubuntu:
>> Triaged
>> Status in zfs-linux package in Ubuntu:
>> Confirmed
>>
>> Bug description:
>> Dracut does not support booting from an encrypted ZFS volume. Steps to
>> reproduce:
>>
>> 1. In a VM install Ubuntu 24.10 with an encrypted ZFS volume
>> 2. Install dracut afterwards: sudo apt install dracut zfs-dracut
>> 3. Add rd.shell to the boot arguments
>> 4. Reboot
>>
>> The boot will fail:
>>
>> ```
>> dracut-pre-mount[817]: Warning: ZFS: Key /run/keystore/rpool/system.key
>> for rpool hasn't appeared. Trying anyway.
>> dracut-pre-mount[863]: Key load error: Failed to open key material
>> file: No such file or directory
>> [FAILED] Failed to mount sysroot.mount - /sysroot.
>> ```
>>
>> The initrd should have asked for the password, but it did not.
>>
>> ProblemType: Bug
>> DistroRelease: Ubuntu 24.10
>> Package: dracut-core 102-3ubuntu2
>> ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1
>> Uname: Linux 6.8.0-31-generic x86_64
>> NonfreeKernelModules: zfs
>> ApportVersion: 2.28.1-0ubuntu4
>> Architecture: amd64
>> CasperMD5CheckResult: pass
>> CurrentDesktop: ubuntu:GNOME
>> Date: Fri Jun 21 09:35:42 2024
>> InstallationDate: Installed on 2024-06-20 (1 days ago)
>> InstallationMedia: Ubuntu 24.10 "Oracular Oriole" - Daily amd64
>> (20240617)
>> ProcEnviron:
>> LANG=de_DE.UTF-8
>> PATH=(custom, no user)
>> SHELL=/bin/bash
>> TERM=xterm-256color
>> XDG_RUNTIME_DIR=<set>
>> SourcePackage: dracut
>> UpgradeStatus: No upgrade log present (probably fresh install)
>>
>> To manage notifications about this bug go to:
>>
>> https://bugs.launchpad.net/ubuntu/+source/dracut/+bug/2070066/+subscriptions
>>
>>

[Will Rouesnel (w-rouesnel)]

This is still a problem today on 24.04 when trying to setup dracut with native ZFS encryption (via a regular 24.04 installation and then apt install dracut zfs-dracut).

The problem is the keystore isn't getting mounted before the ZFS mount is attempted. Not sure what the best way to debug it is.

[Will Rouesnel (w-rouesnel)]

Okay I've been able to get this to work: the problem is that dracut doesn't install anything from /etc/crypttab unless it's run in --host-only mode, but if it is, then it generally fails to install anything cryptography related (under an Ubuntu ZFS-on-root native encryption setup).

It's worth noting no combination of rd.auto rd.luks=1 would seem to detect the keystore partition, but adding:

```
#/etc/dracut.conf.d/00-crypttab.conf
install_items+=" /etc/crypttab "
```

to my dracut.conf file *did* get the encrypted partition to mount. However since dracut has no idea what it should do with that.

It's possible to use an undocumented feature here to fix this explicitly in the simple config:

```
#/etc/dracut.conf.d/01-keystore-rpool-mnt.conf
fstab_lines+=" /dev/mapper/keystore-rpool /run/keystore/rpool auto "
```

(note yes this is a malformed line - dracut appends '0 0 2' to whatever you put here for the last element)

So the problem seems to be that zfs-dracut needs to explicitly handle the Ubuntu keystore convention, since I can't see how dracut would figure it out otherwise - i.e detecting a keystore should trigger a decrypt operation (or better, force the relevant crypttab line to be included so tpm2-device etc. options can be used) and then the scripts need to execute the mount point.

I've tested this setup as letting you login with a password, but it has another problem: since the ZFS scripts don't know they're waiting for their own decryption (they're doing udevsettle) then after about 15-20 seconds dracut crashes to the recovery shell from the password prompt.