[18.04] Installation boot failure. WARNING: invalid line in /etc/crypttab

[Dylan Taylor]

I worked with "TJ-" on Ubuntu IRC (#ubuntu) on April 27th in order to debug this. On a new Ubuntu 18.04 installation, it is not possible to decrypt the volume when it's installed on an NVMe device with the encryption selected. Changing the device-mapper name to the luks-$UUID format apparently did something to correct it, but there's still something else going on.

--- update from Tj ---

Dylan repeatedly installed 18.04 Desktop setting up F.D.E. with (originally) a complex passphrase but later also a simple (all ASCII) phrase.

At boot-time the installed system fails to unlock the device during initial ramdisk processing, getting stuck in a loop and seemingly never dropping to the shell for user diagnosis and correction.

Original messages (copied from the attached photo):

Please unlock disk nvme0n1p3_crypt: <--- types passphrase
  Volume group "ubuntu-vg" not found
  Cannot process volume group ubuntu-vg
device-mapper: reload ioctl on failed: invalid argument <--- note multiple space gap
Failed to setup dm-crypt key mapping for device /dev/disk/by-uuid/4bdade98-fdbe-4a9e-b287-283b4c52c1a0.
Check that kernel supports aes-xts-plain64 cipher (check syslog for more info).

The volume is accessible from the Try Ubuntu session (unlocks correctly).

My [Tj] suspicion was a keyboard mapping issue or initrd.img missing required kernel modules for cryptographic functions.

We used a chroot environment to investigate and added "set -x" to the start of /usr/share/initamfs-tools/hooks/cryptroot to capture the initrd.img build using

update-initramfs -uv |& tee /tmp/uir.log

The resulting log (see attached 'broken' log) shows the warning in the title of this bug twice and the root device ignored as far as configuring the initrd.img goes:

WARNING: invalid line in /etc/crypttab

Analysing the log of the shell commands shows an awk command failing to recognise the sole crypttab entry because it is expecting the device-mapper name format to be "luks-${UUID}" not "nvme0n1p3_crypt".

Dylan made a back-up then changed the entry to use the seemingly correct name format and rebuilt the initrfd.img (see attached 'working' log). There is no warning now and the cryptroot hook goes ahead and adds kernel modules and tooling to the initrd.img.

The system still fails to boot but with different messages:

Please unlock device luks-98c2ce02-6cb1-4a2d-a086-1e8cf78a3c58 <--- types passphrase
[ timestamp ] device-mapper: table: 253:0: crypt: unknown target type
device-mapper: reload ioctol on failed: Invalid argument
Failed to setup dm-crypt key mapping for device /dev/disk/by-uuid/98c2ce02-6cb1-4a2d-a086-1e8cf78a3c58.
Check that kernel supports aes-xts-plain64 cipher (check syslog for more info).
cryptsetup (luks-98c2ce02-6cb1-4a2d-a086-1e8cf78a3c58): cryptsetup failed, bad password or options?

[Dylan Taylor (dylanmtaylor)]

My usual passphrase is something complex similar to "--[-Word#Word.A#More#Words*123*Word-]--", with all of those symbols included, but with the passphrase abcdefg12, I still hit this issue.

Devices: https://paste.ubuntu.com/p/QMSmBgtnD9/
fstab: https://paste.ubuntu.com/p/5NSRGqJHY5/
logs from "update-initramfs -vu -k 4.15.0-20-generic |& tee /tmp/uir.log" https://paste.ubuntu.com/p/cP6Cj8W59j/

[Dylan Taylor (dylanmtaylor)]

Full lshw output, in case this can't be reproduced in a VM easily: paste.ubuntu.com/p/jx2zR7JpTR/

[TJ (tj)]

Original /etc/crypttab:

nvme0n1p3_crypt UUID=98c2ce02-6cb1-4a2d-a086-1e8cf78a3c58 none luks,discard

changed to

luks-98c2ce02-6cb1-4a2d-a086-1e8cf78a3c58 UUID=98c2ce02-6cb1-4a2d-a086-1e8cf78a3c58 none luks,discard

With the change the update-initramfs hooks/cryptroot logs shows the cryptsetup support files are correctly added.

[TJ (tj)]

Dylan: I've updated the bug description to describe the situation as we left it.

There is a test I'd like you to do which involves forcing a break in initrd processing and dropping to the shell before the cryptsetup prompt.

Once there I'd like you to verify the NVME device is visible along with the UUIDs. I'm wondering if the scripts are so dumb that they don't check for the presence of the underlying block device (with UUID 98c2ce02-6cb1-4a2d-a086-1e8cf78a3c58 a.k.a. partition #3) before prompting you for the passphrase.

At boot-time interrupt GRUB by tapping the Esc key to get the GRUB boot menu. There, highlight the default (top) entry and press 'e' to edit it. Navigate down to the line beginning "linux ..." (the kernel command-line) and add to it "break=premount" then press Ctrl+X (or F10) to boot with that change.

That should lead to an initramfs busybox shell command-line.

Note: other possible break=XXX values are:

$ grep maybe_break /usr/share/initramfs-tools/init
maybe_break top
maybe_break modules
maybe_break premount
maybe_break mount
maybe_break mountroot
maybe_break bottom
maybe_break init

Once in the shell check the device is known with some or all of these commands and any others you can think of (busybox is limited and additional tools you'd expect in the full install won't be available):

blkid
ls -l /dev/disk/by-*/
ls -l /dev/mapper/

You can also attempt to manually unlock using:

cryptsetup open --type=luks /dev/nvme0n1p3 luks-98c2ce02-6cb1-4a2d-a086-1e8cf78a3c58

(set the /dev/nvme0n1p3 underlying device path to whatever your exploration has found, including possibly "/dev/disk/by-uuid/98c2ce02-6cb1-4a2d-a086-1e8cf78a3c58" )

If that works (or fails) the resulting messages might be important so be prepared to photograph them.

If it works, then you will see the new node ("luks-98c2ce02-6cb1-4a2d-a086-1e8cf78a3c58") with

ls -l /dev/mapper/

And now, if this hasn't already happened automatically via udev, you should be able to force LVM discovery of ubuntu-vg with

vgchange -ay

At this point if "ls -l /dev/mapper/" now does show a luks-98c2ce02-6cb1-4a2d-a086-1e8cf78a3c58 node you can try to continue the boot by pressing Ctrl+D or typing "exit". If those don't help then re-running the entire script might be possible instead using

exec /init

[TJ (tj)]

Correct to the final instruction of my last comment:

At this point if "ls -l /dev/mapper/" now does show the LVM ** ubuntu--vg-root ** node for the root file-system you can try to continue the boot by pressing Ctrl+D or typing "exit". If those don't help then re-running the entire script might be possible instead using

exec /init

[Dylan Taylor (dylanmtaylor)]

TJ, some new information - this is a regression. Trying the EXACT same install path with the "complex" passphrase I usually use on an old bionic "daily" build worked perfectly. I was able to boot into the system and setup my applications. I'd rather not wipe it right now, but something broke between the official 18.04 build and this one:

dtaylor@dylantaylor-pc:~/Downloads$ md5sum bionic-desktop-amd64.iso
9cf2ab79a6cb8fd3cdefc34e7d1a9841 bionic-desktop-amd64.iso
dtaylor@dylantaylor-pc:~/Downloads$ stat bionic-desktop-amd64.iso
  File: bionic-desktop-amd64.iso
  Size: 1829273600 Blocks: 3572816 IO Block: 4096 regular file
Device: fd02h/64770d Inode: 19008339 Links: 1
Access: (0664/-rw-rw-r--) Uid: ( 1000/ dtaylor) Gid: ( 1000/ dtaylor)
Access: 2018-04-27 19:52:52.751664153 -0400
Modify: 2018-04-04 14:37:45.311616468 -0400
Change: 2018-04-25 13:51:20.865647567 -0400
 Birth: -

[Dylan Taylor (dylanmtaylor)]

Once I was into the system, I ran "sudo apt clean all; sudo apt update; sudo apt -y full-upgrade", which completed successfully. Upon reboot, everything still worked. It seems to be part of the setup process itself that is broken.

[Steve Langasek (vorlon)]

It is unclear to me what the cause of this bug is. It appears to be a mismatch between the device name as encoded in /etc/crypttab, and the actual device name used at runtime by the installer, preventing the cryptsetup initramfs hook from probing and detecting the correct device (and therefore the correct kernel modules) at initramfs generation time.

If that's the case, it seems more like an installer bug than a cryptsetup bug, because why should /etc/crypttab have a different name for a device than the one shown by the kernel?

What is the cryptsetup config as inserted into the initramfs, in the case where things are broken? (unmkinitramfs /boot/initrd.img-$(uname -r) /tmp/unpack; cat /tmp/unpack/main/conf/conf.d/cryptroot)

[Mathieu Trudel-Lapierre (cyphermox)]

How was the system installed? Did this use d-i (the alternate installer image), or ubiquity?

[Steve Langasek (vorlon)]

Information was requested in October to help us reproduce this. We would like to have this solved for 18.04.2 but need more information.

[Adam Conrad (adconrad)]

Comment 11 mentions bionic-desktop images, so I'm assuming ubiquity without more information to go on.

[Dylan Taylor (dylanmtaylor)]

This system was installed with Ubiquiti.

[Mathieu Trudel-Lapierre (cyphermox)]

I can't make any sense of it. I have similar setups here; the systems get installed (with 18.04 as well as Disco) with nvme#n#p#_crypt devices, and those load properly.

Anything else you can add about the specifics of your system? Seems like you used guided partitioning, just enabling disk encryption?

Is there any way you can reproduce this on another system, or the same system, using a recent 18.04 image or a Disco image? What flavour of Ubuntu were you installing?