[19.04] GRUB_ENABLE_CRYPTODISK=y ignored if /boot/ in LUKSv2 format

[Jerome Warnier]

On my freshly-installed and up-to-date 18.10 with full-disk encryption, I decided to upgrade to 19.04 (dev) using _update-manager -d_.
Everything seemed fine, until I rebooted and the Grub configuration apparently had been overwritten, and it does not offer to enter the master password, so cannot find the root VG.

This is all happening on my Lenovo T480 PC, with only Ubuntu (no dual-boot).

I tried boot-repair, applying any suggestion it threw at me, but it could not fix the issue.

I did try to add GRUB_ENABLE_CRYPTODISK=y in /etc/default/grub and successfully run update-grub in a chroot, but this did not fix the issue.

Feel free to ask for more details, but diagnostic is limited, as the system is not booting, from the very early stages.

Thanks.

[Jerome Warnier (jwarnier)]

To me, this seems related to #1062623.

[TJ (tj)]

I've confirmed this with a new 19.04 install, and it's not just the d-r-u process.

In a manually configured system with:

sda3 -> LUKS_BOOT > /boot/
sda4 -> sda4_crypt -> LVM -> /dev/mapper/ubuntu-rootfs -> /

# grep -rn GRUB_ENABLE_CRYPTODISK /etc/default/grub /etc/grub.d/ /usr/lib/grub/ /usr/sbin/grub* /usr/share/grub/
/etc/default/grub:33:GRUB_ENABLE_CRYPTODISK=y
Binary file /usr/sbin/grub-install matches
/usr/sbin/grub-mkconfig:245: GRUB_ENABLE_CRYPTODISK \
/usr/share/grub/grub-mkconfig_lib:77: if [ x$GRUB_ENABLE_CRYPTODISK = xy ]; then
/usr/share/grub/grub-mkconfig_lib:162: if [ x$GRUB_ENABLE_CRYPTODISK = xy ]; then

# grub-install /dev/sda
# update-grub

both fail to add the luks and crypto modules. Core image doesn't get them:

grub-install: info: grub-mkimage --directory '/usr/lib/grub/i386-pc' --prefix '/grub' --output '/boot/grub/i386-pc/core.img' --format 'i386-pc' --compression 'auto' --config '/boot/grub/i386-pc/load.cfg' 'ext2' 'biosdisk' 'search_fs_uuid'

and nor does /boot/grub/grub.cfg:

...
menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-a0f21372-a236-4e33-9bfa-9921ca89d5b7' {
        recordfail
        load_video
        gfxmode $linux_gfx_mode
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod ext2
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root 997c6523-62d6-4274-bf2b-3fa0bef7f181
        else
          search --no-floppy --fs-uuid --set=root 997c6523-62d6-4274-bf2b-3fa0bef7f181
        fi
        linux /vmlinuz-5.0.0-16-generic root=/dev/mapper/ubuntu-rootfs ro quiet splash $vt_handoff
        initrd /initrd.img-5.0.0-16-generic
}
...

[Mathieu Trudel-Lapierre (cyphermox)]

If you manage to boot off a USB key or something else and discover the drives, can you then run grub-probe on the device to check if grub correctly identifies the disk as a crypto disk?

Running the following command:

sudo grub-probe --device=/dev/mapper/<your crypto device> --target=abstraction

Should get you output a bit like the following:
cryptodisk
luks
gcry_rijndael
gcry_rijndael
gcry_sha256

If not, then the device is not getting recognized properly.

At a glance, my first guess would be that the format changes in LUKS that happened in 19.04 are causing an issue, but I'm not sure. I would not have expected this to be a problem on upgrade.

Please report with the output for the command above if possible, so we can dig in more into what is happening in grub's probing.

[TJ (tj)]

Further investigation revealed that a change to the semantics of 'cryptsetup' tool is responsible for the specific manual build I did.

On 18.04 'cryptsetup luksFormat --type=luks ...' would create a LUKS v1 header. On 19.04 it creates a LUKS v2 header. GRUB only supports LUKS v1 headers currently.

To resolve this on 19.04 we need to use "cryptsetup luksFormat --type=luks1 ...".

If the do-release-upgrade process is breaking existing bootable GRUB file-systems this is likely because there is some LUKSv1 to LUKSv2 conversion being done - which the OS is fine with. If this is correct, presumably the conversion code doesn't check if GRUB is also using the LUKS device and in converting it to v2 breaks GRUB.

My workaround from a liveISO to switch back to LUKS v1 was to do the following (as UID 0 root). I've tried to make this portable but there may be some typing mistakes so please proof-read carefully!

# set the following variables for your target system

# device where OS is installed (usually the same as GRUB_DEV)
DEV_INSTALL="sdZ"
# partition number for /boot/ partition (e.g. from /dev/sdZ3)
PART_BOOT="3"
# partition for root file-system (e.g. from /dev/sdZ4)
PART_ROOT="4"
# device-mapper name of unencrypted /boot/ device
LUKS_BOOT="LUKS_BOOT"
# LVM names of root file-system
VG="ubuntu"
LV="rootfs"

# shouldn't need to edit any variables from here onwards

# device to install GRUB on
export DEV_GRUB="/dev/$INSTALL_DEV"

# device-mapper name of the unencrypted /boot device
FS_BOOT="/dev/mapper/$LUKS_BOOT"
# encrypted /boot device
DEV_BOOT="/dev/${DEV_INSTALL}${PART_BOOT}"

# device-mapper name of the rootfs device
FS_ROOT="/dev/mapper/$VG-$LV"
LUKS_ROOT="${INSTALL_DEV}${PART_ROOT}_crypt"
# encrypted rootfs device
DEV_ROOT="/dev/${INSTALL_DEV}${PART_ROOT}"

# mountpoint for root file-system
TARGET="/target"
# boot directory
BOOT="$TARGET/boot"
# back-up directory
BAK="/tmp/boot.bak"

mkdir $TARGET $BAK
cryptdisk_start $LUKS_ROOT
lvm vgchange -ay
mount $FS_ROOT $TARGET
cryptdisk_start $LUKS_BOOT
mount $FS_BOOT $BOOT
cp -a $BOOT/* $BAK/
umount $BOOT
cryptdisk_stop $LUKS_BOOT
cryptsetup luksFormat --type=luks1 $DEV_BOOT
# use the existing UUID from crypttab
cryptsetup luksUUID --uuid $( sed -n '/'$LUKS_BOOT'/ s/.*UUID=\([^ ]*\).*/\1/ p' $TARGET/etc/crypttab ) $DEV_BOOT
cryptdisk_start $LUKS_BOOT
mkfs.ext -L /boot $FS_BOOT
mount $FS_BOOT $BOOT
cp -a $BAK/* $BOOT/

for n in proc sys dev etc/resolv.conf; do mount --rbind /$n $TARGET/$n; done
chroot $TARGET

# inside the chroot of the broken OS
mount -a
update-grub
grep -o 'cryptodisk' || echo "Eror: grub.cfg has no LUKS support"
grub-install -v $DEV_GRUB >& /tmp/grub-install.log
grep 'grub-mkimage' /tmp/grub-install.log | grep -o cryptodisk || echo "Error: GRUB has no LUKS support installed"

# if all OK...
exit

# clean up
for n in etc/resolv.conf dev sys proc $BOOT $TARGET; do umount $TARGET/$n; done
cryptdisk_stop $LUKS_BOOT
cryptdisk_stop $LUKS_ROOT

# finished
exit

# reboot and test

[Russ Long (tfmm)]

I can confirm similar behavior, but slightly different setup.

I installed 19.04 from scratch, and used the disk encryption with lvm option during setup.

Everything was working fine until upgrading to the latest kernel, 5.0.0-16-generic.

The previous kernel I had installed, 5.0.0-15-generic, still works fine.

I see no differences in the grub config for the 2 kernels:

 menuentry 'Ubuntu, with Linux 5.0.0-16-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.0.0-16-generic-advanced-69cc3212-6aef-415d-a65d-30362fc514f6' {
  recordfail
  load_video
  gfxmode $linux_gfx_mode
  insmod gzio
  if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
  insmod part_gpt
  insmod ext2
  if [ x$feature_platform_search_hint = xy ]; then
    search --no-floppy --fs-uuid --set=root 070883ed-c707-4199-b2b2-84145dc3355d
  else
    search --no-floppy --fs-uuid --set=root 070883ed-c707-4199-b2b2-84145dc3355d
  fi
  echo 'Loading Linux 5.0.0-16-generic ...'
  linux /vmlinuz-5.0.0-16-generic root=/dev/mapper/ubuntu--vg-root ro quiet splash $vt_handoff
  echo 'Loading initial ramdisk ...'
  initrd /initrd.img-5.0.0-16-generic
}

 menuentry 'Ubuntu, with Linux 5.0.0-15-generic' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.0.0-15-generic-advanced-69cc3212-6aef-415d-a65d-30362fc514f6' {
  recordfail
  load_video
  gfxmode $linux_gfx_mode
  insmod gzio
  if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
  insmod part_gpt
  insmod ext2
  if [ x$feature_platform_search_hint = xy ]; then
    search --no-floppy --fs-uuid --set=root 070883ed-c707-4199-b2b2-84145dc3355d
  else
    search --no-floppy --fs-uuid --set=root 070883ed-c707-4199-b2b2-84145dc3355d
  fi
  echo 'Loading Linux 5.0.0-15-generic ...'
  linux /vmlinuz-5.0.0-15-generic root=/dev/mapper/ubuntu--vg-root ro quiet splash $vt_handoff
  echo 'Loading initial ramdisk ...'
  initrd /initrd.img-5.0.0-15-generic
 }

[Russ Long (tfmm)]

Disregard my post, my issue was unrelated. (I inadvertently must have removed my crypttab file, sigh.)

[Launchpad Janitor (janitor)]

[Expired for grub2 (Ubuntu) because there has been no activity for 60 days.]