ubiquity does not support existing LUKS encrypted partitions

[Tim Edwards]

The ubiquity installer, including both the text version on the Ubuntu Server live CD and its GUI version does not support using existing LUKS encrypted partitions. That means out of the box it wants to completely wipe all existing partitions, this is a serious failure for what's now the default installer in Ubuntu 20.04.

There is a workaround that involves running 'cryptsetup luksOpen' before running the installer, so that it can see the LVM LV's that are on the encrypted device and use them. But even doing this won't work properly because the installer stuffs up the /etc/crypttab and initrd. You have to manually create /etc/crypttab and initrd.

This is a major failure since the installer doesn't support re-using partitions that it itself created.

Steps to reproduce:
- get a system with a blank disk, or just create a new Virtualbox VM
- install Ubuntu 20.04 on an encrypted LUKS partition (using the options in the ubiquity installer), creating a / (root), /home and swap LV on top of the LUKS partition/LVM volume group
- boot the system to verify it starts, then restart into the installer again
- try to re-use the existing partitions, you can't unless you follow the manual workaround below:

**Workaround**
**Ubuntu 20.04: Extra Steps for Re-Using Existing LUKS Encrypted Partition**
(replace nvme0n1p8 with your encrypted LUKS partition)

- BEFORE starting the installer (if in Ubuntu or Kubuntu live CD desktop), or at the first step (if using Ubuntu Server text-based live CD installer):
# open existing LUKS partition (
cryptsetup luksOpen /dev/nvme0n1p8 nvme0n1p8_crypt

- Then, either do this at the end after the installer has run, or boot into a live CD environment (e.g. Kubuntu) and do:
cryptsetup luksOpen /dev/nvme0n1p8 nvme0n1p8_crypt
mount -o subvol=@ /dev/mapper/MainVG-root /mnt/
mount /dev/nvme0n1p7 /mnt/boot/
mount --rbind /dev/ /mnt/dev/
mount --rbind /sys/ /mnt/sys/
mount --rbind /run/ /mnt/run/
mount --rbind /proc/ /mnt/proc/
chroot /mnt /bin/bash -l
blkid | grep crypto_LUKS
# Example: /dev/nvme0n1p8: UUID="8cb9831a-692e-4b0e-936f-72529a3ed56d" TYPE="crypto_LUKS" PARTUUID="139f23d2-a0ff-4f4f-b41f-8083964ac894"
apt install vim
vim /etc/crypttab
# Add a line for the encrypted partiton, e.g:
# nvme0n1p8_crypt UUID="8cb9831a-692e-4b0e-936f-72529a3ed56d" none luks
#
# MAKE SURE There's a newline at end of /etc/crypttab

# update the initramfs
update-initramfs -u -k all

umount -l /mnt

# Optional, probably not needed
grub-install --recheck /dev/nvme0n1
grub-mkconfig -o /boot/grub/grub.cfg

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: ubiquity (not installed)
ProcVersionSignature: Ubuntu 5.4.0-53.59-generic 5.4.65
Uname: Linux 5.4.0-53-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.12
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: KDE
Date: Sat Nov 14 21:30:33 2020
InstallationDate: Installed on 2020-11-02 (11 days ago)
InstallationMedia: Kubuntu 20.04.1 LTS "Focal Fossa" - Release amd64 (20200731)
SourcePackage: ubiquity
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Artem Kovalov (artemkovalov)]

I confirm. That's a quite horrible experience. The previous text-based installer worked perfectly to unlock the disc and map the partitions. It was never supported in the GUI installer and I killed my disk once because of that.

This is a major downturn as encryption is a best-practice for quite a long time.

[Tim Edwards (tkedwards)]

Yeah it's a pretty ridiculous bug to have in an LTS release. Disk encryption is a must these days, and other OSes like Windows support it by default

[Uwe Geuder (ubuntulp-ugeuder)]

I have used the "cryptsetup luksOpen` work-around successfully already years ago.

However, when installing 20.04 with the server installer it does no longer work.
Installation aborts; in the log I found the cause to be

   raise ValueError("encryption key or keyfile must be specified")

from

https://github.com/canonical/curtin/blob/45877d0e1f5774dc424f73b0c1c16b65e0353e63/curtin/commands/block_meta.py#L1394

(Note: This is not necessary the commit that was in use when I used the installer. I just looked at the current master branch to point to the message I got. I did upload the crash report, but I don't have a link to my upload.)

Obviously the installer got confused ny the work-around because it noticed that there is a LUKS container, but it had never gotten any secret related to it.

Is there more than one Server installer, that could explain the discrepancy against the instructions above?

I tried the xubuntu-20.04.2.0 and there the work-around still works.

[Uwe Geuder (ubuntulp-ugeuder)]

> I tried the xubuntu-20.04.2.0

Sorry, that should have been

xubuntu-20.04.2.0 installer

[Uwe Geuder (ubuntulp-ugeuder)]

> Then, either do this at the end after the installer has run, or boot into a live CD environment (e.g. Kubuntu) and do:

The many steps given there are not really necessary. If you installed without downloading updates the following steps should be easier:

1. The first boot after the installation will not succeed, but give you a busybox shell from initramfs.
1. In that shell give the cryptsetup command as given above.
1. Exit the shell using Ctrl-D.
1. The boot will now complete.
1. Create the /etc/crypttab file. Typically you can just copy it from your previous root file system.
1. Update the system using apt or whatever you use. Unless you do this immediately when the installer image was built there will be updates that recreate the initramfs. The recreated initramfs will contain your new /etc/crypttab.
1. Reboot.
1. It works!

[Mauricio Villegas (mauricio-ville)]

I am trying to install ubuntu 20.04 on top of an old LUKS partition with ubuntu 18.04. I have a single encrypted partition so I set it as mount point "/". But the installation then fails saying that the bootloader installation failed. For bootloader I selected the disk where the LUKS partition is. There are other two partitions on that disk, an efi and an ext2. Should the selection for the bootloader installation be different?

[TechnoSwiss (misterb)]

Using the installer for 20.04.3 server using the workaround and the installer is failing with raise ValueError("encryption key or keyfile must be specified") in the log as well.

[Rolf Kutz (vzsze)]

Do you have to use the server installer to use the workaround or can the desktop installer do the same?

[Andri Möll (moll)]

Has anyone come up with a workaround for the "ValueError: encryption key or keyfile must be specified" bug? I'm trying to install Ubuntu Server 20.04.3 here, too, after manually creating a slightly tweaked LUKS device, and the installer keeps failing. Perhaps an older point-release of 20.04 still works? Thanks!

[Tim Edwards (tkedwards)]

This problem's still there on 22.04. I simplified my notes on how to workaround it, HTH:

**Ubuntu 22.04 and Newer: Extra Steps for Re-Using Existing LUKS Encrypted Partition**
This has to be done using the Kubuntu or Ubuntu desktop live CD. Using the Ubuntu Server Live CD (with text-based installer) doesn't work in either of 20.04 or 22.04 as the installer will just crash a few seconds after it starts installing. Steps:
- Boot off Kubuntu Live CD, select Try Kubuntu to get the desktop
- Open a terminal and run: sudo cryptsetup luksOpen /dev/sdX sdX_crypt #replace with nvme0nXpX or whatever the machine has
- Start the installer and run through it. The LVM LV's will appear in the installer. Note that (as of 22.04 at least) makes you create an EFI system partition - hopefully you have one even if it's unformatted
- End the installer and reboot as normal. You'll find it dumps you to an initramfs prompt. At this point again run:
sudo cryptsetup luksOpen /dev/sdX sdX_crypt #replace with nvme0nXpX or whatever the machine has
exit
It'll now continue booting successfully
- Once booted, create a valid /etc/crypttab:
sudo bash
export MAINLUKSPARTITION=sda4 # other examples: nvme0n1p3, whatever the machine has
UUID=$(blkid -s UUID -o value /dev/${MAINLUKSPARTITION})
echo "${MAINLUKSPARTITION}_crypt UUID=${UUID} none luks,discard" >> /etc/crypttab
cat /etc/crypttab
# check it looks right

update-initramfs -u -k all

[Andreas Hasenack (ahasenack)]

> - Start the installer and run through it. The LVM LV's will appear in the installer. Note that
> (as of 22.04 at least) makes you create an EFI system partition - hopefully you have one even if
> it's unformatted

Note this doesn't work anymore with the lunar installer, see https://github.com/canonical/ubuntu-desktop-installer/issues/1770

[borillionstar (babakrezai)]

Still an issue, this makes things difficult with luks encrypted rootfs and database partitions.

[Fnx (td-safemail-tmp)]

Installation with ubuntu 22.04.02 LTS server edition (ubuntu-22.04.2-live-server-amd64). Bug is still an issue.
Manual installation selected. If encrypted LVM is mounted, partition can be selected. A selection of a distinct boot partition seems also mandatory to go further.
However Installation fails during the next step (log: ValueError("encryption key or keyfile must be specified"))

[Josef Wolf (jw-raven)]

I am also affected by this bug with 22.04 LTS server cloud-init base autoinstall, updated to newest subiquity during install.

In early-comands, I
 - create an appropriate /etc/crypttab
 - cryptdisks_start all the encrypted volumes

Still no success, since subiquity verifies volume name using

  dmsetup info -C -o name,uuid,blkdevname,blkdevs_used,subsystem

dmsetup reports "nvme0n1p3" but subiquity expects "/dev/nvme0n1p3" (note the missing "/dev/")

I can't find any way to tell dmsetup to include the "/dev/" when reporting blkdes_used.

For three years it is not possible to properly use encrypted filesystems, which should be standard for decades.

[Claudio Matsuoka (cmatsuoka)]

This still happens in noble daily. I worked around it in a rather nasty way by injecting the key value and adding the missing "/dev" in curtin's block_meta, and after that installation continues as expected.

[Eugene San (eugenesan)]

Nearing 2nd LTS release after the bug was reported and still no solution or even acknowledgement of how serious the problem is.

To make it worse, in main desktop image they completely removed encryption option from manual disk partitioning screen.