Installer fails when sources are defined with an encrypted LVM partition

[Romeo]

Hello,

I am attempting to autoinstall Ubuntu 22.04 LTS onto a VirtualBox VM. The VM's BIOS boots GRUB via PXE. GRUB proceeds to load the initrd and kernel. The kernel gets loaded with the line:

linux /isos/ubuntu_22.04.3/vmlinuz ip=dhcp url=http://<lan_server>/ubuntu_22.04./ubuntu_server_22.04.3.iso autoinstall ds="nocloud-net;s=http://<lan_server>/ubuntu_22.04.3/cloud-init/" root=/dev/ram0 cloud-config-url=/dev/null net.ifnames=0 biosdevname=0

The vmlinuz and initrd were extracted from the Ubuntu ISO in the usual way. The autoinstaller then loads and proceeds.

Expected Behavior: The autoinstaller completes with both my custom repository and an encrypted root filesystem.
Actual Behavior: The autoinstaller crashes on "partitioning" if both of these options are configured.
Steps to reproduce: Attempt an autoinstall of 22.04 with both an encrypted root partition and a custom repository.

I have attached the user-data file that reproduces this issue. If you comment out the storage:layout:password line but leave the apt section, the installation will complete fine. If you comment out the apt section but leave the storage:layout:password line, the installation will complete fine. Commented out in this same file is my manual partitioning for an encrypted disk. I have left it in for completeness's sake, using the auto-provisioner settings instead for simplicity's sake in this bug report. If you use my manually-created disk partitioning, you will get the same crash as you will if you use the auto-generated one.

I can only attach one file to this bug report. I have a subiquity .crash file and a tar of the /var/log/installer/ directory to include in this bug report, which I will attach to replies.

[Romeo (romeo-b)]

Here is the Subiquity .crash file

[Romeo (romeo-b)]

Here is the tar of the /var/log/installer directory of the crashed autoinstall system.

[Romeo (romeo-b)]

I manually ran the command in the final error message, that is:
python3.10 -m curtin -vvv --showtrace -c /var/log/installer/curtin-install/subiquity-partitioning.conf install cp:///tmp/tmp8qst_wz0/mount
I didn't bother with all the environment variables or the log settings. Environment variables are already set in the shell that you get dropped into after the installer crashes. I redirected stdout and stderr to a file, and I have attached that file here. It appears that the partitioning actually works fine but when Subiquity attempts to do some apt work, it brings in the cdrom repo.

=======================================================================================================
Running in chroot, ignoring command 'start'
Get:1 file:/cdrom jammy InRelease
Ign:1 file:/cdrom jammy InRelease
Get:2 file:/cdrom jammy Release
Err:2 file:/cdrom jammy Release
  File not found - /cdrom/dists/jammy/Release (2: No such file or directory)
Hit:3 https://repo.saltproject.io/salt/py3/ubuntu/22.04/amd64/latest jammy InRelease
Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease
Get:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
Get:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB]
Reading package lists...
E: The repository 'file:/cdrom jammy Release' no longer has a Release file.
try 0: command ['apt-get', '--quiet', '--option=Acquire::Languages=none', '--option=Dir::Etc::sourcelist=/tmp/tmpbkroxt02/sources.list', '--option=Dir::Etc::sourceparts=/tmp/tmpbkroxt02/sources.list.d', 'update'] failed, rc: 100
Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/target', 'apt-get', '--quiet', '--option=Acquire::Languages=none', '--option=Dir::Etc::sourcelist=/tmp/tmpbkroxt02/sources.list', '--option=Dir::Etc::sourceparts=/tmp/tmpbkroxt02/sources.list.d', 'update'] with allowed return codes [0] (capture=False)
=======================================================================================================

Perhaps this is a red herring. Maybe it doesn't do this when an encrypted partition is defined, but I am unsure how I would go about testing that.

[Romeo (romeo-b)]

I tried just commenting out the section of subiquity/server/apt.py:

        write_file(
            self.install_tree.p("etc/apt/sources.list"),
            f"deb [check-date=no] file:///cdrom {codename} main restricted\n",
        )

And recreating the subiquity snap to patch it into the ISO so that I could see if this was actually the root of the problem. Sadly I was unable to install the snapcraft snap on my 23.04 system, it gives an error about python3 not being found even though it is available. I was able to get snapcraft installed on a 20.04 system and from there I was able to modify and build subiquity, but when I run the scripts/inject-subiquity-snap.sh script, the ISO that it produces is not net-bootable. I can boot it manually but from there I have no ability to trigger the cloud-init, I just get the graphical interactive setup. I could probably modify the grub.cfg in the ISO, but it is late and I've been at this all day.

Hopefully some Canonical developers with better understanding of the ISO creation and testing workflows can come along and take a look at this.

[Romeo (romeo-b)]

Today I managed to test my hunch. I booted the ISO in autoinstall mode with my custom subiquity.snap which has those three lines of code removed, and it was able to complete just fine. I expected that the installation would take much longer as a result, but it seemed to complete pretty quickly on my rural 12Mbps Internet connection. I'm not sure why, but I assume that means I messed something up. However, the server seems perfectly fine from a usability standpoint since my regular Salt States are able to complete without issue. If anyone is attempting to replicate my success, here are some instructions:

1. Make a new directory for the project, you will need it
2. git clone https://github.com/canonical/subiquity.git
3. sudo snap install --classic snapcraft # I had to do this step on a 20.04 system since it didn't work on my 23.04 system
4. lxc init --auto
5. Turn off your firewall if it is on
6. cd subiquity
7. make install_deps
8. Edit the file subiquity/server/apt.py and remove the section mentioned in my previous comment
9. wget https://cdimage.ubuntu.com/ubuntu-server/jammy/daily-live/current/jammy-live-server-amd64.iso # If you do not already have it
EDIT I forgot step 9.5 which is: snapcraft snap --output subiquity_test.snap
10. sudo ./scripts/inject-subiquity-snap.sh jammy-live-server-amd64.iso subiquity_test.snap ../custom.iso
You are almost there, now you just need to replace the GRUB
11. mkdir ../jammy
12. cd ../jammy
13. 7z -y x ../custom.iso -osource-files
14. cd source-files
15. mv '[BOOT]' ../BOOT
16. Replace the file boot/grub/grub.cfg with your regularly PXE-booted grub.cfg. Make sure to remove the portion of the "linux" line that reads "url=...", otherwise the kernel will attempt to get the ISO from that URL and completely ignore the ISO that you are booting from, making your subiquity changes moot.
17. xorriso -as mkisofs -r -V "Custom 22.04" -o ../../custom_subiquity_and_grub.iso --grub2-mbr ../BOOT/1-Boot-NoEmul.img -partition_offset 16 --mbr-force-bootable -append_partition 2 28732ac11ff8d211ba4b00a0c93ec93b ../BOOT/2-Boot-NoEmul.img -appended_part_as_gpt -iso_mbr_part_type a2a0d0ebe5b9334487c068b6b72699c7 -c '/boot.catalog' -b '/boot/grub/i386-pc/eltorito.img' -no-emul-boot -boot-load-size 4 -boot-info-table --grub2-boot-info -eltorito-alt-boot -e '--interval:appended_partition_2:::' -no-emul-boot .

This is not a final solution, but it does prove the point that the issue comes from attempting to access the cdrom when it is not available in the chroot. I am unsure why this happens only when a luks partition is configured. This new ISO will not boot over the network, which is sad. I think a better patchwork solution would be to just run commands in late-commands to add the repo, add the key, and run an apt update and apt install.

[Romeo (romeo-b)]

A better solution that works with netbooting and doesn't require rebuilding the ISO, add these to late-commands:
- echo "deb [arch=amd64] https://repo.saltproject.io/salt/py3/ubuntu/22.04/amd64/latest/ jammy main" > /target/etc/apt/sources.list.d/salt.list
- curl https://repo.saltproject.io/salt/py3/ubuntu/22.04/amd64/SALT-PROJECT-GPG-PUBKEY-2023.pub > /target/etc/apt/trusted.gpg.d/salt.asc
- cp /etc/resolv.conf /target/etc/resolv.conf # Need DNS in chroot. This gets overwritten by systemd at any opportunity anyways
- chroot /target apt update
- chroot /target apt install -y salt-minion

Of course, alter the commands for whatever repos and packages you need.