Disk Setup / Manual installation missing LVM LV as target

[Steffen Neumann]

Hi,

I am trying 24.04 installation from noble-desktop-amd64.iso (2024-03-20 10:14)
with ubuntu-desktop-bootstrap 0+git.1314d484d

My laptop has an existing (non-encrypted) LVM setup, where I created an LV for noble from the existing jammy installation.

I boot into the noble-desktop-amd64.iso, get to screen "Disk Setup", select "Manual installation"
and I could select the /dev/sda* partitions for installation.

I *can not* select the logical volume, it is simply not showing up.

I can manually list LVs and mount /dev/mapper/lvmssd-noble from the terminal.

Expected behaviour:
Next to sd* partitions, I can select the logical volume /dev/mapper/lvmssd-noble as target.

Yours,
Steffen

(noble-desktop-amd64.iso sha256 55f7de9571f2a85ac8c3c9bb553862449f40e1275d41c7646cd83ec26db24bd2)

[john (johnsmith25)]

I've also installed this way for as long as I can remember.

Please can someone confirm ASAP if this is an intentional change or not.

[Juan Andrés Ghigliazza (tizone)]

Same here.

[john (johnsmith25)]

How is this a wishlist item? Its a complete install blocker...

[Juan Andrés Ghigliazza (tizone)]

In the meantime, it would be great if someone could provide a workaround for the problem.

[Mohegan (jack-mohegan)]

Same here. Can't switch to Noble without erase my disks !

Very important bug !!!

[Graeme (graeme-riddick)]

Just sat down to upgrade my system too and same problem for me. Agree with John, it's a blocker.
Cant install do a /dev/mapper/xxx
Unless there is a way to force it in the YAML file?

[Omar AKHAM (akhamomar)]

Same problem for me. It seems that's an old issue : https://answers.launchpad.net/ubuntu/+source/ubiquity/+question/706331

[Adam Dyess (addyess)]

Yup, this one should be higher than a wishlist. At least with Minotaur, there was the legacy installer option.

to be clear, i want to use the installer to select my LVM partition for installation. This has been available for years, and i hate to see it go. Please bring it back

[Mohegan (jack-mohegan)]

I have a bad solution but it works.
I installed the ubuntu 23.10 legacy version (from here : https://cdimage.ubuntu.com/ubuntu/releases/23.10.1/release/). It can detect the LVM volumes.

After that, I've forced the update version with this command :

sudo do-release-upgrade -d

[Thomas Ward (teward)]

This needs to be increased in severity, in my opinion. It is otherwise **very difficult** to set up DISA STIG compliant setups if we can't manipulate LVMs in the manual installer.

[Anderson Luiz Alves (alacn1)]

It should at least show already created lvm partitions, and other dev/mapper partitions, and allow set it as install target.

[DiagonalArg (diagonalarg)]

@teward / That would make this bug a security issue, wouldn't it? This should have /much/ higher priority. For me, it's a deal breaker.

@graeme-riddick / any ideas on the yaml front? I'm wondering if I should try to dig into that.

Edit: Wow, they've got a lot of big issues. We'll be waiting ...
https://bugs.launchpad.net/ubuntu-desktop-provision/+bug/2048473
https://bugs.launchpad.net/ubuntu-desktop-provision/+bug/1881932

Edit2: These two appear to be the same. They are specifically about LUKS, but in both cases we're talking about /dev/mapper/* not being recognized:

Ubuntu 24.04 (20240409) unable to install on preexisting LUKS partions
https://bugs.launchpad.net/ubuntu-desktop-provision/+bug/2060678

Subiquity Manual Installation, 24.04 / Can't find LUKS targets
https://bugs.launchpad.net/ubuntu-desktop-provision/+bug/2064193

[Andrew H (andyvorld)]

I was able to work around the UI not showing LUKS volumes with using the autoinstall.yaml configs,
https://github.com/asayler/ubuntu_autoinstall/blob/master/user-data-encrypted; with everything but the storage section stripped out. This however will format and clear out the entire drive so its not usable for prexisting partitions with data on them.

Playing around with the yaml files more lead to this https://pastebin.com/2HGrWMYF, in a UEFI setup the auto install requires a grub partition and disk fully defined. For the rest of the data partition like / and /boot we can use a prexisting "type: device" (https://curtin.readthedocs.io/en/latest/topics/storage.html) to provide the block device, and using the "preserve: true" should keep the onboard data intact. Only issue with this is that the newly install os does not have cryptsetup in the initramfs so it needs to be manually installed before rebooting.

[Graeme (graeme-riddick)]

Thanks @Andrew H ! Confirm that this yaml code works perfectly for the install.
A apt install cryptsetup-initramfs in the chrooted environment was indeed all it took after the install and before reboot.
Pity that it took me quite a few hours more to install this time around!

[morph027 (morph027)]

Thanks, just tested inside a VM and it looks good. Only thing is i have to add the precise numbers of the existing partitions, which is annoying but manageable.

For the cryptsetup-initramfs, i also need to add the device to crypttab.

I'm using the late-command w/ a shell script like this https://pastebin.com/0CT774ry

  late-commands:
    - 'curtin in-target --target=/target -- sh -c "wget -qO - https://pastebin.com/raw/0CT774ry | bash"'

: https://canonical-subiquity.readthedocs-hosted.com/en/latest/reference/autoinstall-reference.html#late-commands

[morph027 (morph027)]

Sidenote for autoinstall, all partitions not known to curtin within the yaml will get dropped, so make sure to add dummy partitions with 'preserve: true' for each existing partition and their size (e.g. windows dual boot partitions). Ask me how i know 😉 Good thing i'm testing out in a VM.

Example attached one partition not to be used for the installation is 26.5G. While 100G within the yaml works, for 26.5G i needed to specify the proper bytes or the installer will crash and complain about a size mismatch.

[Mike Ferreira (mafoelffen)]

This is sort of the same as 'part of' my bug report: https://bugs.launchpad.net/ubuntu-desktop-provision/+bug/2065236

But my write-up of that is that there is a bigger problem with the new partitioner.

It's not just NOT recognizing LVM2 structures. It's also not recognizing mdadm, and LUKS containers, etc. So there is a bigger problem where it is not recognizing many volume managers, file systems types, and storage containers, that it used to be able to with the older partitioner.

I will link this bug to mine, (as a mention) since this 'IS' older... but there is still a bigger problem that needs to be addressed with this.

The autoinstall example buried deeply within the snap image work, as an alternate method. They are undocumented.

Eevn though they work as a workaround, they don't address the problem with the new partioner not being able to recognize pre-existing storage containers. That was something that Canonical promised that they would improve and fix when they announced choosing the new partitioner, back in the Lunar DEV-Cycle. So far none of that has transpired.

[Piotr Morgwai Kotarbiński (morgwai)]

If this is a part of some joint apple+m$ conspiracy to move ppl away from linux, then they are doing a really good job ;-) Jokes aside, this bug makes it extremely difficult to use custom partitioning in a secure way, which is a very common usecase (reasons include things like a fresh install while preserving /home, encrypted swap partition for secure hibernate etc). Therefore marking this as "whishlist" and "low importance" is a slap in our faces. At this point I'm really done with ubuntu's nonsense: I'm moving to another distro until this is fixed and will be also encouraging all organizations I cooperate with to do the same...

[Vadim Nevorotin (malamut)]

Low importance? Is it a joke? Ubuntu LTS is fully non-installable - how can it be low priority? The main basic feature of installer - install to existing partitions (whatever LUKS, LVM, ZFS or any other block devices supported in Linux) - is completely missing!

Ability to select existing device to install to is a absolutely mandatory feature, without it it's not an installer at all!

[sva42c8 (sva42c8)]

Have been assigned a (yearly) ticket to workout
a migration path again.
But I fail miserably to complete the task again,
but still company allocates every year 3days for research :-).
So here I am:

Encrypted Ubuntu 24.04 side by side to existing
encrypted Windows is not possible.
If it works out( encryption+hibernate), windows partition could be dropped and ubuntu partition can be further enlarged.

Additional encrypted Windows is easy and all requirements below are passed

This bug hasn't hit my research in 22.04,
 so we wait for a working dist-upgrade in August.

Our must have requirements for Desktops and Laptops:
- Bitlocker/Veracrypt/Filevault FDE
- SecureBoot
- hibernation support with enabled encryption.
- Docking support and hotplug and restore window positions
- Resume Hibernation with secure boot like in Windows, Docked and Undocked.

The last point secureboot+hibernate will be a big bumper,
 but that is another problem for next year's reoccuring
ticket(Lockdown patch)

[David DIDIER (ddidier)]

Wishlist and low priority... You fools! I'm seriously considering to install another distribution even if I'm a very long time user of Ubuntu. And this is clearly a security issue.

[Bob McChesney (bmcchesney)]

I was hit with this problem trying to install Ubuntu 24.04 alongside an existing Linux Mint deployment tonight. Annoying but it looks like some people have had success with using autoinstall.yaml.

Perhaps someone could please provide a bit of assistance for me and my existing configuration:

https://pastebin.com/izL3bP1L (Using pastebin as output was illegible here)

What should I be doing to have a safe/sane autoinstall.yaml for my scenario where I want:

* All partitions to be left as is (not reformatted) except the new vgsystem-ubroot
* UEFI bootloader to go in nvme0n1p1
* Grub, initfs, kernel etc to go in nvme0n1p5
* Ubuntu to be installed in vgsystem-ubroot at "/"
* New user to be created in vgsystem-home at "/home/", leaving existing files

I've had a few tries in a VM but I'm getting it wrong every time, so if someone has the required insight, please help.

[Matt Borgerson (mborgerson)]

I also wanted to preserve some existing partitions (namely EFI/Windows) while replacing an older Ubuntu 22.04 LUKS setup. Since I could not find a way to get the installer to recognize the unlocked partition, I just installed to an unencrypted partition and migrated the installed filesystem into a new encrypted partition.

Basically: launch the installer, before continuing in the installer use the Disks utility to replace your encrypted partition with an EXT4 filesystem, select it as the root mount point during manual setup and continue installation normally.

Reboot to confirm installation success.

Reboot into the live installer once again. Mount the install partition and copy the installed filesystem contents out to /tmp/rootfs (via rsync -aXHAS /mnt /tmp/rootfs). Then unmount and use the Disks utility again to create your preferred encrypted partition setup, mount it and copy the filesystem contents back in. Finally, chroot into the filesystem, install cryptsetup, create an /etc/crypttab, update /etc/fstab, then run update-initramfs and update-grub. Once rebooting you should have a new Ubuntu 24.04 install on an encrypted partition along side your other, untouched partitions on the disk.

Hope this helps.

[Bob McChesney (bmcchesney)]

Matt, your suggestion sounds good but unfortunately I have (perfectly reasonably IMO) consumed all of the space after EFI, Windows, and /boot with a LUKS LVM partition, so that's only where I can put new volumes. I think the autoinstall.yaml one-shot is what I need.

[DiagonalArg (diagonalarg)]

@bmcchesney / Just do a pure vanilla install to a USB stick, and copy the files from there. It works well.

https://bugs.launchpad.net/ubuntu-desktop-provision/+bug/2060678/comments/4

[Volkan Ozdemir (volkan-ozdemir)]

Severity of this bug should be increased as this is a security issue. Custom partitions & cryptsetup is definitely necessary for high security environments.

[Bob McChesney (bmcchesney)]

I beg to differ. It is definitely a convenience issue but it's not a security issue. Let's not call everything a security issue just because it affects setups with encryption.

[David Lou (duckweed)]

Managed to do an installation following the steps in the https://gist.github.com/jrandiny/4b60d64b6fe598337a86064578e2b3cd to generate a autoinstall.yaml.

Most helpful workaround I see so far.

[Seb Bonnard (sebma)]

@addyess I have tried the Ubuntu 23.10.1 GUI installer and it does see not my LVM structures on /dev/sda2 :

ubuntu@ubuntu:~$ sudo pvs
  PV VG Fmt Attr PSize PFree
  /dev/sda2 VG_ALL lvm2 a-- 237.47g 112.47g
ubuntu@ubuntu:~$ sudo lvs
  LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
  home VG_ALL -wi-a----- 100.00g
  rootFS VG_ALL -wi-a----- 25.00g
ubuntu@ubuntu:~$ xprop _NET_WM_PID
_NET_WM_PID(CARDINAL) = 13799
ubuntu@ubuntu:~$ ps -fp 13799
UID PID PPID C STIME TTY TIME CMD
ubuntu 13799 13763 34 14:41 ? 00:00:04 /snap/ubuntu-desktop-installer/1267/bin/ubuntu_desktop_installer
ubuntu@ubuntu:~$

What is the Ubuntu Minotaur legacy installer and how do I launch it ?

Related : https://bugs.launchpad.net/ubuntu-desktop-provision/+bug/2065236

[Jake Anderson (jake-vapourforge)]

Not even being able to use mdraid is certainly a step back

[Vincent Danjean (vdanjean)]

I'm in charge of Linux Install Party at my University. I discovered this bug during the first session of the LIP three days ago. I would say I cannot imagine how an linux installer can be released without any support for LVM2/Crypt/...

Nevertheless, the LIP had to be done. It was too late to switch to another distribution (but we will probably switch to Debian if this bug is not fixed next year).
For this year, as we want our student to use LVM2 for their dualboot (it is usually very difficult for them to correctly estimate the size that must be dedicated to Linux wrt windows), I wrote a (small) script so that:
- the students install Ubuntu 24.04 (from the 'try ubuntu' mode in order to be able to continue this receipt) in a (little, about 20GiB) unique plain partition
- but before rebooting:
  - they create a PV (gparted is available in the live session)
  - they download and run my script:
    wget https://gricad-gitlab.univ-grenoble-alpes.fr/polytech/lip/scripts/-/raw/master/add-lvm2-to-ubuntu
    sudo bash add-lvm2-to-ubuntu --pv /dev/nvme0n1pX

By default, the script creates a VG and three LVs (system 30GiB, home 10GiB and swap 4GiB), it recopies the /target content to the system and home LVs, fix /etc/fstab and grub, and add the /target partition back into the VG (as another PV). So that, after the first reboot, we have a Ubuntu booting within LVM2.

Feel free to download my script if your are interested.
I will perhaps add support to encrypt the PVs. Not sure however, I will perhaps use my time to switch the LIP documentation to install Debian instead of Ubuntu.

Regards,
  Vincent

[Modestas (montvid)]

Ubuntu had LUKS support when installing the OS and after an incident I needed to reinstall the ubuntu 24 OS but leave the data intact but the new installer DOES NOT HAVE the manual option to install into LUKS LVM partition. But Pop OS 24 does!!! IT IS A MAJOR BLOCKER for recovery/reinstall of UBUNTU 24. I would like to say a lot of bad words for the botched new installer and the major fail on the devs part assigning this as WISHLIST?!?!?!?!?!?!?!?

I found a resolution to this issue by:

1. Install Pop OS alpha 24 which uses the old installer and sees the LUKS drive and can install into it without loosing ones data.

2. Be happy Pop OS user instead of Ubuntu.

[Roberto Leinardi (leinardi)]

A workaround is to install 22.04, that still has support for LUKS during the installation, and then upgrade to 24.04. But there should really be an official way to have LUKS support during the installation of the latest Ubuntu, this upgrade solution is very time consuming and it will just get worse with every new release.

[paradoxheart (paradoxheart)]

The approach I took to working around this issue was to install to a small temporary partition that was just big enough to hold the OS. Then I created my LUKS LVM setup in the rest of the disk, and used rsync, preserving absolutely everything (the -a flag does not include everything), to copy the OS across to the lvm. Then you can chroot into the lvm partion, install the missing packages that lvm and luks require, and locate and update every UUID reference in the system to the new disk, and rebuild grub. Then when all of this is done, you can delete the temporary partition, and resize your lvm to consume the remaining disk space. Honestly though, this "Arch Linux on hardmode" type install is fit only for a one off, or when you're duplicating disks, and has plenty of pitfalls to catch you out and brick your computer, so I wouldn't recommend it.

[Mahiar Mody (mahiarmody)]

Please UPGRADE the importance of this bug from "low" to "CRITICAL". This type of manual partitioning is not needed merely for dual booting with Windows, but also when you one would like to custom install / (root), /home, SWAP, /boot, /boot/efi, etc. partitions on separate disks.

Such an install is necessary when for instance if you have a small SSD disk and a larger SATA disk. In this scenario, it would be impossible to install the whole of Ubuntu merely on the SSD (owing to low capacity), and it would not make sense to ignore the SSD and install Ubuntu only in the SATA, ignoring the speed benefits of the SSD.

Having the flexibility to encrypt the entire SSD using LUKS2, and the SATA using LUKS2, followed by LVM allows one to leverage both disks and lets individuals decide which disk (and which partition in each disk) should be used for the /, /home, SWAP, /boot, /boot/efi, etc.

The Ubuntu installer MUST therefor allow such partitions to be selected and set during a manual install. The previous LTS release viz. Ubuntu 22.04 had no issues with this. In my humble opinion, such an important feature MUST be supported by a long-term-support release.

I look forward to seeing this feature enabled once again in Ubuntu 24.02.2 installer.

[SergeiS (sergei-redleafsoft)]

I was previously manually creating the mdadm devices, setting up LUKS and formatting ext4 with correct stride and stripe-width params to have an optimal result. None of that is possible with the current installer. It recognizes an existing LUKS partition but doesn't allow to use it in any way. I ended up installing with LVM and LUKS on top of that, then rebooting into a rescue distro and recreating all of the partitions my way. THIS IS TERRIBLE!

[Fernando del Valle (fernandodelvallearg)]

I couldn't believe it, until I found this bug. I have 4, 5 distributions, always in a different LV so I can remove them easily. Goodbye Ubuntu!

[DjznBR (djzn-br)]

Came here to say that... I was experimenting on a Virtual Machine over Virtual Box and created a simple LVM scheme with cfdisk. EFI partition, /boot partition, LVM partition. Then I created the PV, VG and three LVs, for root partition, /home and a swap partition. When I "spun" the installer, the new Flutter installer, from 25.04, I was shocked the installer could not understand what LVs are... doesn't even get listed. So this is a major blocker I suppose, for many folks!

[Thomas Horsten (thomas-horsten)]

This is a terrible regression/bug. It prevents people from using the most secure installation type. Why is it "Wishlist" with "Low" priority?

I wrote a step-by-step guide to manually install it using a temporary non-encrypted installation and then migrating it to a LUKS+LVM setup here: https://thomashorsten.substack.com/p/installing-ubuntu-2404-with-lukslvm