support installing multiple ESPs

[Michael Hudson-Doyle]

This is not going to be especially easy (will require curtin changes) but should be done anyway.

[Tom Reynolds (tomreyn)]

Is this about supporting ESP on hardware or firmware/fake RAID then? Since it could not work on software RAID, I guess (the UEFI would be unable to read it, at least in a reliable way).

I think the right solution for those of us who like to have all partitions mirrored via software RAID ARRAYs is to have an ESP (type) partition of the same size (and maybe also at the same position) on both the first and second system disk, and some form of automation to regularly (on shutdown? or on boot?) check the active ESP for changes and to cp -p from the primary to the second file system whenever this is the case. Basically a failover / hot standby configuration.

Ideally (I can dream) this would be something which is configurable on the installer.

[Michael Hudson-Doyle (mwhudson)]

Well, you *can* put ESP on software raid, if you use a md metadata version that puts the metadata at the end so that the ESP still looks like a regular vfat partition to the firmware, e.g. https://outflux.net/blog/archives/2018/04/19/uefi-booting-and-raid1/ -- but that does seem more complicated than I was expecting because I didn't realize that the firmware writes to the ESP under some circumstances!

[Michael Hudson-Doyle (mwhudson)]

This bug was filed as "support ESP on raid" but that's not really possible. What we will do is support installing (and updating) multiple ESPs, which will achieve the goal of allowing an install that can survive the total failure of a disk.

[Tom Reynolds (tomreyn)]

Related: bug 1466150 discusses how ESP on RAID is a non supportable configuration, grub failing to install in this scenario

[Johan Ehnberg (johan-ehnberg)]

We are using this type of boot redundancy in production with a simple addon script in Bionic. The installation is done with two non-RAID ESP. It would be great to have it as an officially supported feature, raid or not.

Here is the script for reference (BOOTMIRROR and BOOTMIRRORD are set by our orchestration, typically as /dev/nvme1n1p1 and /dev/nvme1n1):
    mkdir -p /boot/efibackup
    parted $BOOTMIRRORD set 1 esp on
    parted $BOOTMIRRORD set 1 boot on
    mkfs.vfat -F32 -n "EFI System Partition" $BOOTMIRROR
    echo "UUID=`blkid -s UUID -o value $BOOTMIRROR` /boot/efibackup vfat umask=0077 0 0" >> /etc/fstab
    mount /boot/efibackup
    echo -e '#!/bin/bash\nsudo rsync -aC /boot/efi/ /boot/efibackup' > /etc/cron.daily/copyefi
    chmod 755 /etc/cron.daily/copyefi
    /etc/cron.daily/copyefi

[Michael Hudson-Doyle (mwhudson)]

It's very very late but this is landing for focal.

[Tony Middleton (ximera)]

Do you know whether this will be included in Debian 10?

[Tony Middleton (ximera)]

Thank you for this change. When applying an update to a 20.04 system I was given the option of specifying two EFI partitions. However when I did I ended up with an unbootable system, just went into the grub command line.

I will raise a bug for this but it may be a little while as I need to recreate the problem in more controlled conditions and I'm a bit busy at the moment.

[Michael Hudson-Doyle (mwhudson)]

The work that has been done so far is Ubuntu specific, Debian are welcome to take it too but I don't know of any plans to do so.

Sorry to hear it didn't work, if you can reproduce the situation and manage to fish the contents of /var/log/installer from the installed system that would be great. I have only been able to test this in VMs so far (thanks to lockdown!).

[Tony Middleton (ximera)]

What I have now is an Ubuntu Mini test system with root, including /boot, on a raid1 array over two disks and 2 EFI partitions mounted at /boot/efi and /boot/eficopy.

When I run "dpkg-reconfigure grub-efi-amd64" the displays indicate that grub-install is run for both of the EFI partitions. The contents of the partitions look OK as do the EFI entries shown by efibootmgr. The system boots OK from /boot/efi.

However when booting from /boot/eficopy the system just drops into the grub command line, with no apparent error messages. I'm happy to bug this but not sure what diagnostics would help as this would not be during the install process.

[Tony Middleton (ximera)]

Apologies but I'm not sure what is happening here. The next day my machine wouldn't boot from either entry - I must have changed something but I'm not sure what. However, I purged and reinstalled all the grub packages and now everything seems to be fine. I'll have to watch out for the next grub update.

[Johan Ehnberg (johan-ehnberg)]

Does the release fix ensure the ESP contents are in sync? I was unable to find any indication of such in the code nor in a test install.

[Michael Hudson-Doyle (mwhudson)]

The code to do that is in the grub-efi-amd64's maintainer scripts.

[Johan Ehnberg (johan-ehnberg)]

The installer properly populates both ESP's, and I can confirm I get a bootable system after yanking the first disk with just the installer's efforts. Brilliant!

However, after that grub-install is only populating the one mounted in /boot/efi so I will turn to those maintainers to check if it makes sense to mirror the feature there:

https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1876974

[Johan Ehnberg (johan-ehnberg)]

This is how ours is set up to create two ESP's:

    config:
    - {type: disk, ptable: gpt, wwn: DISKA, wipe: superblock, preserve: false, grub_device: false, id: disk-1}
    - {type: disk, ptable: gpt, wwn: DISKB, wipe: superblock, preserve: false, grub_device: false, id: disk-2}
    - {type: partition, device: disk-1, size: 134217728, wipe: superblock, flag: boot, grub_device: true, number: 1, preserve: false, id: partition-efi-1}
    - {type: partition, device: disk-2, size: 134217728, wipe: superblock, flag: boot, grub_device: true, number: 1, preserve: false, id: partition-efi-2}
<other partitions>
    - {type: format, fstype: fat32, volume: partition-efi-1, preserve: false, id: format-efi-1}
    - {type: format, fstype: fat32, volume: partition-efi-2, preserve: false, id: format-efi-2}
<other filesystems>
    - {type: mount, device: format-efi-1, path: /boot/efi, id: mount-efi-1}
    - {type: mount, device: format-efi-2, path: /boot/efibackup, id: mount-efi-2}
<other mounts>