curtin should support making installed systems BIOS+UEFI hybrid bootable

[Steve Langasek]

Since 18.04, Ubuntu Desktop systems by default come with both the shim-signed and grub-pc packages installed, which (with proper partitions setup) is sufficient to make the system bootable whether the firmware is booting in BIOS or UEFI mode.

Curtin should also support this for the subiquity case.

[Ryan Harper (raharper)]

IIUC

"proper partition setup" for callers of curtin is:

/boot/EFI partition *and* a grub_bios partition.

Something like:
    config:
      - id: main_disk
        type: disk
        ptable: gpt
        serial: disk-a
        name: main_disk
        wipe: superblock
        grub_device: true
      - id: bios_boot
        type: partition
        size: 1MB
        number: 1
      - device: main_disk
        flag: boot
        id: main_disk_part2
        number: 2
        size: 512M
        type: partition
        wipe: superblock

The curtin part would be to ensure that shim-signed *and* grub-pc packages are installed into target.

[Dimitri John Ledkov (xnox)]

Hm, imho this needs more. One has to run and install grub on one or more devices.

[Ryan Harper (raharper)]

Hi Steve, Dmitri,

Do you have more details on the grub process. Curtin already runs grub on one or more devices as needed. Are there any specific parameters or settings or will the grub install process detect hybrid configuration?

Are there any restrictions on how the partitions are ordered or numbered?

[Steve Langasek (vorlon)]

I don't know what Dimitri's comment refers to. As far as I'm concerned this is already well-defined, and you're correct that the curtin piece is simply enforcing that both shim-signed and grub-pc get installed (and the grub install to disk is triggered for both).

[Dan Watkins (oddbloke)]

Steve, does "the grub install to disk is triggered for both" mean we should be invoking grub-install twice (once for each target), or do we have another mechanism that we can hook into to perform the install for all (available|applicable|...) grub targets?

[Steve Langasek (vorlon)]

On Wed, Apr 08, 2020 at 05:39:32PM -0000, Dan Watkins wrote:
> Steve, does "the grub install to disk is triggered for both" mean we
> should be invoking grub-install twice (once for each target), or do we
> have another mechanism that we can hook into to perform the install for
> all (available|applicable|...) grub targets?

There is no other mechanism; the installer needs to invoke grub-install for
each target. Once installed, the shim-signed + grub-pc packages can take
care of keeping it updated.

[Ryan Harper (raharper)]

We will need two formats for the hybrid mode, ptable: gpt and ptable: dos

For gpt, the config from comment #1 applies, we'll have both bios_grub and an efi partition (possibly a secondary for resilient boot).

For dos, we'll have:

    config:
      - id: main_disk
        type: disk
        ptable: dos
        serial: disk-a
        name: main_disk
        wipe: superblock
      - device: main_disk
        id: main_disk_part1
        number: 1
        size: 10G
        type: partition
        grub_device: true
        flag: boot
      - device: main_disk
        id: main_disk_part2
        number: 2
        size: 512M
        type: partition
        grub_device: true
        flag: esp
        wipe: superblock

Curtin currently uses 'boot' to mean toggle the DOS boot flag so we'll need to
do something to trigger marking this. I've suggested 'esp' here to
map to the 0xef code and we'll need to see if parted will let us do this, or
use sfdisk or something else to directly toggle the type.

[Steve Langasek (vorlon)]

On Fri, May 22, 2020 at 04:09:51PM -0000, Ryan Harper wrote:
> We will need two formats for the hybrid mode, ptable: gpt and ptable:
> dos

I don't see why this should be the case. Are we not emitting GPT everywhere
today, even when the system is booting in BIOS mode? Late-model BIOSes do
support booting from GPT; I would think at this point anything that doesn't
support GPT is too old to be of practical concern.

So a GPT table with a bios_grub partition and an EFI System Partition should
suffice.

> For gpt, the config from comment #1 applies, we'll have both bios_grub
> and an efi partition (possibly a secondary for resilient boot).

Resilient boot is only applicable in the case of multiple disks; and then
each disk in the root set should have exactly one ESP and exactly one
bios_grub partition.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer https://www.debian.org/
<email address hidden> <email address hidden>

[Ryan Harper (raharper)]

> On Fri, May 22, 2020 at 04:09:51PM -0000, Ryan Harper wrote:
> > We will need two formats for the hybrid mode, ptable: gpt and ptable:
> > dos
>
> I don't see why this should be the case. Are we not emitting GPT everywhere
> today, even when the system is booting in BIOS mode? Late-model BIOSes do
> support booting from GPT; I would think at this point anything that doesn't
> support GPT is too old to be of practical concern.
>
> So a GPT table with a bios_grub partition and an EFI System Partition should
> suffice.

Curtin has to deal with either construct; The design of the storage config is
from MAAS, Subiquity, user-defined autoconf install, Field Engineer creating a
one-off, etc. What I'm saying here is that curtin should validate that
one can create a hybrid partition layout with either GPT and MSDOS. Note,
I first saw such a dos-based hybrid looking at the partition layout of our
own installer iso. =)

"/dev/sdc": {
    "ID_FS_LABEL": "Ubuntu-Server_18.04.3_LTS_amd64",
    "ID_FS_UUID_ENC": "2019-08-05-20-00-00-00",
    "ID_FS_LABEL_ENC": "Ubuntu-Server\\x2018.04.3\\x20LTS\\x20amd64",
    "ID_BUS": "usb",
    "ID_FS_BOOT_SYSTEM_ID": "EL\\x20TORITO\\x20SPECIFICATION",
    "ID_FS_TYPE": "iso9660",
    "DEVNAME": "/dev/sdc",
    "ID_PART_TABLE_TYPE": "dos",
    "partitiontable": {
        "label": "dos",
        "id": "0x2f10bd40",
        "unit": "sectors",
        "partitions": [
            {
                "bootable": true,
                "size": 1736704,
                "type": "0",
                "node": "/dev/sdc1",
                "start": 0
            },
            {
                "size": 4928,
                "type": "ef",
                "node": "/dev/sdc2",
                "start": 1295064
            }
        ],
        "device": "/dev/sdc"
    },
},

>
> > For gpt, the config from comment #1 applies, we'll have both bios_grub
> > and an efi partition (possibly a secondary for resilient boot).
>
> Resilient boot is only applicable in the case of multiple disks; and then

Yes

> each disk in the root set should have exactly one ESP and exactly one
> bios_grub partition.

Yes. We currently are testing UEFI with GPT layout only; with hybrid we
need to test that resilient boot works with the MSDOS hybrid layout as well.

[Ryan Harper (raharper)]

Lastly, much of the focus on DOS partition tables comes from subiquity's re-use
existing partitions; it's still extremely common to have a dos partition table
rather than gpt.