Improve documentation around initrdless boot

[Dimitri John Ledkov]

[Impact]
(sort of prerequisite of bug 1929860 to avoid having to edit the patches and possibly introduce bugs that way, because these changes port the service to systemd unit)

 * Sometimes it is not obvious why initrdless boot is generated; when is it attempted; and when it fails and boots with initrd.

 * Improve UX by leaving breadcrumbs and emit messages about initrdless boot, in more places

[Test Case]

 * Boot system with initrdless boot
   - for example minimal cloud image from https://cloud-images.ubuntu.com/minimal/releases/
   - boot without graphical terminal, i.e. -nographics qemu option
 * Upgrade to new grub package
 * Uncomment GRUB_TERMINAL=console in /etc/default/grub

 1. Call `sudo update-grub`, a message should be emitted that initrdless boot will be attempted.

 2. Inspecting /boot/grub/grub.cfg, there are comments that initrdless boot will be attempted

 * Reboot

 * Observe the following:

 3. Inspecting console log, there are echo message from grub about attempting initrdless boot.

 * Install linux-generic kernel flavour; and remove kvm kernel flavour, reboot.

 * Observe in the console log

 1. There should be message about attempting initrdless boot
 2. kernel panic
 3. message about initrdless boot failing and attempting to boot with initrd

[Regression Potential]

 * This change adds a few more user visible message, without translations. The actual behaviour that is happening is unchanged. This simply better notifies users about what is happening, and can be inspected interactively, statically, and from boot logs.

[Other Info]

 * CPC requested these UX improvements, as there have been concerns that users are failing to discover why initrdless boot is happening; or failing; and how to effect it.

[Dimitri John Ledkov (xnox)]

                             GNU GRUB version 2.04

 ┌────────────────────────────────────────────────────────────────────────────┐
 │*Ubuntu │
 │ Advanced options for Ubuntu │
 │ Ubuntu 20.10 (20.10) (on /dev/sda1) │
 │ Advanced options for Ubuntu 20.10 (20.10) (on /dev/sda1) │
 │ │
 │ │
 │ │
 │ │
 │ │
 │ │
 │ │
 │ │
 │ │
 └────────────────────────────────────────────────────────────────────────────┘

      Use the ↑ and ↓ keys to select which entry is highlighted.
      Press enter to boot the selected OS, `e' to edit the commands
      before booting or `c' for a command-line.
   The highlighted entry will be executed automatically in 0s.

GRUB_FORCE_PARTUUID set, attempting initrdless boot.
Linux version 5.8.0-1008-kvm (buildd@lcy01-amd64-016) (gcc (Ubuntu 10.2.0-13ubuntu1) 10.2.0, GNU ld (GNU)
Command line: BOOT_IMAGE=/boot/vmlinuz-5.8.0-1008-kvm root=PARTUUID=f6bea054-a6e1-4734-91e2-eb6425be905b1
x86/fpu: x87 FPU will use FXSAVE
BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
BIOS-e820: [mem 0x0000000000100000-0x000000007ffdffff] usable

So it is not obvious if grub output will be visible in the console log, because GRUB_TERMINAL=console might not be set.

Also, previously the first message from the boot was from kernel. But my patch introduced 'GRUB_FORCE_PARTUUID' option.

Also, somehow on the minimal cloud image 'Ubuntu 20.10' was detected on /dev/sda1 - aka the only Ubuntu that there is..... and somehow I'm getting 10s grub menu prompt.

It almost feels like maybe kernel should be emitting dmesg message that it performing initrdless boot. rather that just:

EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
VFS: Mounted root (ext4 filesystem) readonly on device 8:1.
Run /sbin/init as init process

[Dimitri John Ledkov (xnox)]

GRUB_FORCE_PARTUUID set, attempting initrdless boot.
[ 0.000000] Linux version 5.8.0-25-generic (buildd@lcy01-amd64-022) (gcc (Ubuntu 10.2.0-13ubuntu1) 10)
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.8.0-25-generic root=PARTUUID=f6bea054-a6e1-4734-1
[ 0.000000] KERNEL supported cpus:

(panic)

(grub again)

GRUB_FORCE_PARTUUID set, initrdless boot failed. Attempting with initrd.
[ 0.000000] Linux version 5.8.0-25-generic (buildd@lcy01-amd64-022) (gcc (Ubuntu 10.2.0-13ubuntu1) 10)
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.8.0-25-generic root=PARTUUID=f6bea054-a6e1-4734-0
[ 0.000000] KERNEL supported cpus:

second boot

[Dimitri John Ledkov (xnox)]

I'm not quite sure this is discoverable enough. I kind of wonder if kernel could say if it is booting with or without initrd; before it panics about unable to find rootfs or `/sbin/init`.

And I wonder if our grub-common.service should actually log something to console & journal.

I.e. the fact that GRUB_FORCE_UUID was set, and yet booting without initrd failed. And maybe like unset GRUB_FORCE_UUID? or add something in grubenv to not attempt initrdless boot the next time?

Ideally i'd like to warn about failure to boot initrdless; and maybe not attempt it again, unless manually cleared?

[Dimitri John Ledkov (xnox)]

i think we should print warnings when we see initrdless_boot_fallback_triggered=1 in the grubenv.

[Dimitri John Ledkov (xnox)]

Oct 27 12:58:07 cloudimg systemd[1]: Starting Record successful boot for GRUB...
Oct 27 12:58:07 cloudimg sh[697]: grub: GRUB_FORCE_PARTU
UID set, initrdless boot paniced, fallback triggered.
Oct 27 12:58:07 cloudimg systemd[1]: Started Record successful boot for GRUB.

[Dimitri John Ledkov (xnox)]

Preparing test packages in https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4319

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.04-1ubuntu37

---------------
grub2 (2.04-1ubuntu37) hirsute; urgency=medium

  * debian/patches/grub-install-backup-and-restore.patch: Fix-up the patch
    to correctly initialyze the names of the modules to restore. LP:
    #1907085
  * 10_linux: emit messages when initrdless boot is configured, attempted
    and fails triggering fallback. LP: #1901553
  * grub-common.service: port init.d script to systemd unit. Add warning
    message, when initrdless boot fails triggering fallback. LP: #1901553
  * debian/rules: undo po/ directory patching in
    override_dh_autoreconf_clean.
  * minilzo: built using the distribution's minilzo
  * ubuntu-fix-reproducible-squashfs-test.patch: fix squashfs-test with
    new squashfs-tools in hirsute.
  * rhboot-f34-make-exit-take-a-return-code.patch,
    rhboot-f34-dont-use-int-for-efi-status.patch: allow grub to exit
    non-zero under EFI, this should allow falling back to the next
    BootOrder BootEntry.
  * rhboot-f34-tcp-add-window-scaling-support.patch: speed up netboot
    transfer speed.
  * rhboot-f34-support-non-ethernet.patch,
    ubuntu-fixup-rhboot-f34-support-non-ethernet.patch,
    ubuntu-fixup-rhboot-f34-support-non-ethernet-2.patch:
    add support for link layer addresses of up to 32-bytes.
  * rhboot-f34-make-pmtimer-tsc-calibration-fast.patch:
    speed up calibration time, especially when booting VMs.

 -- Dimitri John Ledkov <email address hidden> Sat, 12 Dec 2020 00:50:47 +0000

[Łukasz Zemczak (sil2100)]

Hello Dimitri, or anyone else affected,

Accepted grub2 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.04-1ubuntu26.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Julian Andres Klode (juliank)]

1. I upgraded grub2 to 2.04-1ubuntu26.13 from proposed using sudo apt install '?source-package(grub2)?installed'. It ran update-grub as postinst, and I saw the new message

GRUB_FORCE_PARTUUID is set, will attempt initrdless boot

2. I rebooted, I see message from grub saying

GRUB_FORCE_PARTUUID set, attempting initrdless boot.

before it loads the kernel.

3. After replacing kvm kernel with generic kernel, I see attempt message, but the generic kernel seems to boot fine without the initramfs, so I can't verify further.

[Julian Andres Klode (juliank)]

Retrying test 3 with UEFI and a virtio-scsci device, I succeed: The kernel panics, I then see

GRUB_FORCE_PARTUUID set, initrdless boot failed. Attempting with initrd.

and it boots fine

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.04-1ubuntu26.13

---------------
grub2 (2.04-1ubuntu26.13) focal; urgency=medium

  [ Julian Andres Klode ]
  * unapply all patches, use gbp pq instead of git-dpm

  [ Dimitri John Ledkov ]
  * 10_linux: emit messages when initrdless boot is configured, attempted and
    fails triggering fallback. LP: #1901553
  * grub-common.service: port init.d script to systemd unit. Add warning
    message, when initrdless boot fails triggering fallback. LP: #1901553
  * debian/grub-common.service: change type to oneshot, add wantedby
    sleep.target, after sleep.target. The service will now start after resume
    from hybernation. (LP: #1929860)
  * grub-initrd-fallback.service: add wantedby sleep.target, after
    sleep.target. The service will now start after resume from hybernation.
    LP: #1929860
  * grub-initrd-fallback.service, debian/grub-common.service: only start units
    when booted with grub. Use presence of /boot/grub/grub.cfg as proxy. LP:
    #1925507

 -- Julian Andres Klode <email address hidden> Thu, 12 Aug 2021 11:18:25 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for grub2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Julian Andres Klode (juliank)]

This is being SRUed to bionic as part of backporting the feature

[Steve Langasek (vorlon)]

An upload of grub2 to bionic-proposed has been rejected from the upload queue for the following reason: "The debdiff for this is a mess, there is delta to all the packages under debian/patches, impossible to review; please reupload without the extra delta".