"error: can't find command hwmatch" during grub

[Luis Alberto Pabón]

[Impact]

 * Users on non-pc/i386 platforms (e.g., efi/amd64) may observe
   the message "error: can't find command `hwmatch'" every boot.

 * Also observed on cloud instances, generating support tickets
   asking for clarification/impacts. (which could be deflected.)

 * There' little or no impact, as hwmatch is only used to check
   hardware device ID against a list for issues w/ graphics/KMS.
   When this error happens, the default is set to keep graphics.

 * This has possibly always happened on non-pc/i386, as hwmatch
   is patched/exists only in 'grub-core/commands/_i386/pc_' dir,
   thus there doesn't seem to be any serious issues as a result.

[Fix]

 * Check for `$grub_platform != pc` then do not call hwmatch;
   just keep current behavior (ie, keep graphics/framebuffer.)

[Test Plan]

 * Boot a VM with EFI (LXD or multipass/libvirt/ovmf) and see
   the serial console show the error message (or not, w/ fix.)

 - lxd:

  $ lxc launch ubuntu:impish impish-vm --vm --console
  Creating impish-vm
  Starting impish-vm
  To detach from the console, press: <ctrl>+a q
  BdsDxe: loading Boot0001 "UEFI QEMU QEMU HARDDISK " from ...
  BdsDxe: starting Boot0001 "UEFI QEMU QEMU HARDDISK " from ...
  error: can't find command `hwmatch'.
  ...

 - multipass/libvirt/ovmf:

 sudo snap install multipass
 sudo multipass set local.driver=libvirt
 sudo snap connect multipass:libvirt

 multipass launch -c 1 -d 4g -m 1g -n hwmatch-i impish

 sudo apt -y install ovmf
 sudo mkdir -p /var/lib/libvirt/qemu/nvram

 virsh edit hwmatch-i
   # insert in <os> section:
   <loader readonly='yes' type='pflash'>/usr/share/OVMF/OVMF_CODE.fd</loader>
   <nvram>/var/lib/libvirt/qemu/nvram/hwmatch-i.fd</nvram>

 virsh shutdown hwmatch-i
 virsh start --console hwmatch-i

 * Before:
 ...
 BdsDxe: loading Boot0003 "ubuntu" from ...\shimx64.efi
 BdsDxe: starting Boot0003 "ubuntu" from ...\shimx64.efi
 error: can't find command `hwmatch'.
 [ 0.000000] Linux version ...

 * After:
 sudo dpkg -i grub_common_*.deb # test package; g-c is enough.
 sudo update-grub && sudo reboot
 ...
 BdsDxe: loading Boot0003 "ubuntu" from ...\shimx64.efi
 BdsDxe: starting Boot0003 "ubuntu" from ...\shimx64.efi
 [ 0.000000] Linux version ...

[Where problems could occur]

 * Issues with graphics/flickering during boot on systems with
   graphics adapters in use might be hit in case of regression.

 * Theoretically the risk is low, since hwmatch apparently has
   never worked on non-pc/i386, and we replace the broken call
   while keeping current behavior observed with it ("no change".)

[Other Info]

 * While it might be interesting to try and enable/fix it for
   other platforms, let's have a trivial fix to be considered
   to backport to stable releases, at this time (less changes.)

 * The same behavior is linux_gfx_mode=keep, despite the error.
   This happens because in the error case, grub shell evaluates
   if hwmatch ...; then` to true, and `if [ $match = 0 ]; then`
   to true too (as it's undefined) so `set linux_gfx_mode=keep`.

   Before/After in grub shell (same behavior):

    grub> hwmatch
    error: can't find command `hwmatch'.

    grub> echo $grub_platform
    efi

    grub> echo $linux_gfx_mode
    keep

 * Other details in Debian bug 990836 [1].

 * It looks like grub2-unsigned follows grub2 automatically,
   so not providing debdiffs to it/just grub2.

[Links]
 * [1] https://bugs.debian.org/990836

[Original Description]
Upon disabling my grub menu using grub-customizer, I've noticed that just before the system boots and goes into the crypto password prompt the following error message appears on the display:

```
error: can't find command hwmatch
```

Looks like this command is referenced from /boot/grub/grub/cfg

```
    if hwmatch ${prefix}/gfxblacklist.txt 3; then
```
However, the mod doesn't exist on grub's mod folder:

```
~ ls -la /boot/grub/x86_64-efi/*.mod|wc -l
269

~ ls -la /boot/grub/x86_64-efi/hw*
fish: No matches for wildcard “/boot/grub/x86_64-efi/hw*”

```

ProblemType: Bug
DistroRelease: Ubuntu 19.04
Package: grub-efi-amd64-signed 1.115+2.02+dfsg1-12ubuntu2
ProcVersionSignature: Ubuntu 5.0.0-25.26-generic 5.0.18
Uname: Linux 5.0.0-25-generic x86_64
ApportVersion: 2.20.10-0ubuntu27.1
Architecture: amd64
CurrentDesktop: Unity
Date: Sun Aug 18 00:22:41 2019
InstallationDate: Installed on 2019-08-17 (0 days ago)
InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
SourcePackage: grub2-signed
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Bill Miller (wbmilleriii)]

This problem affected me on a fresh install of Ubuntu 18.04. The package grub-efi-amd64-bin doesn't include the hwmatch module in the long list of modules it does install. But the script 10_linux in /etc/grub.d/ references this hwmatch module, causing the error.

Note hwmatch *is* included in the modules installed by the BIOS version of grub, grub-pc-bin.

I worked around this issue by commenting out the if-then structure in the 10_linux and updating grub (see this Ask Ubuntu post for the code https://askubuntu.com/questions/1174374/grub-issues-message-at-boot-error-cant-find-command-hwmatch/1174386#1174386) but I am unsure what functionality is missing because of this error of omission.

[Rostislav Stříbrný (rstribrn)]

Rather than editing /etc/grub/10_linux, I suggest adding this line to the /etc/default/grub:
    GRUB_GFXPAYLOAD_LINUX=keep

So the problematic if-then statement is not reached at all.
Don't forget executing "sudo update-grub" afterwards.

However, its still just another workaround...

[Thomas Piekarski (t-piekarski)]

Hello,

thanks for supplying that workaround. I can confirm this issue with 19.10 as well.
Both workarounds work for me and I'll take a closer look into why hwmatch is missing.
Whats the current progress and any news about it?

[Psi-Jack (erenfro)]

Interesting. This issue was normally hidden by grub being hidden in the first place, but once I enabled it and a reasonable timeout, I was able to see it. Either way, I see it, I don't know what the potential downside is to it not being there, or what hwmatch itself does.

[Thomas Piekarski (t-piekarski)]

Hello,

well also a hidden error is still an error :)

The module hwmatch is for white/blacklisting hardware vendors according to the source file. Actually there is no further information available in the info documentation about this module. I'll check why this module is not around for x86_64 EFI, check most recent version at the git repository and try to provide a patch.

[Blastoff (wolph)]

hwmatch needs to be added to https://salsa.debian.org/grub-team/grub/-/blob/master/debian/build-efi-images

[Thomas Piekarski (t-piekarski)]

Thanks for finding the issue :)

[ruud (ruudschellekens)]

I see this bug again in Ubuntu 20.04. The error appear briefly just before GRUB show the boot menu

uname -a ouptut:
Linux ThinkPad-E550 5.4.0-29-generic #33-Ubuntu SMP Wed Apr 29 14:32:27 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[Luis Alberto Pabón (copong)]

Any reason the (failed) call to hwmatch is important? Ubuntu has worked fine for quite some time now without it, so perhaps it can be removed altogether from grub?

[Rodrigo Valentim (rodrigo-is)]

Well, that's happening with me every time I forget to use my "workaround", because my boot fail, asking to load kernel first. Before find this bug here, I found several users with same issue and unable to boot too, so I think it's important to fix it, otherwise people like me may have more issues and because of this we can't see the others, ending up doing the "wrong fix".

When that happen I use my "workaround", which is turn off the laptop, wait around 10 seconds (less than that the issue continues), than turn on keeping some keys down (normally I use the down key + ESC), with that I can't see the error and boot works fine.

I don't know if this bug is the reason for failling the boot or why my "workarond" works, but that's is happening only with Kubuntu 20.04 for me, I tried other distro and was OK.

[Ishrak (irik77587)]

On ubuntu 20.04.1, grub menu was not showing, enabled console mode. Now grub is showing when ESC pressed, but this error appears

Ref: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1863434
Solution: #48

[Mauricio Faria de Oliveira (mfo)]

Patch submitted to Debian in 990836. [1]

Waiting for review feedback there, if possible, for a bit, then moving on w/ Ubuntu.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=990836

[Mauricio Faria de Oliveira (mfo)]

Moving on with Ubuntu.
Attaching patch for impish for initial review.

[Julian Andres Klode (juliank)]

Please submit a proper merge against the git repo and make sure to use gbp pq to export the patch instead of manually editing debian/patches (I can see in your patch, your formatting is off).

The repository is located at https://code.launchpad.net/~ubuntu-core-dev/grub/+git/ubuntu

[Mauricio Faria de Oliveira (mfo)]

@juliank, will do; thanks for the process/tools hint!

[Mauricio Faria de Oliveira (mfo)]

Merge proposal [1] submitted against the git repo, done w/ gbp pq.

[1] https://code.launchpad.net/~mfo/grub/+git/grub/+merge/407470

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.04-1ubuntu48

---------------
grub2 (2.04-1ubuntu48) jammy; urgency=medium

  * d/p/0241-Call-hwmatch-only-on-the-grub-pc-platform.patch:
    Fix "error: can't find command `hwmatch'." on non-i386/pc
    platforms such as x86_64/efi. (LP: #1840560)

 -- Mauricio Faria de Oliveira <email address hidden> Thu, 04 Nov 2021 10:48:06 -0300

[Mauricio Faria de Oliveira (mfo)]

Changes tested for SRU to Impish, Hirsute, and Focal.
(Let's skip Bionic unless there's a strong necessity.)

The results are equivalent on Impish/Hirsute/Focal.
(hwmatch error line no longer present.)

...

Since the changes are low impact and low priority,
let's just stage them instead of pushing SRUs now.

Only the MR for Focal could be submitted since MRs
in LP can't create new branches / need an existing
branch, and both Impish and Hirsute would need new
branch in the package's git repo so only Focal has
a MR proposed (that mentions this + pull requests.)

...

- https://code.launchpad.net/~mfo/grub/+git/grub/+ref/lp1840560-impish

- https://code.launchpad.net/~mfo/grub/+git/grub/+ref/lp1840560-hirsute

- https://code.launchpad.net/~mfo/grub/+git/grub/+ref/lp1840560-focal
- https://code.launchpad.net/~mfo/grub/+git/grub/+merge/412712

cheers,
Mauricio

...

Steps:

$ lxc launch ubuntu:impish impish-vm --vm
$ while true; do lxc console impish-vm; done
To detach from the console, press: <ctrl>+a

BdsDxe: loading Boot0001 "UEFI QEMU QEMU HARDDISK " ...
BdsDxe: starting Boot0001 "UEFI QEMU QEMU HARDDISK " ...
error: can't find command `hwmatch'.
GRUB_FORCE_PARTUUID set, attempting initrdless boot.
EFI stub: UEFI Secure Boot is enabled.
...

$ sudo dpkg -i grub-common_*.deb && sudo update-grub && sudo reboot
...

To detach from the console, press: <ctrl>+a
BdsDxe: loading Boot0007 "ubuntu" ...
BdsDxe: starting Boot0007 "ubuntu" ...
GRUB_FORCE_PARTUUID set, attempting initrdless boot.
EFI stub: UEFI Secure Boot is enabled.

[Julian Andres Klode (juliank)]

Some tasks were incorrectly marked 'In Progress' despite not having an upload in the SRU queue. They have been reset to 'Triaged'.

[Julian Andres Klode (juliank)]

Hirsute hippo is about to be end of life, so will not receive this update

[Julian Andres Klode (juliank)]

Removing wrong block-proposed-* tags, these were never uploaded on their own, so there is nothing to block.

[Łukasz Zemczak (sil2100)]

Hello Luis, or anyone else affected,

Accepted grub2 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.04-1ubuntu26.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Luis Alberto Pabón (copong)]

I can confirm 2.04-1ubuntu26.15 from focal-proposed fixes it. Thank you Lukasz 👍

[Mauricio Faria de Oliveira (mfo)]

Verification done on focal-proposed.
There is no 'hwmatch error' message.

(test steps in bug description)
$ multipass launch -c 1 -d 4g -m 1g -n hwmatch-f focal

$ virsh edit hwmatch-f # OVMF firmware
$ virsh shutdown hwmatch-f

$ virsh start --console hwmatch-f
Domain 'hwmatch-f' started
Connected to domain 'hwmatch-f'
Escape character is ^] (Ctrl + ])
BdsDxe: loading Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x4,0x0)
BdsDxe: starting Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x4,0x0)
error: can't find command `hwmatch'.
[ 0.000000] Linux version ...
...

$ dpkg -s grub-common | grep Version:
Version: 2.04-1ubuntu26.13

$ echo "deb http://archive.ubuntu.com/ubuntu focal-proposed main restricted" | sudo tee /etc/apt/sources.list.d/proposed.list
$ sudo apt update
$ sudo apt install --yes grub-commmon

$ dpkg -s grub-common | grep Version:
Version: 2.04-1ubuntu26.15

$ sudo poweroff
...
[ 53.978674] reboot: Power down

$ virsh start --console hwmatch-f
Domain 'hwmatch-f' started
Connected to domain 'hwmatch-f'
Escape character is ^] (Ctrl + ])
BdsDxe: loading Boot0003 "ubuntu" from HD(15,GPT,12CC0C76-621A-4ADF-B865-28DD708A6ADF,0x2800,0x35000)/\EFI\ubuntu\shimx64.efi
BdsDxe: starting Boot0003 "ubuntu" from HD(15,GPT,12CC0C76-621A-4ADF-B865-28DD708A6ADF,0x2800,0x35000)/\EFI\ubuntu\shimx64.efi
[ 0.000000] Linux version ...
...

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.04-1ubuntu26.15

---------------
grub2 (2.04-1ubuntu26.15) focal; urgency=medium

  [ Mauricio Faria de Oliveira ]
  * Call hwmatch only on the grub-pc platform (LP: #1840560)

grub2 (2.04-1ubuntu26.14) focal; urgency=medium

  * grub-check-signatures: Support gzip compressed kernels (LP: #1954683)

 -- Julian Andres Klode <email address hidden> Tue, 11 Jan 2022 16:09:48 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for grub2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.