grub2 incompatibility between at_keyboard and gfxterm modules lead to impossible to type passphrase or access ubuntu grub2 menu on a fully encrypted installation

[Fabian]

Hello,

On a freshly installed Ubuntu 20.04.1 LTS (tested with Mate & Ubuntu with two different brands and configurations of laptops) fully luks encrypted systems using this guide https://help.ubuntu.com/community/Full_Disk_Encryption_Howto_2019 as a basis with the following configurations:

laptop 1 (Dell): x86_64 BIOS/GPT
/dev/sda1 => bios_grub
/dev/sda2 => luks encrypted LVM with VG,LV /boot, /root, /home, /swap

laptop 2 (Acer): x86_64 UEFI/GPT
/dev/sda1 => ESP
/dev/sda2 => luks encrypted LVM with VG,LV /boot, /root, /home, /swap

It's impossible with :
=> 1st stage grub: /dev/sda1 bios_grub (core.img) or ESP partition (grub.efi)
=> 2nd stage grub: /dev/sda2/LUKS_VOL/LVM/boot/grub
to:
1- set and use a local grub keyboard layout (French) to type passphrase in 1st stage grub
2- access (edit mode or typing command under grub console) in 2nd stage grub

Here all my tests results from the last 3 days on the two laptops at same time with same results BIOS (Dell) or EFI (Acer):

[0] Default settings after installation :
1st stage grub: US keyboard layout (default)
2nd stage grub: gfxterm module loaded (install defaults)
=> 1st stage grub: almost impossible to type a strong passphrase with accentuated characters and special characters due to typos error and keyboard layout difference and unlock encrypted luks volume.
=> 2nd stage grub: working without trouble, can access ubuntu grub menus, edit command and type command in the grub console as normal.

[1] Modfied settings after installation : 1st stage grub only
1st stage grub: French keyboard layouts with at_keyboard module ([1A] grub core image config at the end)
2nd stage grub: gfxterm module loaded (no modification from default installation)
=> 1st stage grub: can type strong passphrase and unlock at the first attempt the luks encrypted volume
=> 2nd stage grub: Impossible to access grub menus, edit command and type command in the grub console

[2] Modfied settings after installation : 2nd stage grub only
1st stage grub: default grub core.img after installation with US layout
2nd stage grub: gfxterm module loaded (install defaults) + at_keybord, "terminal_input at_keyboard", keymap fr.gkb ([2A] grub.cfg config at the end)
=> 1st stage grub: almost impossible to type strong passphrase as in test case [0]
=> 2nd stage grub: almost Impossible to access grub menus, edit command and type command in the grub console). Each keystroke must be repeated 7 times or more to get a result (and PC speaker beep every 1x or 4x keystrokes but it's really secondary). Ressetting from "terminal_input at_keyboard" to "terminal_input console" (by removing at_keyboard) or from "terminal_output gfxterm" to "terminal_output console" (by removing gfxterm) (by typing t(x7+2xbeep)e(x10+3xbeep)r(x7+4xbeep)m(x7+1xbeep)i.n..) resolved the situation.

[3] Modfied settings after installation : 1st stage grub only
1st stage grub: French keyboard layout with at_keybord + gfxterm module
2nd stage grub: gfxterm module loaded (no modification from default installation)
=> 1st stage grub: Impossible to type strong passphrase as in test case [0] but this time it's because gfxterm and at_keyboard are loaded at the same time and each keystroke must be typed 7x or more times (sometimes more than 10 times + beep) to be taken into account, so impossible to know what you are typing because nothing is printed at screen while you type your password. By accessing grub minimal console due to a limited core.img and removing at_keybord with "terminal_input console" or removing gfxterm with "terminal_output console" resolved the situation but typing such command as set prefix=(memdisk), ... cryptomount -u xyzaabac, set root='lvmid/Eh2HHw-Bceq-VfsK-Kz01-kaeg-dRbz-Rwcntf/nVAE5f-GOrJ-wvu3-RTN0-mzAe-G4Oo-mzTQrN' and then the passphrase without any typos erros at first attempt with a different keyboard layout was a real challenge.
=> 2nd stage grub: Impossible to access grub menus, edit command and type command in the grub console when at_keybord (eg: "terminal_input at_keyboard") is loaded even in 1st stage grub.

Conclusion:
It seems that if grub at_keyboard module is loaded with "terminal_input at_keyboard" (to load a french keyboard layout and tested with german, italian layouts to be sure that the french one was not the source of troubles ...) even only in the 1st stage grub ... it's almost impossible to access any grub menu (edit command or console) in grub (same level) or even in another child grub as the 2nd stage if grub gfxterm module is loaded too.

So, for international users typing a strong passphrase is almost impossible without using something like "test123" or "qwerty123" and keeping access on any child grub.

For me it's something like critical because I can use a secure installation with all partition/data encrypted but I can't type a strong passphrase (eg "C0udéeM+(@rpantÀ)") or access any child grub menu or console if any grub at_keyboard module is loaded (at a upper level or at same level) with a gfxterm module somewhere else (at same level or at a child level).

I found an old bug report at debian.org but it seems resolved with grub 2.02+dfsg1-11.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=741464

I apologize for my english language mistakes.
Regards,
Fabien

[1A] grub core.img (1st grub stage) with french keyboard layout (Dell Laptop only but I can provide too the Acer laptaop with EFI difference to generate a grub.efi)
Based on Archwiki tutorial: https://wiki.archlinux.org/index.php/GRUB/Tips_and_tricks#Manual_configuration_of_core_image_for_early_boot

insmod echo
insmod gzio
insmod part_gpt
insmod cryptodisk
insmod luks
insmod gcry_rijndael
insmod gcry_rijndael
insmod gcry_sha256
insmod lvm
insmod ext2
insmod keylayouts
insmod at_keyboard
set root=(memdisk)
set prefix=($root)/
terminal_input at_keyboard
keymap /fr.gkb
cryptomount -u xyz123abc456def789ghi
set root='lvmid/Eh2HHw-Bceq-VfsK-Kz01-kaeg-dRbz-Rwcntf/nVAE5f-GOrJ-wvu3-RTN0-mzAe-G4Oo-mzTQrN'
set prefix=($root)/grub
configfile grub.cfg

[2A] Ubuntu grub (2nd stage) modification (Dell or Acer laptop)
insmod keylayouts
insmod at_keyboard
terminal_input at_keyboard
keymap $(prefix)/fr.gkb

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[John Doe (user210220)]

I can confirm this bug.
But a qemu machine seems unaffected.

There is another effect:

Step 1: Let the GRUB console start.

Step 2: select console as terminal_input and terminal_output. This is the default anyways.

Step 3: set a non-US keyboard layout
since console is the active terminal_input, the new layout has no effect yet

Step 4: set console AND at_keyboard (both! This is possible) as terminal_input
now ALL keys you press are handled by at_keyboard. You can detect this because the non-US keyboard layout is used for all keys pressed. There is no delay, everything just works great right now.

Step 5: activate gfxterm as terminal_output
Now strange things happen: press a key that would be affected by the non-US keboard layout. You get random output now: the character you see can be affected by the non-US layout, it can be the US layout character or the keypress does nothing at all.
The only difference to step 4: gfxterm is active

Why does gfxterm affect at_keyboard?

[John Doe (user210220)]

This solved the non-responsive keyboard bug for me:
https://<email address hidden>/msg29091.html

Non-US keyboard layouts and non-ASCII character output are separate problems. I did not find an official solution for those and had to write my own patches.

[John Doe (user210220)]

The 'at' in the URL above seems to be a problem. I can see the URL only, if I'm logged in. Here is an alternative:
https://lists.gnu.org/archive/html/grub-devel/2019-08/msg00014.html

[Fabian (moocan2112)]

@John Doe (user210220)
Many many thanks for your return but I can't test it.
I found the related commit here https://github.com/rhboot/grub2/commit/33203ca.patch/
I tried to build grub2-2.04 from sources (all dependencies installed) with at_keyboard patch but it failed each time (with or without the patch).
I found some bug reports about my errors (with 2.02) ... but no answers.
Thus I need to do this for x86_64 BIOS and EFI.
As I have this bug for all computers and all I can do is maybe buying a qwerty keyboard for each of them to type a passphrase (even this is not sure because loading usb_keyboard module prevent some of them to boot because no partition are detected when this module is loaded).
Sorry, I'm really saddened that if we do not have a QWERTY US keyboard layout on such a thing that a boot process, you are blocked to type a passphrase or just to boot.
Best Regards

[John Doe (user210220)]

./autogen.sh
./configure --with-platform=efi --target=x86_64 --disable-werror
make

[Fabian (moocan2112)]

Impossible ...
25+ times that I'm trying to build and always "FAIL squashfs_test (exit status: 1)"
You can see in the attached shquashfs_test.log

I don't see errors and when I try open the generated /tmp/tmp.kOJOtwDgFh/squash4_gzip_512_4096_1_0.img ... I can open it.
So I don't understand, I don't find anything and don't know where to search now.

So each time "FAIL squashfs_test", # FAIL: 1, bye bye .. no keyboard !

============================================================================
Testsuite summary for GRUB 2.04-1ubuntu46
============================================================================
# TOTAL: 82
# PASS: 66
# SKIP: 15
# XFAIL: 0
# FAIL: 1
# XPASS: 0
# ERROR: 0
============================================================================
See ./test-suite.log
Please report to <email address hidden>
============================================================================
make[6]: *** [Makefile:12057: test-suite.log] Error 1
make[6]: Leaving directory '/home/sysadmin/grub-sources/grub2-2.04/obj/grub-pc'
make[5]: *** [Makefile:12165: check-TESTS] Error 2
make[5]: Leaving directory '/home/sysadmin/grub-sources/grub2-2.04/obj/grub-pc'
make[4]: *** [Makefile:12964: check-am] Error 2
make[4]: Leaving directory '/home/sysadmin/grub-sources/grub2-2.04/obj/grub-pc'
make[3]: *** [Makefile:11942: check-recursive] Error 1
make[3]: Leaving directory '/home/sysadmin/grub-sources/grub2-2.04/obj/grub-pc'
make[2]: *** [Makefile:12966: check] Error 2
make[2]: Leaving directory '/home/sysadmin/grub-sources/grub2-2.04/obj/grub-pc'
dh_auto_test: error: cd obj/grub-pc && make -j1 check VERBOSE=1 returned exit code 2
make[1]: *** [debian/rules:370: debian/stamps/build-grub-pc] Error 25
make[1]: Leaving directory '/home/sysadmin/grub-sources/grub2-2.04'
make: *** [debian/rules:201: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
debuild: fatal error at line 1182:

I'm disappointed.
Regards

[Thibault (thibaultcdlfdc)]

I have a bug which I think is related. When I type:

grub> terminal_input
Active input terminals:
console
Available input terminals:
serial_* serial at_keyboard

Yet, when I type:

grub> terminal_input at_keyboard
error: terminal 'at_keyboard' isn't found.

I have Secure Boot activated.

[Mate Kukri (mkukri)]

https://help.ubuntu.com/community/Full_Disk_Encryption_Howto_2019

> DISCLAIMER: Doing this is an unsupported configuration for Ubuntu, and may cause up-to and including boot breakage on upgrades. If possible please use the password based or TPM FDE options provided by the Ubuntu installer.