[Regression] Focal kernel 5.4.0-92.103 fails to boot when Secure Encrypted Virtualization(SEV) is enabled

Bug #1956575 reported by Louis Bouchard
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Confirmed
Undecided
Unassigned
Xenial
Invalid
Undecided
Unassigned
Bionic
Confirmed
High
Unassigned
Focal
Fix Released
High
Unassigned
Hirsute
Invalid
Undecided
Unassigned

Bug Description

[Impact]

The latest Focal kernel (linux-image-5.4.0-92-generic) fails to boot when SEV is enabled.

The kernel panics with the following backtrace :

[ 1.531125] ledtrig-cpu: registered to indicate activity on CPUs
[ 1.531760] EFI Variables Facility v0.08 2004-May-17
[ 1.532575] general protection fault: 0000 [#1] SMP NOPTI
[ 1.533116] CPU: 11 PID: 1 Comm: swapper/0 Not tainted 5.4.157-debug6 #15
[ 1.533788] Hardware name: Scaleway SCW-ENT1-L, BIOS 0.0.0 02/06/2015
[ 1.534429] RIP: 0010:efi_mokvar_sysfs_init+0x9d/0x184
[ 1.534949] Code: 00 48 85 c0 0f 85 b3 00 00 00 48 c7 c7 b0 db e2 a1 41 bd f4 ff ff ff e8 98 90 16 ff e9 e6 00 00 00 48 85 d2 0f 85 a5 00 00 00 <80> 3b 00 0f 84 b5 00 00 00 48 85 db 0f 84 ac 00 00 00 48 8b 3d 85
[ 1.535120] RSP: 0018:ffffb96e4001bdf8 EFLAGS: 00010202
[ 1.535120] RAX: b6262a23e510e179 RBX: b625e392251db281 RCX: 0000000000000000
[ 1.535120] RDX: ffffb96e400cd000 RSI: ffff978437e5dc38 RDI: ffffffffa2121860
[ 1.535120] RBP: ffffb96e4001be10 R08: 0000000000000000 R09: 0000000000000228
[ 1.538947] R10: 0000000000000001 R11: 0000000000000000 R12: ffff978437e555a0
[ 1.538947] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 1.538947] FS: 0000000000000000(0000) GS:ffff97843f6c0000(0000) knlGS:0000000000000000
[ 1.538947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.538947] CR2: 00000000ffffffff CR3: 00080011a060a001 CR4: 0000000000360ee0
[ 1.538947] Call Trace:
[ 1.538947] ? efi_rci2_sysfs_init+0x29a/0x29a
[ 1.538947] do_one_initcall+0x4a/0x200
[ 1.538947] kernel_init_freeable+0x1c0/0x263
[ 1.538947] ? rest_init+0xb0/0xb0
[ 1.538947] kernel_init+0xe/0x110
[ 1.538947] ret_from_fork+0x22/0x40
[ 1.538947] Modules linked in:
[ 1.545871] ---[ end trace 815dc8177e65da02 ]---
[ 1.546328] RIP: 0010:efi_mokvar_sysfs_init+0x9d/0x184
[ 1.546872] Code: 00 48 85 c0 0f 85 b3 00 00 00 48 c7 c7 b0 db e2 a1 41 bd f4 ff ff ff e8 98 90 16 ff e9 e6 00 00 00 48 85 d2 0f 85 a5 00 00 00 <80> 3b 00 0f 84 b5 00 00 00 48 85 db 0f 84 ac 00 00 00 48 8b 3d 85
[ 1.548703] RSP: 0018:ffffb96e4001bdf8 EFLAGS: 00010202
[ 1.549218] RAX: b6262a23e510e179 RBX: b625e392251db281 RCX: 0000000000000000
[ 1.549916] RDX: ffffb96e400cd000 RSI: ffff978437e5dc38 RDI: ffffffffa2121860
[ 1.550617] RBP: ffffb96e4001be10 R08: 0000000000000000 R09: 0000000000000228
[ 1.551337] R10: 0000000000000001 R11: 0000000000000000 R12: ffff978437e555a0
[ 1.552036] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 1.552737] FS: 0000000000000000(0000) GS:ffff97843f6c0000(0000) knlGS:0000000000000000
[ 1.553529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.554093] CR2: 00000000ffffffff CR3: 00080011a060a001 CR4: 0000000000360ee0
[ 1.554818] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[ 1.555335] Kernel Offset: 0x1fa00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 1.555335] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---
-

The previous kernel (linux-image-5.4.0-91-generic) boots correctly with SEV.

Bisection of the kernels b/w 5.4.0-91 and 5.4.0-92 identified the following commit as the source of regresssion :

# git bisect good
7ca05228f713c24eb55574b36e32d9b54c5a1b76 is the first bad commit
commit 7ca05228f713c24eb55574b36e32d9b54c5a1b76
Author: Lenny Szubowicz <email address hidden>
Date: Fri Sep 4 21:31:05 2020 -0400

    efi: Support for MOK variable config table

    BugLink: https://bugs.launchpad.net/bugs/1928679

...

Since the panic happens in very early stages of boot, no trace of the panic is present in the log files and apparently it is not possible to collect a kernel crash dump, the crashkernel has not been enabled yet.

[Test case]

The kernel need to boot properly with SEV enabled.

[Potential regression]

The fix is very specific and restricted and should cause regressions, however it touches ioremap that is widely used.

Revision history for this message
Louis Bouchard (louis) wrote (last edit ):

Details of the bisected commit :

https://kernel.ubuntu.com/git/ubuntu/ubuntu-focal.git/commit/?id=7ca05228f713c24eb55574b36e32d9b54c5a1b76

commit 7ca05228f713c24eb55574b36e32d9b54c5a1b76
Author: Lenny Szubowicz <email address hidden>
Date: Fri Sep 4 21:31:05 2020 -0400

    efi: Support for MOK variable config table

    BugLink: https://bugs.launchpad.net/bugs/1928679

    Because of system-specific EFI firmware limitations, EFI volatile
    variables may not be capable of holding the required contents of
    the Machine Owner Key (MOK) certificate store when the certificate
    list grows above some size. Therefore, an EFI boot loader may pass
    the MOK certs via a EFI configuration table created specifically for
    this purpose to avoid this firmware limitation.

    An EFI configuration table is a much more primitive mechanism
    compared to EFI variables and is well suited for one-way passage
    of static information from a pre-OS environment to the kernel.

    This patch adds initial kernel support to recognize, parse,
    and validate the EFI MOK configuration table, where named
    entries contain the same data that would otherwise be provided
    in similarly named EFI variables.

    Additionally, this patch creates a sysfs binary file for each
    EFI MOK configuration table entry found. These files are read-only
    to root and are provided for use by user space utilities such as
    mokutil.

    A subsequent patch will load MOK certs into the trusted platform
    key ring using this infrastructure.

    Signed-off-by: Lenny Szubowicz <email address hidden>
    Link: https://<email address hidden>
    Signed-off-by: Ard Biesheuvel <email address hidden>
    (cherry picked from commit 58c909022a5a56cd1d9e89c8c5461fd1f6a27bb5)
    Signed-off-by: Dimitri John Ledkov <email address hidden>
    Acked-by: Stefan Bader <email address hidden>
    Acked-by: Tim Gardner <email address hidden>
    Signed-off-by: Stefan Bader <email address hidden>

 arch/x86/kernel/setup.c | 1 +
 arch/x86/platform/efi/efi.c | 3 +
 drivers/firmware/efi/Makefile | 1 +
 drivers/firmware/efi/arm-init.c | 1 +
 drivers/firmware/efi/efi.c | 6 +
 drivers/firmware/efi/mokvar-table.c | 360 ++++++++++++++++++++++++++++++++++++
 include/linux/efi.h | 34 ++++
 7 files changed, 406 insertions(+)
 create mode 100644 drivers/firmware/efi/mokvar-table.c

Changed in linux (Ubuntu Focal):
status: New → Confirmed
Changed in linux (Ubuntu):
status: New → Confirmed
summary: [Regression] Focal kernel 5.4.0-92.103 fails to boot when Secure
- Encrypted Virtualization(SEV) is enagbled
+ Encrypted Virtualization(SEV) is enabled
Louis Bouchard (louis)
description: updated
Revision history for this message
Louis Bouchard (louis) wrote :

According to the following advisory[1] from HP, this is a known issue which is fixed upstream :

"Advisory: Red Hat Enterprise Linux 8.3 - Guest Stops Responding at efi_mokvar_sysfs_init+0xa9/0x19d with AMD Secure Encrypted Virtualization (SEV) Enabled"

The upstream fix cited in the article is the following :

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d651ee9c71bb12fc0c8eb2786b66cbe5aa3e43b

x86/ioremap: Map EFI-reserved memory as encrypted for SEV

Adding this commit on top of kernel 5.4.0-92.103 does fix the booting issue.

[1] https://support.hpe.com/hpesc/public/docDisplay?docId=a00119071en_us&docLocale=en_US

Revision history for this message
Marcelo Cerri (mhcerri) wrote :
description: updated
Changed in linux (Ubuntu Focal):
status: Confirmed → In Progress
Changed in linux (Ubuntu Hirsute):
status: New → Invalid
Changed in linux (Ubuntu Focal):
importance: Undecided → High
Revision history for this message
Marcelo Cerri (mhcerri) wrote :

I also prepared a bionic/linux-gcp-5.4 test kernel with the fix and I confirmed it also fixed the same boot problem in confidential computing instances.

Revision history for this message
Khaled El Mously (kmously) wrote :

Hello @Louis,

Are you able to confirm whether the 4.15 kernels are affected by this problem?

Revision history for this message
Marcelo Cerri (mhcerri) wrote :

4.4 doesn't have the offending commit so marking it as invalid.

Changed in linux (Ubuntu Xenial):
status: New → Invalid
Revision history for this message
Louis Bouchard (louis) wrote :

Hello @kmously, yes I am able to reproduce the same panic on 4.15.0-166.

Changed in linux (Ubuntu Bionic):
status: New → Confirmed
importance: Undecided → High
Changed in linux (Ubuntu Focal):
status: In Progress → Fix Committed
Revision history for this message
Louis Bouchard (louis) wrote :
Download full text (4.9 KiB)

Here is the result of a similar bisect on the bionic kernel :

https://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/commit/?id=091554e4a5b2a7647830a1c7beea781148b51509

091554e4a5b2a7647830a1c7beea781148b51509 is the first bad commit
commit 091554e4a5b2a7647830a1c7beea781148b51509
Author: Lenny Szubowicz <email address hidden>
Date: Tue Nov 30 12:04:00 2021 +0100

    efi: Support for MOK variable config table

    BugLink: https://bugs.launchpad.net/bugs/1928679

    Because of system-specific EFI firmware limitations, EFI volatile
    variables may not be capable of holding the required contents of
    the Machine Owner Key (MOK) certificate store when the certificate
    list grows above some size. Therefore, an EFI boot loader may pass
    the MOK certs via a EFI configuration table created specifically for
    this purpose to avoid this firmware limitation.

    An EFI configuration table is a much more primitive mechanism
    compared to EFI variables and is well suited for one-way passage
    of static information from a pre-OS environment to the kernel.

    This patch adds initial kernel support to recognize, parse,
    and validate the EFI MOK configuration table, where named
    entries contain the same data that would otherwise be provided
    in similarly named EFI variables.

    Additio...

Read more...

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux/5.4.0-94.106 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
Louis Bouchard (louis)
tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 5.4.0-94.106

---------------
linux (5.4.0-94.106) focal; urgency=medium

  * focal/linux: 5.4.0-94.106 -proposed tracker (LP: #1956628)

  * [Regression] Focal kernel 5.4.0-92.103 fails to boot when Secure Encrypted
    Virtualization(SEV) is enabled (LP: #1956575)
    - x86/ioremap: Map EFI-reserved memory as encrypted for SEV

 -- Khalid Elmously <email address hidden> Thu, 06 Jan 2022 16:56:46 -0500

Changed in linux (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

Hi Louis,

Thank you very much for reporting the issue and helping us debug and verify the fix!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers