| 2021-06-08 13:22:15 |
Joshua Powers |
bug |
|
|
added bug |
| 2021-06-08 14:17:53 |
Chris Newcomer |
summary |
Google Confidnetial Compute fails to boot with 1.47 |
Google Confidential Compute fails to boot with 1.47 |
|
| 2021-06-08 16:54:33 |
Steve Langasek |
bug task added |
|
linux-gcp (Ubuntu) |
|
| 2021-06-30 03:50:02 |
Khaled El Mously |
bug task deleted |
shim-signed (Ubuntu) |
|
|
| 2021-07-08 05:27:19 |
Khaled El Mously |
nominated for series |
|
Ubuntu Hirsute |
|
| 2021-07-08 05:27:19 |
Khaled El Mously |
bug task added |
|
linux-gcp (Ubuntu Hirsute) |
|
| 2021-07-08 05:49:38 |
Khaled El Mously |
description |
# Overview
Hirsute and Impish daily builds are currently not booting on Google Confidential Compute. Confidential compute is Google's platform that enables the use of Secure Encrypted Virtualization extension via AMD EPYC CPUs. Booting an image with version 1.45 works, but once upgraded to 1.47, the VM no longer boots, and instead the kernel panics.
Launching the image with secure boot, but without confidential compute works as expected.
# Expected result
The system is able to reboot after the upgrade.
# Actual result
Kernel panic: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Steps to reproduce
Launch a VM in GCE with confidential compute enabled with a serial v20210511a or later and look at the serial log for the kernel panic. Example CLI command to launch a VM:
$ gcloud beta compute instances create $USER-confidential-testing --zone=us-west1-b --machine-type=n2d-standard-2 --image=daily-ubuntu-2104-hirsute-v20210511a --image-project=ubuntu-os-cloud-devel --confidential-compute --maintenance-policy=TERMINATE
The last known good working image is daily-ubuntu-2104-hirsute-v20210510. The upgrade that fails is when shim signed is updated from 1.46+15.4-0ubuntu1 to 1.47+15.4-0ubuntu2
# Logs & notes
* 20210510 manifest (good): https://paste.ubuntu.com/p/QjnMPcJj7G/
* 20210511a manifest (bad): https://paste.ubuntu.com/p/PvJQwRXHcG/
* diff between manifests: https://paste.ubuntu.com/p/4nJtGxqGn7/
* serial logs of failed boot: https://paste.ubuntu.com/p/mHrvVc6qBc/ |
# Overview
Hirsute and Impish daily builds are currently not booting on Google Confidential Compute. Confidential compute is Google's platform that enables the use of Secure Encrypted Virtualization extension via AMD EPYC CPUs. Booting an image with version 1.45 works, but once upgraded to 1.47, the VM no longer boots, and instead the kernel panics.
Launching the image with secure boot, but without confidential compute works as expected.
# Expected result
The system is able to reboot after the upgrade.
# Actual result
Kernel panic: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Steps to reproduce
Launch a VM in GCE with confidential compute enabled with a serial v20210511a or later and look at the serial log for the kernel panic. Example CLI command to launch a VM:
$ gcloud beta compute instances create $USER-confidential-testing --zone=us-west1-b --machine-type=n2d-standard-2 --image=daily-ubuntu-2104-hirsute-v20210511a --image-project=ubuntu-os-cloud-devel --confidential-compute --maintenance-policy=TERMINATE
The last known good working image is daily-ubuntu-2104-hirsute-v20210510. The upgrade that fails is when shim signed is updated from 1.46+15.4-0ubuntu1 to 1.47+15.4-0ubuntu2
# Logs & notes
* 20210510 manifest (good): https://paste.ubuntu.com/p/QjnMPcJj7G/
* 20210511a manifest (bad): https://paste.ubuntu.com/p/PvJQwRXHcG/
* diff between manifests: https://paste.ubuntu.com/p/4nJtGxqGn7/
* serial logs of failed boot: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Cause:
shim changed the memory type for pages reserved for EFI runtime services, from EfiRuntimeServicesData to EfiBootServicesData.
Memory reserved for EFI runtime/boot services must be remapped as encrypted in the kernel (during boot) if SEV (secure encrypted virtualization) is enabled. The original kernel implementation of ioremap only correctly mapped the region as encrypted for EfiRuntimeServicesData regions, so when shim changed the type to EfiBootServicesData the kernel bug was exposed
# Fix:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d651ee9c71bb12fc0c8eb2786b66cbe5aa3e43b
# Regression potential
The fix could potentially cause boot failures, if a memory region is marked encrypted when it isn't. I assume it would cause a panic similar to the one seen here for this bug:
general protection fault, probably for non-canonical address 0x314836c31124d346: 0000 [#1] SMP NOPTI |
|
| 2021-07-08 05:50:47 |
Khaled El Mously |
description |
# Overview
Hirsute and Impish daily builds are currently not booting on Google Confidential Compute. Confidential compute is Google's platform that enables the use of Secure Encrypted Virtualization extension via AMD EPYC CPUs. Booting an image with version 1.45 works, but once upgraded to 1.47, the VM no longer boots, and instead the kernel panics.
Launching the image with secure boot, but without confidential compute works as expected.
# Expected result
The system is able to reboot after the upgrade.
# Actual result
Kernel panic: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Steps to reproduce
Launch a VM in GCE with confidential compute enabled with a serial v20210511a or later and look at the serial log for the kernel panic. Example CLI command to launch a VM:
$ gcloud beta compute instances create $USER-confidential-testing --zone=us-west1-b --machine-type=n2d-standard-2 --image=daily-ubuntu-2104-hirsute-v20210511a --image-project=ubuntu-os-cloud-devel --confidential-compute --maintenance-policy=TERMINATE
The last known good working image is daily-ubuntu-2104-hirsute-v20210510. The upgrade that fails is when shim signed is updated from 1.46+15.4-0ubuntu1 to 1.47+15.4-0ubuntu2
# Logs & notes
* 20210510 manifest (good): https://paste.ubuntu.com/p/QjnMPcJj7G/
* 20210511a manifest (bad): https://paste.ubuntu.com/p/PvJQwRXHcG/
* diff between manifests: https://paste.ubuntu.com/p/4nJtGxqGn7/
* serial logs of failed boot: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Cause:
shim changed the memory type for pages reserved for EFI runtime services, from EfiRuntimeServicesData to EfiBootServicesData.
Memory reserved for EFI runtime/boot services must be remapped as encrypted in the kernel (during boot) if SEV (secure encrypted virtualization) is enabled. The original kernel implementation of ioremap only correctly mapped the region as encrypted for EfiRuntimeServicesData regions, so when shim changed the type to EfiBootServicesData the kernel bug was exposed
# Fix:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d651ee9c71bb12fc0c8eb2786b66cbe5aa3e43b
# Regression potential
The fix could potentially cause boot failures, if a memory region is marked encrypted when it isn't. I assume it would cause a panic similar to the one seen here for this bug:
general protection fault, probably for non-canonical address 0x314836c31124d346: 0000 [#1] SMP NOPTI |
# Overview
Hirsute and Impish daily builds are currently not booting on Google Confidential Compute. Confidential compute is Google's platform that enables the use of Secure Encrypted Virtualization extension via AMD EPYC CPUs. Booting an image with version 1.45 works, but once upgraded to 1.47, the VM no longer boots, and instead the kernel panics.
Launching the image with secure boot, but without confidential compute works as expected.
# Expected result
The system is able to reboot after the upgrade.
# Actual result
Kernel panic: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Steps to reproduce
Launch a VM in GCE with confidential compute enabled with a serial v20210511a or later and look at the serial log for the kernel panic. Example CLI command to launch a VM:
$ gcloud beta compute instances create $USER-confidential-testing --zone=us-west1-b --machine-type=n2d-standard-2 --image=daily-ubuntu-2104-hirsute-v20210511a --image-project=ubuntu-os-cloud-devel --confidential-compute --maintenance-policy=TERMINATE
The last known good working image is daily-ubuntu-2104-hirsute-v20210510. The upgrade that fails is when shim signed is updated from 1.46+15.4-0ubuntu1 to 1.47+15.4-0ubuntu2
# Logs & notes
* 20210510 manifest (good): https://paste.ubuntu.com/p/QjnMPcJj7G/
* 20210511a manifest (bad): https://paste.ubuntu.com/p/PvJQwRXHcG/
* diff between manifests: https://paste.ubuntu.com/p/4nJtGxqGn7/
* serial logs of failed boot: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Cause:
shim changed the memory type for pages reserved for EFI runtime services, from EfiRuntimeServicesData to EfiBootServicesData.
Memory reserved for EFI runtime/boot services must be remapped as encrypted in the kernel (during boot) if SEV (secure encrypted virtualization) is enabled. The original kernel implementation of ioremap only correctly mapped the region as encrypted for EfiRuntimeServicesData regions, so when shim changed the type to EfiBootServicesData the kernel bug was exposed
# Fix:
Both EfiRuntimeServicesData and EfiBootServicesData must be mapped as encrypted if SEV is active, as per:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d651ee9c71bb12fc0c8eb2786b66cbe5aa3e43b
# Regression potential
The fix could potentially cause boot failures, if a memory region is marked encrypted when it isn't. I assume it would cause a panic similar to the one seen here for this bug:
general protection fault, probably for non-canonical address 0x314836c31124d346: 0000 [#1] SMP NOPTI |
|
| 2021-07-08 05:51:21 |
Khaled El Mously |
description |
# Overview
Hirsute and Impish daily builds are currently not booting on Google Confidential Compute. Confidential compute is Google's platform that enables the use of Secure Encrypted Virtualization extension via AMD EPYC CPUs. Booting an image with version 1.45 works, but once upgraded to 1.47, the VM no longer boots, and instead the kernel panics.
Launching the image with secure boot, but without confidential compute works as expected.
# Expected result
The system is able to reboot after the upgrade.
# Actual result
Kernel panic: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Steps to reproduce
Launch a VM in GCE with confidential compute enabled with a serial v20210511a or later and look at the serial log for the kernel panic. Example CLI command to launch a VM:
$ gcloud beta compute instances create $USER-confidential-testing --zone=us-west1-b --machine-type=n2d-standard-2 --image=daily-ubuntu-2104-hirsute-v20210511a --image-project=ubuntu-os-cloud-devel --confidential-compute --maintenance-policy=TERMINATE
The last known good working image is daily-ubuntu-2104-hirsute-v20210510. The upgrade that fails is when shim signed is updated from 1.46+15.4-0ubuntu1 to 1.47+15.4-0ubuntu2
# Logs & notes
* 20210510 manifest (good): https://paste.ubuntu.com/p/QjnMPcJj7G/
* 20210511a manifest (bad): https://paste.ubuntu.com/p/PvJQwRXHcG/
* diff between manifests: https://paste.ubuntu.com/p/4nJtGxqGn7/
* serial logs of failed boot: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Cause:
shim changed the memory type for pages reserved for EFI runtime services, from EfiRuntimeServicesData to EfiBootServicesData.
Memory reserved for EFI runtime/boot services must be remapped as encrypted in the kernel (during boot) if SEV (secure encrypted virtualization) is enabled. The original kernel implementation of ioremap only correctly mapped the region as encrypted for EfiRuntimeServicesData regions, so when shim changed the type to EfiBootServicesData the kernel bug was exposed
# Fix:
Both EfiRuntimeServicesData and EfiBootServicesData must be mapped as encrypted if SEV is active, as per:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d651ee9c71bb12fc0c8eb2786b66cbe5aa3e43b
# Regression potential
The fix could potentially cause boot failures, if a memory region is marked encrypted when it isn't. I assume it would cause a panic similar to the one seen here for this bug:
general protection fault, probably for non-canonical address 0x314836c31124d346: 0000 [#1] SMP NOPTI |
# Overview
Hirsute and Impish daily builds are currently not booting on Google Confidential Compute. Confidential compute is Google's platform that enables the use of Secure Encrypted Virtualization extension via AMD EPYC CPUs. Booting an image with version 1.45 works, but once upgraded to 1.47, the VM no longer boots, and instead the kernel panics.
Launching the image with secure boot, but without confidential compute works as expected.
# Expected result
The system is able to reboot after the upgrade.
# Actual result
Kernel panic: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Steps to reproduce
Launch a VM in GCE with confidential compute enabled with a serial v20210511a or later and look at the serial log for the kernel panic. Example CLI command to launch a VM:
$ gcloud beta compute instances create $USER-confidential-testing --zone=us-west1-b --machine-type=n2d-standard-2 --image=daily-ubuntu-2104-hirsute-v20210511a --image-project=ubuntu-os-cloud-devel --confidential-compute --maintenance-policy=TERMINATE
The last known good working image is daily-ubuntu-2104-hirsute-v20210510. The upgrade that fails is when shim signed is updated from 1.46+15.4-0ubuntu1 to 1.47+15.4-0ubuntu2
# Logs & notes
* 20210510 manifest (good): https://paste.ubuntu.com/p/QjnMPcJj7G/
* 20210511a manifest (bad): https://paste.ubuntu.com/p/PvJQwRXHcG/
* diff between manifests: https://paste.ubuntu.com/p/4nJtGxqGn7/
* serial logs of failed boot: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Cause:
shim changed the memory type for pages reserved for EFI runtime services, from EfiRuntimeServicesData to EfiBootServicesData.
Memory reserved for EFI runtime/boot services must be remapped as encrypted in the kernel (during boot) if SEV (secure encrypted virtualization) is enabled. The original kernel implementation of ioremap only correctly mapped the region as encrypted for EfiRuntimeServicesData regions, so when shim changed the type to EfiBootServicesData the kernel bug was exposed
# Fix:
Both EfiRuntimeServicesData and EfiBootServicesData must be mapped as encrypted if SEV is active, as per:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d651ee9c71bb12fc0c8eb2786b66cbe5aa3e43b
# Regression potential
The fix could potentially cause boot failures, if a memory region is marked encrypted when it shouldn't be. I assume in that case it would cause a panic similar to the one seen here for this bug:
general protection fault, probably for non-canonical address 0x314836c31124d346: 0000 [#1] SMP NOPTI |
|
| 2021-07-08 05:52:53 |
Khaled El Mously |
bug task added |
|
linux (Ubuntu) |
|
| 2021-07-08 05:53:03 |
Khaled El Mously |
bug task deleted |
linux-gcp (Ubuntu) |
|
|
| 2021-07-08 05:53:10 |
Khaled El Mously |
bug task deleted |
linux-gcp (Ubuntu Hirsute) |
|
|
| 2021-07-08 05:53:19 |
Khaled El Mously |
linux (Ubuntu): assignee |
|
Khaled El Mously (kmously) |
|
| 2021-07-08 05:53:22 |
Khaled El Mously |
linux (Ubuntu Hirsute): assignee |
|
Khaled El Mously (kmously) |
|
| 2021-07-08 05:53:27 |
Khaled El Mously |
linux (Ubuntu): status |
New |
In Progress |
|
| 2021-07-08 05:53:29 |
Khaled El Mously |
linux (Ubuntu Hirsute): status |
New |
In Progress |
|
| 2021-07-08 05:55:04 |
Khaled El Mously |
description |
# Overview
Hirsute and Impish daily builds are currently not booting on Google Confidential Compute. Confidential compute is Google's platform that enables the use of Secure Encrypted Virtualization extension via AMD EPYC CPUs. Booting an image with version 1.45 works, but once upgraded to 1.47, the VM no longer boots, and instead the kernel panics.
Launching the image with secure boot, but without confidential compute works as expected.
# Expected result
The system is able to reboot after the upgrade.
# Actual result
Kernel panic: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Steps to reproduce
Launch a VM in GCE with confidential compute enabled with a serial v20210511a or later and look at the serial log for the kernel panic. Example CLI command to launch a VM:
$ gcloud beta compute instances create $USER-confidential-testing --zone=us-west1-b --machine-type=n2d-standard-2 --image=daily-ubuntu-2104-hirsute-v20210511a --image-project=ubuntu-os-cloud-devel --confidential-compute --maintenance-policy=TERMINATE
The last known good working image is daily-ubuntu-2104-hirsute-v20210510. The upgrade that fails is when shim signed is updated from 1.46+15.4-0ubuntu1 to 1.47+15.4-0ubuntu2
# Logs & notes
* 20210510 manifest (good): https://paste.ubuntu.com/p/QjnMPcJj7G/
* 20210511a manifest (bad): https://paste.ubuntu.com/p/PvJQwRXHcG/
* diff between manifests: https://paste.ubuntu.com/p/4nJtGxqGn7/
* serial logs of failed boot: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Cause:
shim changed the memory type for pages reserved for EFI runtime services, from EfiRuntimeServicesData to EfiBootServicesData.
Memory reserved for EFI runtime/boot services must be remapped as encrypted in the kernel (during boot) if SEV (secure encrypted virtualization) is enabled. The original kernel implementation of ioremap only correctly mapped the region as encrypted for EfiRuntimeServicesData regions, so when shim changed the type to EfiBootServicesData the kernel bug was exposed
# Fix:
Both EfiRuntimeServicesData and EfiBootServicesData must be mapped as encrypted if SEV is active, as per:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d651ee9c71bb12fc0c8eb2786b66cbe5aa3e43b
# Regression potential
The fix could potentially cause boot failures, if a memory region is marked encrypted when it shouldn't be. I assume in that case it would cause a panic similar to the one seen here for this bug:
general protection fault, probably for non-canonical address 0x314836c31124d346: 0000 [#1] SMP NOPTI |
# Overview
Hirsute and Impish daily builds are currently not booting on Google Confidential Compute. Confidential compute is Google's platform that enables the use of Secure Encrypted Virtualization extension via AMD EPYC CPUs. Booting an image with version 1.45 works, but once upgraded to 1.47, the VM no longer boots, and instead the kernel panics.
Launching the image with secure boot, but without confidential compute works as expected.
# Expected result
The system is able to reboot after the upgrade.
# Actual result
Kernel panic: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Steps to reproduce
Launch a VM in GCE with confidential compute enabled with a serial v20210511a or later and look at the serial log for the kernel panic. Example CLI command to launch a VM:
$ gcloud beta compute instances create $USER-confidential-testing --zone=us-west1-b --machine-type=n2d-standard-2 --image=daily-ubuntu-2104-hirsute-v20210511a --image-project=ubuntu-os-cloud-devel --confidential-compute --maintenance-policy=TERMINATE
The last known good working image is daily-ubuntu-2104-hirsute-v20210510. The upgrade that fails is when shim signed is updated from 1.46+15.4-0ubuntu1 to 1.47+15.4-0ubuntu2
# Logs & notes
* 20210510 manifest (good): https://paste.ubuntu.com/p/QjnMPcJj7G/
* 20210511a manifest (bad): https://paste.ubuntu.com/p/PvJQwRXHcG/
* diff between manifests: https://paste.ubuntu.com/p/4nJtGxqGn7/
* serial logs of failed boot: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Cause:
shim changed the memory type for pages reserved for EFI runtime services, from EfiRuntimeServicesData to EfiBootServicesData.
Memory reserved for EFI runtime/boot services must be remapped as encrypted in the kernel (during boot) if SEV (secure encrypted virtualization) is enabled. The original kernel implementation of ioremap only correctly mapped the region as encrypted for EfiRuntimeServicesData regions, so when shim changed the type to EfiBootServicesData the kernel bug was exposed
Note that this affects all 5.11 kernels not just gcp
# Fix:
Both EfiRuntimeServicesData and EfiBootServicesData must be mapped as encrypted if SEV is active, as per:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d651ee9c71bb12fc0c8eb2786b66cbe5aa3e43b
# Regression potential
The fix could potentially cause boot failures, if a memory region is marked encrypted when it shouldn't be. I assume in that case it would cause a panic similar to the one seen here for this bug:
general protection fault, probably for non-canonical address 0x314836c31124d346: 0000 [#1] SMP NOPTI |
|
| 2021-07-08 05:56:57 |
Khaled El Mously |
description |
# Overview
Hirsute and Impish daily builds are currently not booting on Google Confidential Compute. Confidential compute is Google's platform that enables the use of Secure Encrypted Virtualization extension via AMD EPYC CPUs. Booting an image with version 1.45 works, but once upgraded to 1.47, the VM no longer boots, and instead the kernel panics.
Launching the image with secure boot, but without confidential compute works as expected.
# Expected result
The system is able to reboot after the upgrade.
# Actual result
Kernel panic: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Steps to reproduce
Launch a VM in GCE with confidential compute enabled with a serial v20210511a or later and look at the serial log for the kernel panic. Example CLI command to launch a VM:
$ gcloud beta compute instances create $USER-confidential-testing --zone=us-west1-b --machine-type=n2d-standard-2 --image=daily-ubuntu-2104-hirsute-v20210511a --image-project=ubuntu-os-cloud-devel --confidential-compute --maintenance-policy=TERMINATE
The last known good working image is daily-ubuntu-2104-hirsute-v20210510. The upgrade that fails is when shim signed is updated from 1.46+15.4-0ubuntu1 to 1.47+15.4-0ubuntu2
# Logs & notes
* 20210510 manifest (good): https://paste.ubuntu.com/p/QjnMPcJj7G/
* 20210511a manifest (bad): https://paste.ubuntu.com/p/PvJQwRXHcG/
* diff between manifests: https://paste.ubuntu.com/p/4nJtGxqGn7/
* serial logs of failed boot: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Cause:
shim changed the memory type for pages reserved for EFI runtime services, from EfiRuntimeServicesData to EfiBootServicesData.
Memory reserved for EFI runtime/boot services must be remapped as encrypted in the kernel (during boot) if SEV (secure encrypted virtualization) is enabled. The original kernel implementation of ioremap only correctly mapped the region as encrypted for EfiRuntimeServicesData regions, so when shim changed the type to EfiBootServicesData the kernel bug was exposed
Note that this affects all 5.11 kernels not just gcp
# Fix:
Both EfiRuntimeServicesData and EfiBootServicesData must be mapped as encrypted if SEV is active, as per:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d651ee9c71bb12fc0c8eb2786b66cbe5aa3e43b
# Regression potential
The fix could potentially cause boot failures, if a memory region is marked encrypted when it shouldn't be. I assume in that case it would cause a panic similar to the one seen here for this bug:
general protection fault, probably for non-canonical address 0x314836c31124d346: 0000 [#1] SMP NOPTI |
# Overview
Hirsute and Impish daily builds are currently not booting on Google Confidential Compute. Confidential compute is Google's platform that enables the use of Secure Encrypted Virtualization extension via AMD EPYC CPUs. Booting an image with version 1.45 works, but once upgraded to 1.47, the VM no longer boots, and instead the kernel panics.
Launching the image with secure boot, but without confidential compute works as expected.
# Expected result
The system is able to reboot after the upgrade.
# Actual result
Kernel panic: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Steps to reproduce
Launch a VM in GCE with confidential compute enabled with a serial v20210511a or later and look at the serial log for the kernel panic. Example CLI command to launch a VM:
$ gcloud beta compute instances create $USER-confidential-testing --zone=us-west1-b --machine-type=n2d-standard-2 --image=daily-ubuntu-2104-hirsute-v20210511a --image-project=ubuntu-os-cloud-devel --confidential-compute --maintenance-policy=TERMINATE
The last known good working image is daily-ubuntu-2104-hirsute-v20210510. The upgrade that fails is when shim signed is updated from 1.46+15.4-0ubuntu1 to 1.47+15.4-0ubuntu2
# Logs & notes
* 20210510 manifest (good): https://paste.ubuntu.com/p/QjnMPcJj7G/
* 20210511a manifest (bad): https://paste.ubuntu.com/p/PvJQwRXHcG/
* diff between manifests: https://paste.ubuntu.com/p/4nJtGxqGn7/
* serial logs of failed boot: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Cause:
shim changed the memory type for pages reserved for EFI runtime services, from EfiRuntimeServicesData to EfiBootServicesData.
Memory reserved for EFI runtime/boot services must be remapped as encrypted in the kernel (during boot) if SEV (secure encrypted virtualization) is enabled. The original kernel implementation of ioremap only correctly mapped the region as encrypted for EfiRuntimeServicesData regions, so when shim changed the type to EfiBootServicesData the kernel bug was exposed
Note that this affects all 5.11 kernels not just gcp. It is possible that gcp is the only cloud that uses sev currently (for "Confidential Computing").
# Fix:
Both EfiRuntimeServicesData and EfiBootServicesData must be mapped as encrypted if SEV is active, as per:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d651ee9c71bb12fc0c8eb2786b66cbe5aa3e43b
# Regression potential
The fix could potentially cause boot failures, if a memory region is marked encrypted when it shouldn't be. I assume in that case it would cause a panic similar to the one seen here for this bug:
general protection fault, probably for non-canonical address 0x314836c31124d346: 0000 [#1] SMP NOPTI |
|
| 2021-07-08 06:10:38 |
Khaled El Mously |
description |
# Overview
Hirsute and Impish daily builds are currently not booting on Google Confidential Compute. Confidential compute is Google's platform that enables the use of Secure Encrypted Virtualization extension via AMD EPYC CPUs. Booting an image with version 1.45 works, but once upgraded to 1.47, the VM no longer boots, and instead the kernel panics.
Launching the image with secure boot, but without confidential compute works as expected.
# Expected result
The system is able to reboot after the upgrade.
# Actual result
Kernel panic: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Steps to reproduce
Launch a VM in GCE with confidential compute enabled with a serial v20210511a or later and look at the serial log for the kernel panic. Example CLI command to launch a VM:
$ gcloud beta compute instances create $USER-confidential-testing --zone=us-west1-b --machine-type=n2d-standard-2 --image=daily-ubuntu-2104-hirsute-v20210511a --image-project=ubuntu-os-cloud-devel --confidential-compute --maintenance-policy=TERMINATE
The last known good working image is daily-ubuntu-2104-hirsute-v20210510. The upgrade that fails is when shim signed is updated from 1.46+15.4-0ubuntu1 to 1.47+15.4-0ubuntu2
# Logs & notes
* 20210510 manifest (good): https://paste.ubuntu.com/p/QjnMPcJj7G/
* 20210511a manifest (bad): https://paste.ubuntu.com/p/PvJQwRXHcG/
* diff between manifests: https://paste.ubuntu.com/p/4nJtGxqGn7/
* serial logs of failed boot: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Cause:
shim changed the memory type for pages reserved for EFI runtime services, from EfiRuntimeServicesData to EfiBootServicesData.
Memory reserved for EFI runtime/boot services must be remapped as encrypted in the kernel (during boot) if SEV (secure encrypted virtualization) is enabled. The original kernel implementation of ioremap only correctly mapped the region as encrypted for EfiRuntimeServicesData regions, so when shim changed the type to EfiBootServicesData the kernel bug was exposed
Note that this affects all 5.11 kernels not just gcp. It is possible that gcp is the only cloud that uses sev currently (for "Confidential Computing").
# Fix:
Both EfiRuntimeServicesData and EfiBootServicesData must be mapped as encrypted if SEV is active, as per:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d651ee9c71bb12fc0c8eb2786b66cbe5aa3e43b
# Regression potential
The fix could potentially cause boot failures, if a memory region is marked encrypted when it shouldn't be. I assume in that case it would cause a panic similar to the one seen here for this bug:
general protection fault, probably for non-canonical address 0x314836c31124d346: 0000 [#1] SMP NOPTI |
# Overview
Hirsute and Impish daily builds are currently not booting on Google Confidential Compute. Confidential compute is Google's platform that enables the use of Secure Encrypted Virtualization extension via AMD EPYC CPUs. Booting an image with version 1.45 works, but once upgraded to 1.47, the VM no longer boots, and instead the kernel panics.
Launching the image with secure boot, but without confidential compute works as expected.
# Expected result
The system is able to reboot after the upgrade.
# Actual result
Kernel panic: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Steps to reproduce
Launch a VM in GCE with confidential compute enabled with a serial v20210511a or later and look at the serial log for the kernel panic. Example CLI command to launch a VM:
$ gcloud beta compute instances create $USER-confidential-testing --zone=us-west1-b --machine-type=n2d-standard-2 --image=daily-ubuntu-2104-hirsute-v20210511a --image-project=ubuntu-os-cloud-devel --confidential-compute --maintenance-policy=TERMINATE
The last known good working image is daily-ubuntu-2104-hirsute-v20210510. The upgrade that fails is when shim signed is updated from 1.46+15.4-0ubuntu1 to 1.47+15.4-0ubuntu2
# Logs & notes
* 20210510 manifest (good): https://paste.ubuntu.com/p/QjnMPcJj7G/
* 20210511a manifest (bad): https://paste.ubuntu.com/p/PvJQwRXHcG/
* diff between manifests: https://paste.ubuntu.com/p/4nJtGxqGn7/
* serial logs of failed boot: https://paste.ubuntu.com/p/mHrvVc6qBc/
# Cause:
shim changed the memory type for pages reserved for EFI runtime services, from EfiRuntimeServicesData to EfiBootServicesData.
Memory reserved for EFI runtime/boot services must be remapped as encrypted in the kernel (during boot) if SEV (secure encrypted virtualization) is enabled. The original kernel implementation of ioremap only correctly mapped the region as encrypted for EfiRuntimeServicesData regions, so when shim changed the type to EfiBootServicesData the kernel bug was exposed
Note that this affects all 5.11 kernels not just gcp. It is possible that gcp is the only cloud that uses sev currently (for "Confidential Computing").
# Fix:
Both EfiRuntimeServicesData and EfiBootServicesData must be mapped as encrypted if SEV is active, as per:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8d651ee9c71bb12fc0c8eb2786b66cbe5aa3e43b
# Test
Without the fix applied, confirmed that I was able to reproduce the issue described here (complete failure to boot, kernel panic)
With fix, confirmed no issues booting
# Regression potential
The fix could potentially cause boot failures, if a memory region is marked encrypted when it shouldn't be. I assume in that case it would cause a panic similar to the one seen here for this bug:
general protection fault, probably for non-canonical address 0x314836c31124d346: 0000 [#1] SMP NOPTI |
|
| 2021-07-08 06:11:02 |
Khaled El Mously |
summary |
Google Confidential Compute fails to boot with 1.47 |
Google Confidential Compute fails to boot with shim version 1.47 |
|
| 2021-07-22 23:51:29 |
Ubuntu Kernel Bot |
tags |
|
verification-needed-hirsute |
|
| 2021-07-23 18:10:13 |
Joshua Powers |
tags |
verification-needed-hirsute |
verification-done-hirsute |
|
| 2021-08-03 14:36:08 |
Ubuntu Kernel Bot |
tags |
verification-done-hirsute |
verification-done-hirsute verification-needed-focal |
|
| 2021-08-06 08:24:55 |
Kleber Sacilotto de Souza |
bug task added |
|
linux-gcp (Ubuntu) |
|
| 2021-08-06 08:25:38 |
Kleber Sacilotto de Souza |
bug task deleted |
linux-gcp (Ubuntu) |
|
|
| 2021-08-06 08:26:29 |
Kleber Sacilotto de Souza |
bug task added |
|
linux-gcp (Ubuntu) |
|
| 2021-08-06 08:26:54 |
Kleber Sacilotto de Souza |
bug task deleted |
linux (Ubuntu Hirsute) |
|
|
| 2021-08-06 08:27:00 |
Kleber Sacilotto de Souza |
nominated for series |
|
Ubuntu Hirsute |
|
| 2021-08-06 08:27:00 |
Kleber Sacilotto de Souza |
bug task added |
|
linux (Ubuntu Hirsute) |
|
| 2021-08-06 08:27:00 |
Kleber Sacilotto de Souza |
bug task added |
|
linux-gcp (Ubuntu Hirsute) |
|
| 2021-08-06 08:27:08 |
Kleber Sacilotto de Souza |
linux (Ubuntu Hirsute): status |
New |
In Progress |
|
| 2021-08-06 08:27:28 |
Kleber Sacilotto de Souza |
linux (Ubuntu Hirsute): assignee |
|
Khaled El Mously (kmously) |
|
| 2021-08-06 08:27:42 |
Kleber Sacilotto de Souza |
linux-gcp (Ubuntu Hirsute): status |
New |
Fix Released |
|
| 2021-08-06 08:31:15 |
Kleber Sacilotto de Souza |
linux (Ubuntu): status |
In Progress |
Fix Released |
|
| 2021-08-06 08:32:09 |
Kleber Sacilotto de Souza |
linux-gcp (Ubuntu): status |
New |
Fix Committed |
|
| 2021-08-06 19:54:03 |
Kelsey Steele |
linux (Ubuntu Hirsute): status |
In Progress |
Fix Committed |
|
| 2021-08-18 23:09:34 |
Launchpad Janitor |
linux-gcp (Ubuntu): status |
Fix Committed |
Fix Released |
|
| 2021-08-18 23:09:34 |
Launchpad Janitor |
cve linked |
|
2021-33909 |
|
| 2022-01-26 22:01:41 |
Brian Murray |
linux (Ubuntu Hirsute): status |
Fix Committed |
Won't Fix |
|
