TPM event log does not contain events measured after ExitBootServices

[Jordan Hand]

The TPM event log (/sys/kernel/security/tpm0/binary_bios_measurements) does not contain any events that are measured by UEFI after the kernel's EFI Boot stub calls ExitBootServices().

This means that PCR values calculated from the event log will not match the actual PCR values on the machine for PCR indices into which these events are measured.

There are upstream patches to fix this in the mainline kernel tree: https://<email address hidden>/
---
ProblemType: Bug
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2019-06-20 (43 days ago)
InstallationMedia: Ubuntu-Server 18.04.2 LTS "Bionic Beaver" - Release amd64 (20190210)
IwConfig:
 eth0 no wireless extensions.

 lo no wireless extensions.
Lspci:

Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: Microsoft Corporation Virtual Machine
Package: linux (not installed)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 hyperv_fb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.18.0-22-generic root=UUID=fcb8dc9d-4dd3-490f-9f1c-fa6364770bb0 ro
ProcVersionSignature: Ubuntu 4.18.0-22.23~18.04.1-generic 4.18.20
RelatedPackageVersions:
 linux-restricted-modules-4.18.0-22-generic N/A
 linux-backports-modules-4.18.0-22-generic N/A
 linux-firmware 1.173.3
RfKill:

Tags: bionic
Uname: Linux 4.18.0-22-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin lxd plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 01/30/2019
dmi.bios.vendor: Microsoft Corporation
dmi.bios.version: Hyper-V UEFI Release v4.0
dmi.board.asset.tag: None
dmi.board.name: Virtual Machine
dmi.board.vendor: Microsoft Corporation
dmi.board.version: Hyper-V UEFI Release v4.0
dmi.chassis.asset.tag: 8486-4870-7514-9524-5524-7794-69
dmi.chassis.type: 3
dmi.chassis.vendor: Microsoft Corporation
dmi.chassis.version: Hyper-V UEFI Release v4.0
dmi.modalias: dmi:bvnMicrosoftCorporation:bvrHyper-VUEFIReleasev4.0:bd01/30/2019:svnMicrosoftCorporation:pnVirtualMachine:pvrHyper-VUEFIReleasev4.0:rvnMicrosoftCorporation:rnVirtualMachine:rvrHyper-VUEFIReleasev4.0:cvnMicrosoftCorporation:ct3:cvrHyper-VUEFIReleasev4.0:
dmi.product.family: Virtual Machine
dmi.product.name: Virtual Machine
dmi.product.sku: None
dmi.product.version: Hyper-V UEFI Release v4.0
dmi.sys.vendor: Microsoft Corporation

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1838796

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Jordan Hand (jorhand)]

apport information

[Jordan Hand (jorhand)]

apport information

[Jordan Hand (jorhand)]

apport information

[Jordan Hand (jorhand)]

apport information

[Jordan Hand (jorhand)]

apport information

[Jordan Hand (jorhand)]

apport information

[Jordan Hand (jorhand)]

apport information

[Jordan Hand (jorhand)]

apport information

[Jordan Hand (jorhand)]

apport information

[Tyler Hicks (tyhicks)]

Hi Jordan - This seems like a nice enhancement and something that will be in Ubuntu once we are shipping a kernel that's v5.3 or newer. I don't expect that we'll backport these patches to our stable releases that ship kernels older than v5.3. This seems to reflect the TPM subsystem maintainer's thoughts as he did not target these patches for linux-stable.

If you feel like these patches should be backported, please provide additional justification. Otherwise, we'll have this functionality in a future release. Thanks!

[Marcelo Cerri (mhcerri)]

A 4.15 test kernel is available for validation with the backported patches: https://kernel.ubuntu.com/~mhcerri/azure/lp1838796.1/

[VINAY RAJESH (vinaykotak)]

I have verified the kernel image provided above. The PCR5 values in the TCG logs and in the TPM match. I have also verified that the ExitBootServices event is present in the binary_bios_measurements. However, I see there is a mismatch for PCR4 and PCR7 between the TCG logs and the TPM values. I am not sure if that is expected or is it something to be concerned about.

PCR4 logs the EFI Service Application events. Attaching screenshots of the PCR values and PCR4 log events for your reference.

[Chris Coulson (chrisccoulson)]

I briefly tested the kernels and I'm seeing that the log is consistent with the PCR values in the TPM. May I ask what tool it is you're using in those screenshots so that I can try it?

[prashant (prash200)]

Hi Chris,

I repeated the experiment with the above Kernel, but PCR#7 still doesn't match.

I am using a custom tool to parse binary_bios_measurements. Attaching the binary_bios_measurements binary and parsed XML for your reference. Can you please try to parse the binary using your tool and check if the values in tpm2_pcrread.out file match? According to the tool I am using, PCR5 matches but not PCR7.

Something to note is that I have a custom key in MOK (so did Vinay), moreover MokList gets extended in PCR7 (at least in the logs). Can this the reason of discrepancies?

-Prashant

[VINAY RAJESH (vinaykotak)]

Hi Chris,

Can you please point me to the parser tool that you used to parse the binary_bios_measurements? We can try that tool at our end to see if the our tool has a bug.

[Chris Coulson (chrisccoulson)]

I think the reason for your issue is that the final 2 events extended to PCR7 are recorded twice in the log, most likely because the test kernel from comment 12 doesn't contain https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=166a2809d65b282272c474835ec22c882a39ca1b

I didn't see the same issue because I'm testing on a configuration with a version of shim that doesn't call GetEventLog() before starting grub (see https://github.com/rhboot/shim/commit/fd7c3bd920ba39082cb7c619afb7203d150a4cd3), and so the final 2 events that shim record don't end up in the final events table and aren't duplicated.

Note that some additional follow-up changes would be required too - eg, https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b61fbc887af7a13a1c90c84c1feaeb4c9780e1e2, https://<email address hidden>/, https://<email address hidden>/ and https://<email address hidden>/

[VINAY RAJESH (vinaykotak)]

Hi Chris,

There are few observations we made while testing.

1. On baseline Ubuntu, we see a PCR7 mismatch. Could you please confirm if this is a known issue and what is the reason for this mismatch?

2. We were able to validate that there were duplicate entries in the TCG logs with the test kernel and extending those entries in the PCR matched the TCG log PCR values. But the same is not true for the baseline Ubuntu, we did not see duplicate values in the baseline Ubuntu measurements. Does the test kernel try to fix the PCR7 mismatch too and also introduces a regression because of duplicate entries?

3. We also noticed that there are no bios measurements exposed by the kernel when secure boot is turned off. Is it possible to get bios measurements in that scenario, indicating that secure boot is turned off?

[Chris Coulson (chrisccoulson)]

Hi,

In response to your queries:

1) With kernel version 5.0.0-37, I can confirm that the event log provided by the kernel is inconsistent with the TPM for PCR7 in a VM that's running OVMF. This is because of the opposite problem - in this case, the last event is missing from the log exported by the kernel. I'm not sure why that is yet because it occurs before ExitBootServices() and should appear in the firmware's main event log.

There is also a mismatch for PCR5, but this one is expected because the kernel is missing events that occur as a result of or after ExitBootServices() (in this case, it misses 2 EV_EFI_ACTION events). This is the problem that will be addressed by this bug report.

2) The stock kernel for Ubuntu 18.04 doesn't export duplicate events in the event log because it doesn't contain any code to handle the final events table (to retrieve events that are recorded as a result of or after ExitBootServices()). The duplicate events occur in the test kernel with the patches in comment 12 applied because it misses some additional fixes to de-duplicate events that are recorded both to the firmware's main event log and the final events table. Events that occur between the first call to GetEventLog() and ExitBootServices() are recorded by the firmware to both places.

3) I think this is a GRUB issue. AFAICT, GRUB's linux loader only boots the kernel via its EFI stub when secure boot is enabled, and I think you need to boot the kernel with the EFI stub in order for it to retrieve the event log.

[Chris Coulson (chrisccoulson)]

I just noticed I didn't respond to the question in comment 16. The tool I'm using is https://github.com/chrisccoulson/tcglog-parser

[Chris Coulson (chrisccoulson)]

Ok, I think that the truncated log issue with kernel version 5.0.0-37 is a bug in tpm1_bios_measurements_next() which is fixed by https://lore.kernel.org/patchwork/patch/1031236/, although I've not verified that this is the case.

[Marcelo Cerri (mhcerri)]

I'm preparing a new test kernel with the additional patches that Chris has mentioned. I will let you know once I have it ready.

[Marcelo Cerri (mhcerri)]

I preliminary test kernel with the missing patches is available at: https://kernel.ubuntu.com/~mhcerri/azure/lp1838796.2/

I will be running some tests on it on the next days.

[Marcelo Cerri (mhcerri)]

The complete set of patches for the test kernel above: https://kernel.ubuntu.com/~mhcerri/azure/lp1838796.2/patches/

[VINAY RAJESH (vinaykotak)]

Hi Marcelo,

Can you please let us know when you are with the tests on your side? We can then go ahead and validate the test kernel at our end.

-Vinay

[VINAY RAJESH (vinaykotak)]

Hi Marcelo,

I tried to validate the test kernel provided by you in comment 23. I am not able to load the kernel. When I select the kernel from the grub menu, the loading gets stuck at "Loading initial ramdisk".

I tried it with secure boot disabled too, just to be sure we are not making any mistakes with the signing part.

I am not sure if I am missing something here. Can you please help resolve this?

Thanks
Vinay

[Marcelo Cerri (mhcerri)]

Hi, Vinay.

I managed to install and boot the test kernel on a gen2 hyper-v VM on a Win10 host. What's the environment you are using?

On my tests I noticed the kernel is failing to retrieve the event log from the firmware. So I was wondering if this setup I'm using is the best option or if I should move to something close to the targeted environment.

[VINAY RAJESH (vinaykotak)]

Hi Marcelo,

I am trying to load the kernel on an x86_64 physical machine. Here is how I installed the .deb pkg on the machine.

"sudo dpkg -i linux-modules-4.15.0-1066-azure_4.15.0-1066.71+lp1838796.2_amd64.deb"

"sudo dpkg -i linux-image-unsigned-4.15.0-1066-azure_4.15.0-1066.71+lp1838796.2_amd64.deb"

I rebooted the system after this and tried to select the kernel from the grub menu.

I am not sure what I am missing.

Here is the output of the os-release file on the machine.

NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Also, the targeted environment will be Hyper V. We are running into some Hyper V issues and thus I am trying to validate the kernel on a physical machine.

-Vinay

[Marcelo Cerri (mhcerri)]

Hi Vinay. I never tried to boot the azure kernel on a physical machine, but I believe it should boot fine. Usually with those test kernels I simply install all the debian package with:

$ sudo apt install ./*.deb

I will try to install it on physical machine today to check if I have the same issues.

I usually provide .deb packages for test kernels because they are quick to build. However, I can provide a test kernel on a PPA if you prefer. Building the kernel on a PPA has the advantage that we can sign the kernel image for secure boot. However by default PPAs do not sign kernel images with our official key. In that case I usually add the PPA key to the firmware so I can boot the whole stack in secure mode (just keep in mind that our grub in xenial is still not enforcing the kernel signature).

With regards to my tests do you see any issues with my environment (Hyper-V gen2 VM on Win10 Pro host with secure boot and vTPM enabled)?

[Marcelo Cerri (mhcerri)]

Vinay, I just noticed you are using Bionic for this test. I believe a 4.15 kernel might boot ok in bionic, but the test kernel is actually intended to Xenial.

For Bionic we need to test the 5.3 linux-azure-edge kernel that can be installed directly from the archive via:

$ sudo apt install linux-azure-edge

[VINAY RAJESH (vinaykotak)]

Hi Marcelo,

Thanks for the information. I will try and validate the Linux-azure-edge kernel.

Regarding your test environment, there are no issues. This is the expected environment for the guest OS.

-Vinay

[VINAY RAJESH (vinaykotak)]

Hi Marcelo,

I am facing the same issue as I was with the .deb packages. When I run "sudo apt install Linux-azure-edge" and reboot, the kernel does not boot.

I am able to boot into the Linux 5.3.040-generic kernel but not the azure edge kernel.

That said, I tried it on both the physical machine and Hyper v with secure boot enabled. The kernel boots fine on Hyper V but not on the physical machine.

Did you get a chance to test it on a physical machine?

-Vinay

[Marcelo Cerri (mhcerri)]

Hi, Vinay.

I tried but then I realized that all linux-azure kernels were stripped down and they will not boot on a regular bare metal machine. But I will test linux-azure-edge on a Hyper-v machine and I will let you know.

[VINAY RAJESH (vinaykotak)]

That sounds good. I will try and test it at end too.

Thanks a lot for your help :)

-Vinay

[VINAY RAJESH (vinaykotak)]

Hi Marcelo,

I tested the Linux-azure-edge kernel at my end and I was able to verify that the PCR value 0 through 7 match.

Thanks a lot for your help and support.

Thanks
Vinay