[intel] [tgl-h][iotg] [hwe-tpm] Fail to install Ubuntu Core on TGL-H when BIOS supports SM3 256

[Doug Jacobs]

Installed focal-iotg-core-20210722.img on the TGL-H.

Boot process stops after it gets errors from TPM and fails to start an emergency shell (see screen shot.)

I also tried to go into the BIOS to clear TPM, as this worked for the EHL board but it still behaves the same.

Once it reaches this point, the keyboard is unresponsive (cannot soft reboot, get to a terminal window, etc.)

Error message when installing uc20 with secure boot and TPM enabled:
the-tool[336]: error: error locking access to sealed keys: cannot execute hash sequence: TPM returned an invalid response for command TPM_CC_EventSequenceComplete: cannot unmarshal response parameters: cannot unmarshal argument at index 0: cannot process list type tpm2.TaggedHashList: cannot process element at index 3 from list type tpm2.TaggedHashList: cannot process custom type tpm2.TaggedHash, inside container type tpm2.TaggedHashList: cannot determine digest size for unknown algorithm TPM_ALG_SM3_256

https://cs.opensource.google/go/x/crypto

Public bug: https://github.com/canonical/go-tpm2/issues/7

[Chao Qin (chaoqin)]

@Doung Jacobs

Could you share with me the link for focal-iotg-core-20210722.img? And we can have a try to reproduce this issue.

And could you guide me about how to replace kernel in Ubuntu core?

[Doug Jacobs (djacobs98)]

The core image I used is located at: https://private-fileshare.canonical.com/~brian/iotg-focal-core-20210722.img

(There is no central repository yet for this project.)

Instructions for installation: I use 2 USB drives.

Make 1 USB bootable using the Startup Disk Creator (or RUFUS if you're using Windows.) Ubuntu Desktop 20.04 is fine. This is your boot drive.

Copy the .img file to the 2nd drive. This is your data drive.

Boot the TGL-H system from your boot drive. Don't install. Just select "Try Ubuntu."
Use the Disks utility to delete all partitions from the internal storage of the TGL-H.

Insert the data drive and use dd to write the image to the internal storage (/dev/nvme01 I think?)
$ sudo dd if=/path/to/img of=/dev/nvme01 bs=1M status=progress

Once this completes, you can reboot the system and remove the USB drives.

During bootup, you will see the errors from TPM, and eventually the boot process just stops.

[Chao Qin (chaoqin)]

It seems I have no permission to access the image link. Could you please grant me access?

[Sachin Mokashi (sachinmokashi)]

Hi @djacobs98:

Response from Intel BIOS Engineer:

From the error screen, it looks some tools-services are failed to load because of unknown TPM hash Algo, SM3_256.
Some security modules/tools in OS are not signed with correct hash algo or not using SM3_256 algo during signing process.

Please give a try using below BIOS settings:

Go to Advanced menu> TPM configuration > TCG2 Configuration > enable PCR Bank  PCR Bank: SM3_256 [x ]

Or can try with different combinations. ( disable SHA256 and check what is the error while booting )

[Doug Jacobs (djacobs98)]

I tried enabling SM3_256 with SHA256, just enabling SM3_256, and disabling all 4.

I get the same error message the-tool:
"error locking access to sealed keys: caannot execute hash sequence: TPM returned an invalid response for command TPM_CC_EventSequenceComplete: cannot unmarshal response parameters: cannot unmarshal argument at index 0: cannot process list type tpm2.TaggedHashList: cannot process element at index 3 from list type tpm2.TaggedHashList: cannot process custom type tpm2.TaggedHash, inside container type tpm2.TaggedHashList: cannot determine digest size for unknown algorithm TPM_ALG_SM3_256"

[Sachin Mokashi (sachinmokashi)]

Hi @djacobs98,

Can you please answer the queries from our BIOS Engineer:

Are you using PTT or discrete TPM? If discrete TPM, then which one?
PTT does support SMX crypto, but needs to enabled in the BIOS.

[Doug Jacobs (djacobs98)]

TPM in this BIOS supports 4 types of PCR: SHA1, SHA256, SHA384, SM3_256.

SHA256 the default.

Next I tried SHA256 & SM3_256

Next I tried all 4 OFF

Finally I tried all 4 ON.

The results each time is the same. When the system boots it displays a message asking if I want to use the new TPM settings (F12 to accept) and then the system starts to boot up, getting the same error message I posted before. The PCR settings don't seem to matter.

[Doug Jacobs (djacobs98)]

According to the BIOS, this TGL-H board does not have a discrete TPM chip. It's running TPM in firmware fTPM 2.0, PTT is active.

[Chao Qin (chaoqin)]

@Doug Jacobs,

There is similar issue for algorithm TPM_ALG_SHA3_256 (0x0027). Maybe need snapd experts' help on this.
https://bugs.launchpad.net/snapd/+bug/1925410

[Doug Jacobs (djacobs98)]

I also tried the latest IOTG core image (iotg-focal-core-unencrypted-20210804.img) on the EHL board. It behaves in the same manner as the TGL-H board.

It is also running fTPM with TPP active, and not using a discrete TPM chip (although it appears to have one.)

[Sachin Mokashi (sachinmokashi)]

Hi @djacobs98,

As mentioned in comment #10, can you please loop in snapd experts to look into this issue as well.

[Sachin Mokashi (sachinmokashi)]

Hi @djacobs98,

Can you please try out this query from our BIOS Engineer -

What are your observations when you turn off only the SM bank? I suspect the message is from the early boot software that does not have support for SM algorithms

[Sachin Mokashi (sachinmokashi)]

Hi @ivan.hu,

As you had mentioned in https://bugs.launchpad.net/intel/+bug/1936899/comments/10,
that tpm2-tss does not support algorithm sm3_256

Can you please take a look at this issue what Canonical team is facing. From the screenshot it looks like Ubuntu Core tpm-tss tools is trying to check for SM_256 and is failing to boot

[ethan.hsieh (ethan.hsieh)]

https://github.com/canonical/go-tpm2/commit/bd7cad4936577a496ebfca7272fef34c32e3bc0b

commit bd7cad4936577a496ebfca7272fef34c32e3bc0b
Author: Chris Coulson <email address hidden>
Date: Tue Aug 10 17:46:55 2021 +0100

    Use HashAlgorithmId instead of crypto.Hash in various places

    Some recent commits changed HashAlgorithmId to crypto.Hash in
    some public APIs (mostly in util/), but this isn't really the
    right thing to do - it creates inconsistency (some arguments
    are represented by go-tpm2 types and others by go types),
    and the TPM spec currently has one hash algorithm with no
    corresponding crypto.Hash value (SM3). Whilst it's not possible
    to use this algorithm in go-tpm2 right now even though there
    is a go implementation of it (it requires it to be registered
    with the crypto package), it may be possible to support it
    in the future regardless of whether it gets a corresponding
    crypto.Hash value.

    Switch the use of crypto.Hash back to HashAlgorithmId in
    various places.

[Ivan Hu (ivan.hu)]

Snapd is not using the tpm2-tss, so it is not the same as mentioned in https://bugs.launchpad.net/intel/+bug/1936899/comments/10.

Instead, snapd is using go-tpm2 from comment#15.
Per talk to Ethan, he is building newer version which has some defined values of SM3_256 for testing, let's wait for the testing result.

[ethan.hsieh (ethan.hsieh)]

The latest go-tpm2[1] seems to support SM3_256.
I tried to build test snapd and kernel snaps but got a dependency issue.
The latest secboot[2] still uses old go-tpm2 API.

I applied the attached patch in comment#17 to go-tpm2[3] which is currently used by snapd.
And, I built a test image with patched snapd and kernel snaps.

Unfortunately, the FDE function still doesn't work with this test image.
The new error message is:
the-tool[334]: panic: crypto: requested hash function #0 is unavailable.
(For details, please refer to the attached photo in comment#18)

As the commit in comment#15 mentioned, the TPM spec currently has one hash algorithm with no corresponding crypto.Hash[4] value (SM3). Whilst it's not possible to use this algorithm in go-tpm2 right now even though there is a go implementation of it.

So, the UC image doesn't support SM3_256 now because Go cryptography libraries[4] doesn't support it.

---
[1] https://github.com/canonical/go-tpm2/blob/master/types_interface.go#L61
[2] https://github.com/snapcore/secboot
[3] go-tpm2, comment id: 32171bd353b113ff4793dc3c65a019d749674bc6
[4] https://cs.opensource.google/go/x/crypto

[ethan.hsieh (ethan.hsieh)]

@sachinmokashi

Could Intel build a BIOS with an option for disabling hash algorithm SM3 256?

[Sachin Mokashi (sachinmokashi)]

Hi @ethan.hsieh,

As mentioned in comment #5 above, there is an option to disable SM3 256 algorithm. Please check #5 for a screenshot of the BIOS option

Please give a try and uncheck SM3_256 using below BIOS settings:

Go to Advanced menu> TPM configuration > TCG2 Configuration > enable PCR Bank  PCR Bank: SM3_256 [x ]

[ethan.hsieh (ethan.hsieh)]

@sachinmokashi

There are two fields in BIOS:
1. Hash Algorithm Bitmap
2. Active PCR Banks

The one you mentioned in comment#21 is "Active PCR Banks".
But, what I want to disable is "Hash Algorithm".
Current BIOS doesn't has an option to disable Hash Algorithm SM3 256.
Could Intel build a BIOS with an option for disabling hash algorithm SM3 256?

[Ivan Hu (ivan.hu)]

The setting,
Go to Advanced menu> TPM configuration > TCG2 Configuration > enable PCR Bank  PCR Bank: SM3_256 [x ]

is for PCR Bank supported Hash Algorithm.

Here, what Ethan mentioned is for remove the SM3_256 supported for the platform, not only the PCR bank SM3_256 supported.

[Sachin Mokashi (sachinmokashi)]

@ethan.hsieh @ivan.hu,

Response from our BIOS/TPM Engineers:

"Hash Algorithm Bitmap is set in BIOS and it’s based on build configuration PCD and active PCR’s supported."

This BIOS is also distributed as reference BIOS for IBVs and hence disabling SM3 would break functionality on other Platforms/BIOS

The way to go would be to enable SM3 on go-tpm2 and tpm2-tss tools, I see your request to enable SM3 in go-tpm2/secboot here:
https://github.com/canonical/go-tpm2/issues/7

[ethan.hsieh (ethan.hsieh)]

@sachinmokashi

I see. I'm trying to enable SM3 on go-tpm2.
Fixing the issue on crypto package, go-tpm2, and secboot is the right way to go.
But, It may take several weeks to enable it.
That is why I'm seeking for a short-term solution, having a BIOS option for disabling SM3".

[Kent Lin (kent-jclin)]

@sachinmokashi,

SM3 is forbidden to ship to some countries. Why Intel will not provide the option to disable it in BIOS?

[Chris Coulson (chrisccoulson)]

I've not read every comment in detail, but I think there is a bit of misunderstanding about what the firmware options discussed here actually do.

Disabling the SM3_256 PCR bank will stop the firmware measuring events to the TPM using SM3_256 and will omit SM3_256 digests from the event log. I assume that the firmware also makes use of the TPM2_PCR_Allocate command to disable all of the PCRs in the SM3_256 bank.

What it does not do is disable SM3_256. If you use the TPM2_PCR_Event or TPM2_EventSequenceComplete command, the TPM will still compute digests for SM3_256 and will respond with a TPML_DIGEST_VALUES structure containing SM3_256 digests. This is where the issue is - because of the way that TPML_DIGEST_VALUES is designed (it doesn't contain sizes in the payload), go-tpm2 needs to know the size of SM3_256 in order to decode the response from the TPM, and it currently doesn't because it relies on go's standard library for this, and that also doesn't support SM3_256.

There is no TCG defined API that would allow platform firmware to disable a digest algorithm via an option in the firmware UI - the TPM's supported algorithms are defined at build time.

[Sachin Mokashi (sachinmokashi)]

+ Adding Intel TPM Expert <email address hidden>

[Chris Coulson (chrisccoulson)]

Both https://github.com/canonical/go-tpm2/commit/96eb110220ece5922dc7b691422fff12735f1880 and https://github.com/snapcore/secboot/pull/166 are intended to resolve the issue in this bug report.

[ethan.hsieh (ethan.hsieh)]

@Chris

I backported your patches to secboot[1] and go-tpm2[2] which are currently used[3] by snapd.
With test image which includes your patches, I can install uc20 when the hash algorithm SM3 is enabled in BIOS.

But, the encrypted partition is still not created successfully because the test kernel snap is unsigned.
When I install uc20, I have to disable secure boot.

I'm trying to build a signed kernel snap and install uc20 with secure boot enabled.
I will keep you informed of the progress.

---
[1] https://github.com/EthanHsieh/secboot/commit/f1b9e1593b2a952be77b63f8b31cc3787b1e3e0d
[2] https://github.com/EthanHsieh/go-tpm2/commit/b5a8526eb2402689024c0deeba778733036a3084
[3] https://github.com/snapcore/snapd/blob/master/vendor/vendor.json

[ethan.hsieh (ethan.hsieh)]

@Chris

The encrypted partition is created after I enroll the signing key by mokutil and enable secure boot.

But, I get new error message (For details, please refer to attached photo):
taskrunner.go: 271: [change 2 "Setup system for run mode" task] failed: cannot make system runnable: cannot seal the encryption keys: cannot add EFI secure boot policy profile: cannot compute secure boot policy profile: secure boot configuration was modified after the initial configuration was measured, without performing a reboot.

Did I miss some patches when I backported patches to secboot/go-tpm2 which are currently used by snapd?

[ethan.hsieh (ethan.hsieh)]

Still can reproduce the issue in comment#31 after cleaning TPM with following steps:

1. Go to BIOS settings
2. [Intel Advanced Menu][TPM Configuration][TCG2 Configuration][TPM2 Operation]
3. Select [TPM2 ClearControl(NO) + Clear]
4. Press F4 to save changes
5. Reboot device

[ethan.hsieh (ethan.hsieh)]

@sachinmokashi
Are steps in comment#32 the correct way to clear TPM?

[Doug Jacobs (djacobs98)]

That is what I was told to do in the past.

[Chris Coulson (chrisccoulson)]

@ethan.hsieh That error message is unexpected, but it doesn't matter too much anyway - there's no support at all for computing PCR digests for systems that boot kernels that are verified with a MOK. The only way to test kernels signed with non-production keys is to take control of the device's signature database (delete the platform key and enroll your own db, KEK and then enroll a custom PK to re-enable secure boot).

[ethan.hsieh (ethan.hsieh)]

Enroll KEK and Signature via BIOS settings but still get the same error message (comment#31).
But, I can install the test image on another x86 machine and FDE is enabled successfully.

Here are steps:
1. Remove key enrolled by mokutil
2. Re-flash uc20 test image
3. Enroll KEK and Signature via BIOS settings:
[Boot Maintenance Manager Menu][Secure Boot Configuration Menu][Secure Boot Mode][Custom Mode][Custom Secure Boot Option]
[KEK Option][Enroll KEK] => PkKek-1-snakeoil.der[1]
[DB Option][Enroll Signature] => PkKek-1-snakeoil.der
4. Clear TPM
[Intel Advanced Menu][TPM Configuration][TCG2 Configuration][TPM2 Operation]

Another x86 machine has different BIOS, so it has different steps to clear TPM and enroll key.
1. Flash uc20 test image
2. Enroll KEK and Signature via BIOS settings:
[Security][Secure Boot][Key Management]
[Key Exchange Keys][Append] => PkKek-1-snakeoil.der
[Authorized Signatures][Append] => PkKek-1-snakeoil.der
3. Clear TPM
$ sudo -s
$ echo 5 > /sys/class/tpm/tpm0/ppi/request

---
[1] PkKek-1-snakeoil
https://raw.githubusercontent.com/snapcore/pc-amd64-gadget/20/snakeoil/PkKek-1-snakeoil.key
https://raw.githubusercontent.com/snapcore/pc-amd64-gadget/20/snakeoil/PkKek-1-snakeoil.pem
# Convert PkKek-1-snakeoil.pem to PkKek-1-snakeoil.der
$ openssl x509 -in PkKek-1-snakeoil.pem -inform PEM -outform DER -out PkKek-1-snakeoil.der

[ethan.hsieh (ethan.hsieh)]

@Chris

I deleted and re-enrolled PK on another intel platform, EHL, and still saw same error message.
For details, please refer to [1].

---
[1] https://bugs.launchpad.net/intel/+bug/1939505/comments/3

[ethan.hsieh (ethan.hsieh)]

@Chris

I used another x86 machine on which I can install uc20 successfully with same test image.
I enrolled the key by mokutil and re-installed uc20.
I got following error message:
taskrunner.go: 271: [change 2 "Setup system for run mode" task] failed: cannot make system runnable: cannot seal the encryption keys: cannot add EFI secure boot policy profile: cannot compute secure boot policy profile: no bootable paths with current EFI signature database.

The error message is different from #31.
Based on test results, the root cause of #31 should not be the key enrolled by mokutil.

[ethan.hsieh (ethan.hsieh)]

Summary:

1. Still can reproduce the issue (#31) after deleting PK and enrolling db/KEK/custom PK vis BIOS settings.
For details, please refer to #38 and [1].

2. Can install the uc20 test image[2][3] with TPM and secure boot enabled on other x86 machines (not TGL-H and EHL).
And, I only enrolled KEK and DB via BIOS settings.
For details, please refer to #36

3. When I enroll the key by mokutil, the error message is different from #31.
For details, please refer to #39.

@Chris
Any thoughts on this?
Based on above test results, it looks like #31 is a BIOS issue?

---
[1] https://bugs.launchpad.net/intel/+bug/1939505/comments/3
[2] https://github.com/EthanHsieh/secboot/commit/f1b9e1593b2a952be77b63f8b31cc3787b1e3e0d
[3] https://github.com/EthanHsieh/go-tpm2/commit/b5a8526eb2402689024c0deeba778733036a3084

[Chris Coulson (chrisccoulson)]

Can I see the event log from this device after booting with secure boot on please? (/sys/kernel/security/tpm0/binary_bios_measurements)

[ethan.hsieh (ethan.hsieh)]

@Chris
The files in comment#42 and #43: I dumped them in Ubuntu classic desktop

[Chris Coulson (chrisccoulson)]

Thanks. The issue is that the firmware provides a debugger which breaks the PCR calculations. I'm not sure whether it's actually desirable to fix this or detect it and provide a better error message given that the ability to attach a debugger defeats any protections offered by full-disk encryption.

[Chris Coulson (chrisccoulson)]

Helps if I add the file

[Sachin Mokashi (sachinmokashi)]

@Chris, Response from Intel TPM folks for #44 -

"
I presume by firmware they mean BIOS.

There is no debugger mode left enabled in normal production flows or SKUs for CSME.
To enable CSME debug mode you'd first unlock the part with an IDLM. I seriously doubt that the part they have is unlocked.
"

Can you please clarify which debugger that you are referring to?

[Chris Coulson (chrisccoulson)]

From the TCG log supplied in comment #43:

$ ./tcglog-dump --alg sha256 --verbose --pcrs 7 ~/Downloads/binary_bios_measurements
 7 a62bd67b2cc295976651b354468c0047f8d1547d25056ded5952aaf5991762a3 EV_EFI_ACTION [ UEFI Debug Mode ]
 7 ccfc4bb32888a345bc8aeadaba552b627d99348c767681ab3141f5b01e40a40e EV_EFI_VARIABLE_DRIVER_CONFIG [ UEFI_VARIABLE_DATA{ VariableName: 8be4df61-93ca-11d2-aa0d-00e098032b8c, UnicodeName: "SecureBoot" } ]
 7 bdac662ac2f50e28cc04a493f47e24f46d18d9dfc6aed0ac41f267380b533194 EV_EFI_VARIABLE_DRIVER_CONFIG [ UEFI_VARIABLE_DATA{ VariableName: 8be4df61-93ca-11d2-aa0d-00e098032b8c, UnicodeName: "PK" } ]
 7 08bef4adae58358b14c751cdddd8123da1f64f5ea85407622f6be90d4621d958 EV_EFI_VARIABLE_DRIVER_CONFIG [ UEFI_VARIABLE_DATA{ VariableName: 8be4df61-93ca-11d2-aa0d-00e098032b8c, UnicodeName: "KEK" } ]
 7 9538dfd2bfb2105587901e9b5f867fc6652fb3972d3c6ce05f4b7dad11f7912b EV_EFI_VARIABLE_DRIVER_CONFIG [ UEFI_VARIABLE_DATA{ VariableName: d719b2cb-3d3a-4596-a3bc-dad00e67656f, UnicodeName: "db" } ]
 7 5c959286c9a5906201fc201930034b6438a5282bfd6542f38dde3384e0b448e0 EV_EFI_VARIABLE_DRIVER_CONFIG [ UEFI_VARIABLE_DATA{ VariableName: d719b2cb-3d3a-4596-a3bc-dad00e67656f, UnicodeName: "dbx" } ]
 7 df3f619804a92fdb4057192dc43dd748ea778adc52bc498ce80524c014b81119 EV_SEPARATOR
 7 4d4a8e2c74133bbdc01a16eaf2dbb5d575afeb36f5d8dfcf609ae043909e2ee9 EV_EFI_VARIABLE_AUTHORITY [ UEFI_VARIABLE_DATA{ VariableName: d719b2cb-3d3a-4596-a3bc-dad00e67656f, UnicodeName: "db" } ]
 7 922e939a5565798a5ef12fe09d8b49bf951a8e7f89a0cca7a51636693d41a34d EV_EFI_VARIABLE_AUTHORITY [ UEFI_VARIABLE_DATA{ VariableName: 605dab50-e046-4300-abb6-3dd810dd8b23, UnicodeName: "SbatLevel" } ]
 7 5e19450c7a75acd95f6af49d0e32b74142972d9dd4c1b8068450653683a13016 EV_EFI_VARIABLE_AUTHORITY [ UEFI_VARIABLE_DATA{ VariableName: 605dab50-e046-4300-abb6-3dd810dd8b23, UnicodeName: "Shim" } ]

The first event there is a EV_EFI_ACTION event with the string "UEFI Debug Mode". Our PCR policy calculations depend on this event not being there for it to work for full disk-encryption. From section 2.3.4.8 of the TCG PC Client Platform Firmware Profile Specification:

"If the platform provides a firmware debugger mode which may be used prior to the
UEFI environment or if the platform provides a debugger for the UEFI environment,
then the platform SHALL extend an EV_EFI_ACTION event into PCR[7] before
allowing use of the debugger. The event string SHALL be “UEFI Debug Mode”. The
Platform Firmware MUST log this measurement in the event log using the string
“UEFI Debug Mode” for the Event Data."

So presence of the event indicates that the platform firmware provides a firmware debugger.

[ethan.hsieh (ethan.hsieh)]

I dumped binary_bios_measurements on a production board (Aaeon EHL) and didn't see "UEFI Debug Mode" from the log. With snapd (2.53.1) and patched kernel snap, I can install UC20 with FDE enabled without any problem.
For detailed log, please see attached file.

---
Board: Aaeon UPN-EHL01
BIOS: UNEHAM0D 5.19
BIOS Release Date: 09/02/2021

[ethan.hsieh (ethan.hsieh)]

The issue is fixed by latest snapd and intel-kernel.
$ snap info snapd | grep stable
  latest/stable: 2.53.2 2021-11-24 (14066) 44MB -
$ snap info intel-kernel | grep stable
  20/stable: 5.13.0-1007.7.3 2021-11-18 (10) 309MB -