[iotg][ehl][ehl-aaeon] Fail to install Ubuntu Core on AAEON EHL when BIOS supports SM3 256

[Pierre Equoy]

CID: 202109-29496
SKU: AAEON UPN-EHL01

Image used: ubuntu-core-20-amd64+intel-iot.img.xz (20211014.2)

Steps to reproduce
==================

1. Using a Ubuntu Live USB, flash /dev/mmcblk0 with the image:

xzcat ubuntu-core-20-amd64+intel-iot.img.xz | sudo dd of=/dev/mmcblk0 bs=32M status=progress; sync

2. Reboot

Expected results
================

UC20 is installed and the setup screens are shown (to setup networks and credentials).

Actual results
==============

At step 1, the content of /dev/mmcblk0 is a 1.2GB partition called `ubuntu-seed`, which seems OK.

The device boots, then shows "error: no such device ubuntu-boot", then goes on to boot something until it's left with the error messages displayed on the screen (see snapshot attached).

---
Related issue: LP: #1938678

[Pierre Equoy (pieq)]

As suggested by Ethan, I went into the BIOS and disabled TPM2 completely.

The installation now proceeds as expected.

[Pierre Equoy (pieq)]

As requested by Ethan:

$ snap list
Name Version Rev Tracking Publisher Notes
core20 20211015 1215 latest/edge canonical✓ base
intel-kernel 5.13.0-1007.7.1 7 20/edge canonical✓ kernel
pc 20-0.4 115 20/edge canonical✓ gadget
snapd 2.52.1+git1353.g6d1a7e7 13739 latest/edge canonical✓ snapd

[Pierre Equoy (pieq)]

In the BIOS > Advanced > Truster Computing, I can enable/disable the following PCR Banks:

- SHA256
- SHA384
- SM3_256

When I filed this bug, the status was:

- SHA256: Enabled
- SHA384: Disabled
- SM3_256: Disabled

After disabling TPM2 entirely, I could successfully install UC20 (see comment #3).

I then tried to enable all the aforementioned PCR banks, and restart UC20 in Recovery mode (by long pressing the `1` key), but I see the error message (see attached screenshot for more): "error locking access to sealer keys: cannot execute hash sequence: TPM returned an invalid response for command TPM_CC_EventSequenceComplete: cannot unmarshal response parameters: cannot unmarshal argument at index 0: cannot process list type tpm2.TaggedHashList:m2.TaggedHash, inside container type tpm2.TaggedHashList: cannot determine digest size for unknown algorithm TPM_ALG_SM3_256"

[Kent Lin (kent-jclin)]

The issue seems related to the secure boot and TPM 2.0.
On AAEON's EHL machine the secure boot could not be enabled.
The Key is not inserted in BIOS. Does FDE in Core required secure boot be enabled or not?

If change TPM setting in BIOS to TPM 1.2. Then the Ubuntu core could be installed successfully.
Advanced -> Trusted Computing -> Device Select

But if you go to BIOS, Advanced -> Trusted Computing, it will show no security device found.

[ethan.hsieh (ethan.hsieh)]

@Kent
In x86 machines, secure boot and TPM are required for UC20 FDE.

[Kent Lin (kent-jclin)]

My Advantech EHL board is broken. I asked Advantech QA help to verify with https://cdimage.ubuntu.com/ubuntu-core/20/stable/current/ubuntu-core-20-amd64+intel-iot.img.xz and wiating his feedback.

I tested Aaeon's TGL-U board (Up-Xtreme i11-UPX-TGL01)[1] which use same TPM chip as Advantech, the issue could not be reproduced and the FDE is enabled as expected.

So in summary, the root cause of this issue is because there is a regression in the snapd in edge channel. Since we should only release the image and snap from the stable channel, so this issue will not block the release.

[1]:https://certification.canonical.com/hardware/202109-29435/

[Pierre Equoy (pieq)]

Retested with latest image:

CID: 202109-29496
SKU: AAEON UPN-EHL01

Image used: ubuntu-core-20-amd64+intel-iot.img.xz (20211020.4)

Manifest file:

snapd 13270
core20 1169
intel-kernel 7
pc 115

I have the same problem as in original description.

Disabling TPM2 entirely allows me to install UC20. Otherwise, the installation blocks at the step shown in the original screenshot taken, no matter if I enable/disable "SM3 256 PCR Bank" in the BIOS.

[ethan.hsieh (ethan.hsieh)]

I dumped binary_bios_measurements on a production board (Aaeon EHL) and didn't see "UEFI Debug Mode" from the log. With snapd (2.53.1) and patched kernel snap, I can install UC20 with FDE enabled without any problem.
For detailed log, please see attached file in [1]

---
Board: Aaeon UPN-EHL01 (This board support SM3 256.)
BIOS: UNEHAM0D 5.19
BIOS Release Date: 09/02/2021

---
[1] https://bugs.launchpad.net/intel/+bug/1938678/comments/48

[ethan.hsieh (ethan.hsieh)]

I can install uc20 beta image[1] on my Aaeon UPN-EHL01 which supports SM3.

$ cat /writable/system-data/var/lib/snapd/seed/.disk/info
20211117.3

$ snap list
Name Version Rev Tracking Publisher Notes
core20 20211117 1245 latest/beta canonical✓ base
intel-kernel 5.13.0-1007.7.3 10 20/beta canonical✓ kernel
pc 20-0.4 115 20/beta canonical✓ gadget
snapd 2.53.2 14066 latest/beta canonical✓ snapd

$ lsblk | grep crypt
│ └─ubuntu-save-17a0870c-5a59-4c9f-9213-2121f85849ce 253:1 0 9M 0 crypt /writable/system-data/var/lib/snapd/save
  └─ubuntu-data-7b4c30e7-88e1-4a88-bc8a-d95e41fd456b 253:0 0 56.3G 0 crypt /run/mnt/base/writable

---
Board: Aaeon UPN-EHL01 (This board support SM3 256.)
BIOS: UNEHAM0D 5.19
BIOS Release Date: 09/02/2021
---
[1] https://cdimage.ubuntu.com/ubuntu-core/20/beta/current/ubuntu-core-20-amd64+intel-iot.img.xz

[ethan.hsieh (ethan.hsieh)]

The issue is fixed by latest snapd and intel-kernel.
$ snap info snapd | grep stable
  latest/stable: 2.53.2 2021-11-24 (14066) 44MB -
$ snap info intel-kernel | grep stable
  20/stable: 5.13.0-1007.7.3 2021-11-18 (10) 309MB -

[Pierre Equoy (pieq)]

Board: Aaeon UPN-EHL01
CID: 202109-29496
BIOS: UNEHAM0D 5.19

Encryption methods:

- SHA256: Enabled
- SHA384: Enabled
- SM3_256: Disabled

Secure Boot Enabled.

Image version:
$ cat /writable/system-data/var/lib/snapd/seed/.disk/info
20211125.4

$ snap list
Name Version Rev Tracking Publisher Notes
core20 20211115 1242 latest/stable canonical✓ base
intel-kernel 5.13.0-1008.8.1 11 20/stable canonical✓ kernel
pc 20-0.4 115 20/stable canonical✓ gadget
snapd 2.53.2 14066 latest/stable canonical✓ snapd

$ lsblk | grep crypt
│ └─ubuntu-save-6d51425c-d129-4ed9-91dd-6450251f849b 253:1 0 9M 0 crypt /writable/system-data/var/lib/snapd/save
  └─ubuntu-data-d032b466-f7ed-405b-af05-49571c261a5c 253:0 0 56.3G 0 crypt /run/mnt/base/writable

The installation completes, disk is encrypted using TPM2 and system can be accessed through SSH.