[ehl] Shutdown hangs on board

[Doug Jacobs]

[Summary]Go to Power -> Shutdown and the screen goes blank, but the system does not shut down. You cannot recover and have to perform a hard reboot.

[Steps to reproduce]
1. Boot up Ubuntu
2. Power -> Shutdown (click ok)

[Expected result]
System should cleanly shut down.

[Actual result]
Screen blanks with a blinking cursor, the system does not shut down, the fan is still on. System is unresponsive, forcing you to do a hard reset.

[Failure rate]100%

[Additional information]
CID: 20215-29063
SKU:
system-manufacturer: Intel Corporation
system-product-name: Elkhart Lake Embedded Platform
bios-version: EHLSFWI1.R00.2091.A00.2002250754
CPU: Genuine Intel(R) CPU 0000 @ 1.90GHz (4x)
GPU: 00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:4571]
kernel-version: 5.11.0-1009-intel

[Stage]
Device froze, issue reported and logs collected right after a reboot

[Doug Jacobs (djacobs98)]

Automatically attached

[Doug Jacobs (djacobs98)]

Automatically attached

[Doug Jacobs (djacobs98)]

Automatically attached

[Doug Jacobs (djacobs98)]

Automatically attached

[Anthony Wong (anthonywong)]

needs to revert "pinctrl: intel: Add Intel Elkhart Lake pin controller support" until there is BIOS fix.

[Sachin Mokashi (sachinmokashi)]

This is a known issue on EHL, please track the solution here:
https://github.com/thesofproject/linux/issues/2828

As mentioned in the link, Please disable CONFIG_PINCTRL_ELKHARTLAKE=y for EHL until MR2 BIOS is released.

No need to revert the patch in this case

[Hsuan-Yu Lin (shanelin)]

Test passed [1] after reverting "pinctrl: intel: Add Intel Elkhart Lake pin controller support"

CID: 202105-29064
BIOS: EHLSFWI1.R00.2091.A00.2002250754

---
[1] The CPU fan stops. But some LEDs are on.

[Hsuan-Yu Lin (shanelin)]

Hi @Doug,

Please use this test kernel to verify this issue.

version: 5.11.0-1011-intel #12+lp1938408~lp1938413

[Installation steps]
1. tar -zxvf revert.tgz
2. cd revert
3. sudo dpkg -i *.deb

[Doug Jacobs (djacobs98)]

System shuts down properly after this workaround. Fan stops, board LEDs remain on (system has no LED power indicator.) System boots properly upon unplugging/plugging back in.

[Doug Jacobs (djacobs98)]

I spoke too soon.

If you continually shutdown/boot the system, it will hang, about 20% of the time, at the splash screen.

[Hsuan-Yu Lin (shanelin)]

Hi @Doug,

Regarding this issue and bug 1938408, could you try to use the old kernel to see if this patch caused the system hang issue?

[Doug Jacobs (djacobs98)]

Which is the Old Kernel you want me to try?

All these bugs were filed using
iotg-focal-core-20210722.img (5.11.0-1011-intel)
I then used the patch you gave me on this kernel and retested.

I have managed to have the system hang on booting after a shutdown twice, but after another 10 shutdowns, it hasn't had any problems.

I will repeat this with the older releases:
iotg-focal-core-20210629.img
iotg-focal-core-20210714.img

[Doug Jacobs (djacobs98)]

Actually I made a mistake. I filed these with iotg-focal-desktop-20210706.img

This is the only image I've used on the EHL board. I got confused with the TGL-H board which I've used both Desktop and Core.

So, What is the old kernel you want me to use?

[Hsuan-Yu Lin (shanelin)]

Regarding comment #7, No need to revert the patch in this case

Test passed (5/5) by disabling CONFIG_PINCTRL_ELKHARTLAKE and remove the kernel module.

I will upload a new kernel for testing later.

[Hsuan-Yu Lin (shanelin)]

Patches are ready and under review.

[Hsuan-Yu Lin (shanelin)]

Hi @Doug,

I have uploaded a new test kernel on my ppa, please refer to [1] or follow the steps below to verify:

[Installation steps]
$ sudo add-apt-repository ppa:shanelin/linux-intel
$ sudo apt-get update
$ sudo apt-get install linux-image-unsigned-5.11.0-1011-intel=5.11.0-1011.13+lp1938408~lp1938413~0805 linux-modules-5.11.0-1011-intel=5.11.0-1011.13+lp1938408~lp1938413~0805 linux-modules-extra-5.11.0-1011-intel=5.11.0-1011.13+lp1938408~lp1938413~0805

---
[1] https://launchpad.net/~shanelin/+archive/ubuntu/linux-intel
[2] Test passed (5/5) for:
CID: 202105-29064
BIOS: EHLSFWI1.R00.2091.A00.2002250754

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Hsuan-Yu Lin (shanelin)]

Test passed (5/5) for:
CID: 202105-29104
BIOS: EHLSFWI1.R00.3044.A01.2101210945
Kernel: $ uname -a
Linux u-Elkhart-Lake-Embedded-Platform 5.11.0-1012-intel #14-Ubuntu SMP Wed Aug 11 08:52:10 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[Doug Jacobs (djacobs98)]

Verified fixed with 5.13.0-1007.

[Alex Henrie (alexhenrie24)]

I am having the same poweroff/reboot problem with linux-image-5.15.0-1016-intel-iotg, but I am able to work around it with initcall_blacklist=ehl_pinctrl_driver_init.