Lenovo X13s fails to boot on 6.8.0-20-generic and 6.8.0-22-generic

[Heinrich Schuchardt]

The last kernel that successfully boots on my Lenovo X13s system is 6.5.0-1004-laptop.

With 6.8.0-20-generic no output whatsoever is shown.
---
ProblemType: Bug
ApportVersion: 2.28.0-0ubuntu1
Architecture: arm64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/seq: zfsdt 2886 F.... pipewire
CRDA: N/A
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
DistroRelease: Ubuntu 24.04
InstallationDate: Installed on 2023-12-12 (120 days ago)
InstallationMedia: Ubuntu 24.04 LTS "Noble Numbat" - Daily arm64+x13s (20231212)
Lspci-vt:
 -[0002:00]---00.0-[01-ff]----00.0 KIOXIA Corporation NVMe SSD Controller BG5 (DRAM-less)
 -[0004:00]---00.0-[01-ff]----00.0 Foxconn International, Inc. T99W175 5G Modem [Snapdragon X55]
 -[0006:00]---00.0-[01-ff]----00.0 Qualcomm Technologies, Inc QCNFA765 Wireless Network Adapter
MachineType: LENOVO 21BXCTO1WW
Package: linux (not installed)
ProcEnviron:
 LANG=en_US.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm-256color
ProcFB: 0 msmdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-6.5.0-1004-laptop root=UUID=7695b3a4-a033-451f-ba6b-56e81a873b79 ro quiet splash clk_ignore_unused pd_ignore_unused vt.handoff=7
ProcVersionSignature: Ubuntu 6.5.0-1004.7-laptop 6.5.3
RelatedPackageVersions:
 linux-restricted-modules-6.5.0-1004-laptop N/A
 linux-backports-modules-6.5.0-1004-laptop N/A
 linux-firmware 20240318.git3b128b60-0ubuntu2
Tags: noble
Uname: Linux 6.5.0-1004-laptop aarch64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
acpidump:

dmi.bios.date: 12/05/2023
dmi.bios.release: 1.59
dmi.bios.vendor: LENOVO
dmi.bios.version: N3HET87W (1.59 )
dmi.board.asset.tag: Not Available
dmi.board.name: 21BXCTO1WW
dmi.board.vendor: LENOVO
dmi.board.version:
 SDK0T76463 WIN
 ptal����8
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: None
dmi.ec.firmware.release: 1.23
dmi.modalias: dmi:bvnLENOVO:bvrN3HET87W(1.59):bd12/05/2023:br1.59:efr1.23:svnLENOVO:pn21BXCTO1WW:pvrThinkPadX13sGen1:rvnLENOVO:rn21BXCTO1WW:rvrSDK0T76463WINptal8:cvnLENOVO:ct10:cvrNone:skuLENOVO_MT_21BX_BU_Think_FM_ThinkPadX13sGen1:
dmi.product.family: ThinkPad X13s Gen 1
dmi.product.name: 21BXCTO1WW
dmi.product.sku: LENOVO_MT_21BX_BU_Think_FM_ThinkPad X13s Gen 1
dmi.product.version: ThinkPad X13s Gen 1
dmi.sys.vendor: LENOVO

[Heinrich Schuchardt (xypron)]

apport information

[Heinrich Schuchardt (xypron)]

apport information

[Heinrich Schuchardt (xypron)]

apport information

[Heinrich Schuchardt (xypron)]

apport information

[Heinrich Schuchardt (xypron)]

apport information

[Heinrich Schuchardt (xypron)]

apport information

[Heinrich Schuchardt (xypron)]

apport information

[Heinrich Schuchardt (xypron)]

apport information

[Heinrich Schuchardt (xypron)]

apport information

[Heinrich Schuchardt (xypron)]

apport information

[Heinrich Schuchardt (xypron)]

apport information

[Heinrich Schuchardt (xypron)]

apport information

[Heinrich Schuchardt (xypron)]

apport information

[Heinrich Schuchardt (xypron)]

apport information

[Heinrich Schuchardt (xypron)]

6.8.0-22-generic with earlycon=efifb stops after the output in the appended screenshot.

[Aaron Honeycutt (aaronhoneycutt)]

I'm able to recreate this issue as well. I also want to note that the 24.04 LTS Beta also does not boot with the "Try or Install" option in GRUB but it does boot the ISO.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/2060868

[Bogdan Melnik (ld86)]

I think I have the same issue which translates to the fact that I can't boot the x13s image from https://cdimage.ubuntu.com/daily-live/current/ (2024-04-17 06:17).

[Anton Leshchenko (antonl1911)]

arm64.nopauth regulator_ignore_unused modprobe.blacklist=qcom_q6v5_pas

are required to boot from usb, but then there is a problem with USB, resulting in an endless loop.

[Juerg Haefliger (juergh)]

The daily x13s images are broken. I'm working on it.

[Jens Glathe (glathe)]

Hmm I see two issues here.

qcom_q6v5_pas should be in /etc/initramfs-tools/modules (it is in mine), or loading of adsp and cdsp firmware will be deferred until rootfs is mounted, with the odd VBUS interruption on type-c connectors at a moment where this will result in an inaccessible rootfs afterwards.
The endless loop is something I'm not seeing with my builds, looked like something IOMMU and DWC3 that were clashing.

[Heinrich Schuchardt (xypron)]

@glathe

We used to have packate linux-image-laptop-23.10 which provided file /usr/share/initramfs-tools/modules.d/laptop. This contained qcom_q6v5_pas. It have recreated this file manually but this does not resolve my problems.

Which follow up package provided with the X13s installer image defines the modules to be included in the initrd?

[Aaron Honeycutt (aaronhoneycutt)]

I've seen a report of the 6.9-rc5 kernel booting on the hardware. I have also had issues with Fedora Rawhide and NixOS booting on the hardware so I think it is an upstream issue with the kernel.

[Jens Glathe (glathe)]

Hi @xypron,

my /etc/initramfs-tools/modules is this:

-------------------------------------------
pwm_bl
phy_qcom_qmp_pcie
pcie_qcom
phy_qcom
qmp_pcie
phy_qcom_qmp_combo
qrtr
drm-dp-aux-bus
phy_qcom_edp
gpio_sbu_mux
i2c_hid_of
i2c_qcom_geni
pmic_glink_altmode
leds_qcom_lpg
qcom_q6v5_pas
panel-edp
gpucc_sc8280xp
dispcc_sc8280xp
msm
nvme
usb_storage
uas
------------------------------

I had severe issues booting from type-c due to late-loading of qcadsp8280.mbn, therefore I experimented a little until found. I am not familiar with the installer at all, so I'm afraid I'm no help there. But the loading of the firmwares goes hand in hand with my /etc/initramfs-tools/hooks/x13s-firmware script:

-----------------------------
#!/bin/sh

set -e

PREREQ=""

prereqs()
{
 echo "$PREREQ"
}

case \\$1 in
# get pre-requisites
prereqs)
 prereqs
 exit 0
 ;;
esac

. /usr/share/initramfs-tools/hook-functions

mv /lib/firmware/updates/qcom/sc8280xp/LENOVO/21BX/qcadsp8280.mbn.disabled /lib/firmware/updates/qcom/sc8280xp/LENOVO/21BX/qcadsp8280.mbn

# Define a list of firmware files to be included
FIRMWARE_FILES="\
/lib/firmware/updates/qcom/sc8280xp/LENOVO/21BX/qcadsp8280.mbn \
/lib/firmware/updates/qcom/sc8280xp/LENOVO/21BX/qccdsp8280.mbn \
/lib/firmware/updates/qcom/sc8280xp/LENOVO/21BX/qcdxkmsuc8280.mbn \
/lib/firmware/updates/qcom/sc8280xp/LENOVO/21BX/qcvss8280.mbn \
/lib/firmware/updates/qcom/sc8280xp/LENOVO/21BX/qcslpi8280.mbn \
/lib/firmware/qcom/a660_sqe.fw.zst \
/lib/firmware/qcom/a660_gmu.bin.zst"

# Copy each firmware file to initramfs
for file in $FIRMWARE_FILES; do
    dir=$(dirname "$file")
    mkdir -p "${DESTDIR}/${dir}"
    cp "${file}" "${DESTDIR}/${dir}/"
done

mv /lib/firmware/updates/qcom/sc8280xp/LENOVO/21BX/qcadsp8280.mbn /lib/firmware/updates/qcom/sc8280xp/LENOVO/21BX/qcadsp8280.mbn.disabled

-----------------------------------

This ensures the crucial files (and some not so crucial) are in the initramfs and accessible before rootfs can be mounted. My boot tests found that you actually run into trouble occasionally when you:

- have a slow boot device, like the stick plugged in swapped condition (then it's USB-2 only)
- have qcadsp8280.mbn in the rootfs as well.

"Late" attempts to load this blob (after mounting rootfs from type-c) will lead to the [VBUS issue](https://gitlab.com/postmarketOS/pmaports/-/blob/master/device/testing/firmware-lenovo-yoga-5g/APKBUILD?ref_type=heads#L44-46) which I experienced for a few ... weeks ... and pulled my hair out. Disabling the blob in the rootfs mitigates the issue to tolerable levels.

For reference: https://github.com/jglathe/linux_ms_dev_kit/discussions/14 There is also an X13s image that boots reliably from type-c: https://drive.google.com/file/d/1e8LA6o-EKKet3_sCQqXR4lOVE_TgOVNI/view?usp=sharing But, danger. It is deliberately rooted, root pw is FsecuritY! and can be reached via ssh. It is for debug purposes.

[Aaron Honeycutt (aaronhoneycutt)]

For the record 6.8.0-31-generic also does not boot on the hardware.

[Juerg Haefliger (juergh)]

The problem is that on upgrade to Noble the package linux-image-laptop is removed which contains config settings for the machine. I'm in the process of creating a new package (ubuntu-x13s-settings) for Noble that provide these settings. Can you try to install ubuntu-x13s-settings from PPA:ubuntu-concept/x13s (https://launchpad.net/~ubuntu-concept/+archive/ubuntu/x13s) and report back?

One of the issues is that ARM pointer authentication is not working (broken Lenovo firmware) and needs to be disabled for the 6.8 kernel (arm64.nopauth kernel commandline option). The above package should set that (amongst other things).

[Aaron Honeycutt (aaronhoneycutt)]

Once I'm back with the system I'll test and let you know ASAP!

[Aaron Honeycutt (aaronhoneycutt)]

@juergh

I'm now able to boot 6.8.0-31-generic with that package (from that PPA) added to my system! Nicely done!

[Jens Glathe (glathe)]

Hmm an additional data point: I accidentally updated the Windows Dev Kit instead of the X13s to 24.04 (yes, really) and it installed the 6.8.0-31-generic kernel, however no dtb - but that's to be expected when flash-kernel is not installed, and being an unsupported device. After symlinking my dtb, and updating initramfs (as described above #25) this kernel also boots and is quite usable. qseecom / efivar support appears to be missing or not compatible, and there are more error messages regarding pmic glink. qrtr-ns and pd-mapper are up ,though.

`Linux volterra 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 02:32:42 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux`

My additional kernel cmdline parameters are `pd_ignore_unused clk_ignore_unused arm64.nopauth efi=noruntime`, like on the X13s.

So... kernel config appears to be about okay when booting from nvme.

[themroc (rauchweihe)]

I'm now able to boot too 6.8.0-31-generic with ubuntu-x13s-settings from PPA:ubuntu-concept/x13s.
Bluetooth is working again too.
Thank you!

[Jens Glathe (glathe)]

May I ask how you you modify the installer ISO with the missing package? I'm a bit at a loss here.

[Bogdan Melnik (ld86)]

Same questions. Should we expect that daily builds will become bootable or we should manually upgrade from 23.10 and install new package manually?

[Juerg Haefliger (juergh)]

See here https://discourse.ubuntu.com/t/status-of-ubuntu-support-for-lenovo-thinkpad-x13s/44652

Patience, I'm close to having a Noble installer ISO for testing but I keep getting distracted.

[Heinrich Schuchardt (xypron)]

With ubuntu-x13s-settings installed I can boot into kernel 6.8.0-31-generic.