Can't enter cryptsetup password on MacBook Pro 2017 (MacBookPro14,1) due to missing kernel modules in initramfs

[Terence Tan]

[ Impact ]

I have a MacBook Pro 2017 (MacBookPro14,1) with an encrypted root partition, which I decrypt via passphrase during boot, at the "Please unlock disk nvme0n1p4_crypt" prompt.

Using 23.04 "Lunar" this worked fine. However after upgrading to 23.10 "Mantic", the built-in keyboard doesn't work for entering the passphrase.

Quickest workaround: plug in a USB keyboard for entering the passphrase.

Workaround: add the following modules to "/etc/initramfs-tools/modules":

spi_pxa2xx_platform
intel_lpss_pci

... then "sudo update-initramfs -u".

I found that both kernel modules were required to get the keyboard recognised in initramfs.

[ Test Plan ]

The -generic kernel and probably all kernel flavors contain the SPI and PCI driver (on amd64). On mantic systems, you can verify the current initrd doesn't contain these modules by decompressing the initramfs and listing its files:
$ lsinitramfs /boot/initrd.img-6.5.0-10-generic | grep 'spi-pxa2xx-platform\|intel-lpss-pci'

After installing the updated initramfs-tools package, your current initramfs should be automatically rebuilt and pick up the kernel modules:
$ lsinitramfs /boot/initrd.img-6.5.0-10-generic | grep 'spi-pxa2xx-platform\|intel-lpss-pci'
usr/lib/modules/6.5.0-10-generic/kernel/drivers/mfd/intel-lpss-pci.ko.zst
usr/lib/modules/6.5.0-10-generic/kernel/drivers/spi/spi-pxa2xx-platform.ko.zst

Test case on MacBook Pro 2017 laptops: Remove all previous workarounds. Install the fixed initramfs-tools version. Then the keyboard should work during boot to enter the passphrase.

[ Where problems could occur ]

This is making the initramfs slightly bigger. In my testing, both added were 23,852 bytes in total (or 0.02 % on a 102 MB initramfs).

[ Remaining original report ]

I did a bit of digging into what changed for 23.10. It could potentially (?) be commit 2df78bbb143884b9601a32608e12e43d40ccb0b0 "Do not install ARM/RISCV specific modules on other architectures".

I had a look into how dracut handles it. Turns out that they specifically include "spi_pxa2xx_platform" after reporting a similar bug report in Fedora: https://bugzilla.redhat.com/show_bug.cgi?id=2166209

I don't know how (if?) they handle the 2nd module, "intel_lpss_pci".

Anyway, I don't need an immediate fix since the workaround works for me. Just posting this for further investigation, and for any other MacBook Pro users in the same situation. Thanks for working on Ubuntu, I think you're all amazing!

[Benjamin Drung (bdrung)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. You did a great job of investigating it.

Commit 2df78bbb143884b9601a32608e12e43d40ccb0b0 and we should do the same as dracut did.

[Benjamin Drung (bdrung)]

This is the relevant dracut commit for spi_pxa2xx_platform: https://github.com/dracutdevs/dracut/commit/df381b7e0cd95f78e40ac70f0f3c96a2fa8dd189

```
$ modprobe --show-depends spi_pxa2xx_platform
insmod /lib/modules/6.5.0-10-generic/kernel/drivers/dma/dw/dw_dmac_core.ko.zst
insmod /lib/modules/6.5.0-10-generic/kernel/drivers/dma/dw/dw_dmac.ko.zst
insmod /lib/modules/6.5.0-10-generic/kernel/drivers/spi/spi-pxa2xx-platform.ko.zst
```

intel_lpss_pci or mfd (from mfd/intel_lpss_pci path) are not directly mentioned in dracut. Neither does `dracut_instmods -o -s "${_blockfuncs}" "=drivers"` install intel_lpss_pci.

dracut does not seem to install intel_lpss_pci:

```
$ schroot-wrapper -p dracut,linux-image-generic,systemd-sysv -c mantic -u root
[...]
(mantic)root@host:~# lsinitrd /boot/initrd.img-6.5.0-10-generic | grep intel_lpss_pci
(mantic)root@host:~#
```

[Benjamin Drung (bdrung)]

Terence, can you verify that intel_lpss_pci is needed and that spi_pxa2xx_platform is not enough?

I found the relevant difference between Fedora and Ubuntu: Fedora builds intel_lpss_pci into the kernel:

```
$ podman run -it fedora:latest /bin/bash
[root@a0c841a1d76f /]# dnf install kernel-core
[root@a0c841a1d76f /]# grep INTEL_LPSS_PCI /lib/modules/6.5.11-300.fc39.x86_64/config
CONFIG_MFD_INTEL_LPSS_PCI=y
```

but Ubuntu builds it as module:

```
$ grep INTEL_LPSS_PCI /boot/config-6.5.0-10-generic
CONFIG_MFD_INTEL_LPSS_PCI=m
```

[Terence Tan (terence-tan)]

Hi Benjamin, I just tried again and can confirm that "intel_lpss_pci" is necessary -- without it, no keyboard.

I removed the intel_lpss_pci entry and booted and collected the attached dmesg.

[Terence Tan (terence-tan)]

With the entry, here's the dmesg:

[Terence Tan (terence-tan)]

And here's the output of "lspci -vv" -- sorry for the multiple messages, couldn't figure out how to attach more than one attachment to a comment!

[Benjamin Drung (bdrung)]

I stripped the timestamp (sed -s 's/\[[0-9. ]*\] //') and sorted the entries of the two dmesg outputs, but I did not find new entries in the diff.

lspci shows that following devices are using the intel_lpss_pci kernel module:

00:15.0 Signal processing controller: Intel Corporation Sunrise Point-LP Serial IO I2C Controller #0 (rev 21)
00:15.2 Signal processing controller: Intel Corporation Sunrise Point-LP Serial IO I2C Controller #2 (rev 21)
00:1e.0 Signal processing controller: Intel Corporation Sunrise Point-LP Serial IO UART Controller #0 (rev 21)
00:1e.3 Signal processing controller: Intel Corporation Sunrise Point-LP Serial IO SPI Controller #1 (rev 21)

and I found this in the dmesg log:

input: Apple SPI Keyboard as /devices/pci0000:00/0000:00:1e.3/pxa2xx-spi.3/spi_master/spi2/spi-APP000D:00/input/input4

which shows that the keyboard is connected via SPI on 00:1e.3.

That explains it. I will upload a fixed initramfs-tools to noble and file a bug against dracut to include intel_lpss_pci. Thanks for your investigation and help. Do you want to help on getting the SRU (see https://wiki.ubuntu.com/StableReleaseUpdates) into mantic?

[Terence Tan (terence-tan)]

Thanks Benjamin for making the fix.

I'm happy if this targets Noble; I'm not overly familiar with the SRU process.

Unsure how much of a real-world impact this bug has; it's users of Ubuntu non-LTS, with a MacBook Pro of a certain year range, using cryptsetup which I don't think is the default (?)

Speaking for myself, I did the manual edits to /etc/initramfs-tools/modules and my system works now, so I'm happy to mark this bug as resolved.

[Benjamin Drung (bdrung)]

Forwarded adding intel_lpss_pci to dracut: https://github.com/dracutdevs/dracut/pull/2556

[Benjamin Drung (bdrung)]

intel_lpss_pci is also needed for the Surface Laptop 4 (see bug #2007050).

[Benjamin Drung (bdrung)]

initramfs-tools 0.142ubuntu15.1 debdiff for mantic attached.

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.142ubuntu16

---------------
initramfs-tools (0.142ubuntu16) noble; urgency=medium

  * Install nvmem, soc, power, and System Power Management Interface (SPMI)
    kernel modules on ARM/RISC-V for MediaTek boards (LP: #2038512)
  * Install intel_lpss_pci and spi_pxa2xx_platform kernel modules for keyboard
    on MacBook Pro 2017 (LP: #2042710)
  * Install surface_aggregator_registry for keyboard on Surface Laptop 4
    (LP: #2007050)
  * framebuffer: Add privacy screen modules to the initrd (LP: #1954320)

 -- Benjamin Drung <email address hidden> Tue, 14 Nov 2023 19:34:42 +0100

[Timo Aaltonen (tjaalton)]

Hello Terence, or anyone else affected,

Accepted initramfs-tools into mantic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.142ubuntu15.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-mantic to verification-done-mantic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-mantic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted initramfs-tools (0.142ubuntu15.1) for mantic have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/0.142ubuntu15.1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/mantic/update_excuses.html#initramfs-tools

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Benjamin Drung (bdrung)]

Verified with linux-image-generic on mantic:

(mantic)bdrung@host:~$ dpkg -s initramfs-tools | grep Version
Version: 0.142ubuntu15.1
(mantic)bdrung@host:~$ lsinitramfs /boot/initrd.img-6.5.0-14-generic | grep 'spi-pxa2xx-platform\|intel-lpss-pci'
usr/lib/modules/6.5.0-14-generic/kernel/drivers/mfd/intel-lpss-pci.ko.zst
usr/lib/modules/6.5.0-14-generic/kernel/drivers/spi/spi-pxa2xx-platform.ko.zst

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for initramfs-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.142ubuntu15.1

---------------
initramfs-tools (0.142ubuntu15.1) mantic; urgency=medium

  * Install intel_lpss_pci and spi_pxa2xx_platform kernel modules for keyboard
    on MacBook Pro 2017 (LP: #2042710)
  * Install surface_aggregator_registry for keyboard on Surface Laptop 4
    (LP: #2007050)

 -- Benjamin Drung <email address hidden> Tue, 14 Nov 2023 20:02:42 +0100