NVME devices are not enumerated on Raspberry PI 5 with Ubuntu 23.10
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| linux-raspi (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
| Mantic |
Fix Released
|
Undecided
|
Unassigned | ||
Bug Description
[Impact]
NVMe devices connected via PCIe are not enumerated on Raspberry Pi 5.
[Fix]
https:/
[Test Case]
Check whether NVMe devices connected via PCIe are detected with lspci/lsblk.
[Where Problems Could Occur]
PCIe devices might misbehave.
[Original Description]
NVME devices are not enumerated on Raspberry PI 5 with Ubuntu 23.10
I've got one of the PCIe NVME extension HATs for rpi 5 [1] and installed a functioning NVME drive Samsung SM961.
Tried to boot it first with an image of RaspberryPI OS from sd-card. Upon boot, the nvme drive was detected and I could mount it and perform some IO:
> uname -a
Linux raspberrypi 6.1.0-rpi8-rpi-2712 #1 SMP PREEMPT Debian 1:6.1.73-1+rpt1 (2024-01-25) aarch64 GNU/Linux
> lspci -nnk
0000:00:00.0 PCI bridge [0604]: Broadcom Inc. and subsidiaries Device [14e4:2712] (rev 21)
Kernel driver in use: pcieport
0000:01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961/SM963 [144d:a804]
Subsystem: Samsung Electronics Co Ltd SM963 2.5" NVMe PCIe SSD [144d:a801]
Kernel driver in use: nvme
0001:00:00.0 PCI bridge [0604]: Broadcom Inc. and subsidiaries Device [14e4:2712] (rev 21)
Kernel driver in use: pcieport
0001:01:00.0 Ethernet controller [0200]: Device [1de4:0001]
Kernel driver in use: rp1
dmesg logs of rpios will be attached, but there is nothing suspicious.
Then I've tried to run Ubuntu 23.10 and reimaged the sd-card with "Ubuntu Server 23.10 (64-bit) (released 2023-10-12).
Upon boot, the drive didn't show up:
> lspci -nnk
00:00.0 PCI bridge [0604]: Broadcom Inc. and subsidiaries Device [14e4:2712] (rev 21)
Kernel driver in use: pcieport
01:00.0 Ethernet controller [0200]: Device [1de4:0001]
Kernel driver in use: rp1
Checked for updates and there was a newer kernel available, thus I've upgraded:
> apt update
> apt list --upgradable
...
linux-image-
After reboot, the nvme still didn't show up. Full dmesg log is attached, but in essence the pcie driver is having troubles to enumerate one for devices on the bus:
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x414fd0b1]
[ 0.000000] Linux version 6.5.0-1010-raspi (buildd@
...
[ 2.612956] brcm-pcie 1000110000.pcie: link down
[ 2.617719] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[ 2.624391] pci 0000:00:00.0: PCI bridge to [bus 01]
...
[ 2.876967] brcm-pcie 1000120000.pcie: link up, 5.0 GT/s PCIe x4 (!SSC)
[ 2.883637] pci 0000:01:00.0: [1de4:0001] type 00 class 0x020000
...
After digging a bit, I found a discussion on the Raspberry's kernel github [2]:
- a bug was introduced sometime in 6.2.x [3]
- a fix landed in 6.6.x [4]
- it boils down to a misused `readw_` instead of `readl_` in the pcie-brcmstb driver
This made me think that given that the upcoming 24.04 is going to be on 6.7 or 6.8 kernel, the fix might be already there.
So I grabbed a build of the "6.7.0-1001.1"[5] kernel from the "Noble Proposed" and installed it on the Mantic on my sd-card.
After reboot, the nvme drive was discovered and proper driver was loaded:
> uname -a
Linux pi 6.7.0-1001-raspi #1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jan 25 12:28:01 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
> lspci -nnk
0000:00:00.0 PCI bridge [0604]: Broadcom Inc. and subsidiaries Device [14e4:2712] (rev 21)
Kernel driver in use: pcieport
0000:01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961/SM963 [144d:a804]
Subsystem: Samsung Electronics Co Ltd SM963 2.5" NVMe PCIe SSD [144d:a801]
Kernel driver in use: nvme
Kernel modules: nvme
0001:00:00.0 PCI bridge [0604]: Broadcom Inc. and subsidiaries Device [14e4:2712] (rev 21)
Kernel driver in use: pcieport
0001:01:00.0 Ethernet controller [0200]: Device [1de4:0001]
Kernel driver in use: rp1
> dmesg
...
[ 2.221266] brcm-pcie 1000110000.pcie: link up, 5.0 GT/s PCIe x1 (!SSC)
[ 2.227947] pci 0000:01:00.0: [144d:a804] type 00 class 0x010802
[ 2.234031] pci 0000:01:00.0: reg 0x10: [mem 0x1b00000000-
[ 2.241396] pci 0000:01:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
...
[ 2.334202] nvme nvme0: pci function 0000:01:00.0
[ 2.338940] nvme 0000:01:00.0: enabling device (0000 -> 0002)
To summarize:
- RaspberryPI OS, kernel 6.1: OK
- Ubuntu 23.10, kernel 6.5.0-1010: FAILS
- Ubuntu 23.10, kernel 6.7.0-1001 from noble: OK
It would be great if the fix[4] for mdio could land 23.10 kernel updates, so NVME drives could be used without waiting for the Noble release later this year.
[1] https:/
[2] https:/
[3] https:/
[4] https:/
[5]
- https:/
- http://
- http://
CVE References
| Dmytro (alkersan) wrote | #1 |
| Dmytro (alkersan) wrote | #2 |
| description: | updated |
| tags: | added: kern-9212 |
| Changed in linux-raspi (Ubuntu Mantic): | |
| status: | New → In Progress |
| description: | updated |
| Changed in linux-raspi (Ubuntu Mantic): | |
| status: | In Progress → Fix Committed |
| Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote | #4 |
| tags: | added: kernel-spammed-mantic-linux-raspi-v2 verification-needed-mantic-linux-raspi |
| Dmytro (alkersan) wrote | #5 |
| tags: |
added: verification-done-mantic-linux-raspi removed: verification-needed-mantic-linux-raspi |
| Launchpad Janitor (janitor) wrote | #6 |
| Changed in linux-raspi (Ubuntu Mantic): | |
| status: | Fix Committed → Fix Released |
| Dave Jones (waveform) wrote | #7 |
| Changed in linux-raspi (Ubuntu): | |
| status: | New → Fix Released |
| tags: | added: kernel-daily-bug |
dmesg of the vanilla raspberry pi os with their 6.1 kernel