Raspberry Pi 3B+ doesnt boot from USB on 23.10 Mantic

[Aurélien Leblond]

[Impact]

Boot from USB on Pi 3B+ doesn't work. When the onboard-usb-hub module is loaded, all attached devices are reconnected which is fatal if one of them is the boot device.

[Fix]

Revert the commit that added support for the hub on the Pi 3B+. This driver is no really needed for the Pi.

[Test Case]

See below.

[Regression Potential]

Other attached devices that contain the same hub and require the driver will no longer function properly. IMO that's a very unlikely scenario.

[Original Description]

Hello all,

I upgraded my Raspberry Pi 3B+ from 22.04 to 23.10.
The storage device where the OS resides is a USB hdd.
It is connected to the Pi via a ASMedia Technology Inc. X820.

After the upgrade to 23.10, Ubuntu crashes during boot.

To eliminate the X820 as a variable:
- I created a Ubuntu server 23.10 directly on a USB stick -> The pi doesn't boot
- I created a Ubuntu server 23.04 the same way -> the pi boots

After investigation, I found that if I copy the /boot/firmware folder from an old 22.04 image and replace the content of the Fat32 boot partition of the HDD with its content, Ubuntu boots again.

From the error I get during boot time, it looks like after booting, the USB HDD is not found or the root partition is not mounted properly.
Booting with an SD card and mounting the HDD, I see that no logs in /var/log folder are being updated

What should be my next step to get proper error messages and investigate further?

Thanks in advance,
Aurélien

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Libera.chat.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/2039786/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Aurélien Leblond (blablack)]

The issue is with the onboard_usb_hub module.

During boot, when the onboard_usb_hub it causes all USB devices to restart, including the disk Ubuntu is booting from.

rpi kernel: usbcore: registered new device driver onboard-usb-hub
rpi kernel: usb 1-1.1: USB disconnect, device number 3
rpi kernel: lan78xx 1-1.1.1:1.0 (unnamed net_device) (uninitialized): Failed to write register index 0x00000010. ret = -19
rpi kernel: lan78xx 1-1.1.1:1.0 (unnamed net_device) (uninitialized): Registers INIT FAILED....
rpi kernel: lan78xx 1-1.1.1:1.0 (unnamed net_device) (uninitialized): Bind routine FAILED
rpi kernel: usb 1-1.1.1: USB disconnect, device number 5
rpi kernel: usbcore: registered new interface driver lan78xx
rpi kernel: usb 1-1.1.3: USB disconnect, device number 6
rpi kernel: sd 0:0:0:0: [sda] Synchronizing SCSI cache
rpi kernel: sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=DRIVER_OK
rpi kernel: sd 0:0:0:0: [sda] Stopping disk
rpi kernel: sd 0:0:0:0: [sda] Start/Stop Unit failed: Result: hostbyte=0x01 driverbyte=DRIVER_OK

I blacklisted 'onboard_usb_hub' to workaround the issue.
echo "blacklist onboard_usb_hub" > /etc/modprobe.d/blacklist-usb_hub.conf

[Juerg Haefliger (juergh)]

Yes, this is bad
https://<email address hidden>/

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-raspi/6.5.0-1008.11 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-mantic-linux-raspi' to 'verification-done-mantic-linux-raspi'. If the problem still exists, change the tag 'verification-needed-mantic-linux-raspi' to 'verification-failed-mantic-linux-raspi'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Aurélien Leblond (blablack)]

Tested - issue doesn't happen anymore.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-raspi - 6.5.0-1008.11

---------------
linux-raspi (6.5.0-1008.11) mantic; urgency=medium

  * mantic/linux-raspi: 6.5.0-1008.11 -proposed tracker (LP: #2041533)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync update-dkms-versions helper
    - debian/dkms-versions -- update from kernel-versions (main/2023.10.30)

  * drop all references to is_rust_module.sh in kernels >= 6.5 (LP: #2038611)
    - [Packaging] raspi: drop references to is_rust_module.sh

  * Unnecessary armhf DTB (LP: #2039431)
    - [Config] raspi: Set ARCH_BRCMSTB=n for armhf

  * disable shiftfs (LP: #2038522)
    - [Config] raspi: disable shiftfs

  * openvswitch fails on raspberry pi 4 (LP: #2040524)
    - [Packaging] raspi: Include openvswitch in linux-modules

  * Fan speed control not working on Pi 5 under Ubuntu 23.10 (LP: #2041741)
    - driver: thermal: step_wise: Fix uninitialized variable

  * Raspberry Pi 3B+ doesnt boot from USB on 23.10 Mantic (LP: #2039786)
    - SAUCE: Revert "usb: misc: onboard-hub: add support for Microchip USB2514B
      USB 2.0 hub"

  [ Ubuntu: 6.5.0-14.14 ]

  * mantic/linux: 6.5.0-14.14 -proposed tracker (LP: #2042660)
  * Boot log print hang on screen, no login prompt on Aspeed 2600 rev 52 BMC
    (LP: #2042850)
    - drm/ast: Add BMC virtual connector
  * arm64 atomic issues cause disk corruption (LP: #2042573)
    - locking/atomic: scripts: fix fallback ifdeffery
  * Packaging resync (LP: #1786013)
    - [Packaging] update annotations scripts

  [ Ubuntu: 6.5.0-12.12 ]

  * mantic/linux: 6.5.0-12.12 -proposed tracker (LP: #2041536)
  * Packaging resync (LP: #1786013)
    - [Packaging] update annotations scripts
    - [Packaging] update helper scripts
    - debian/dkms-versions -- update from kernel-versions (main/2023.10.30)
  * CVE-2023-5633
    - drm/vmwgfx: Keep a gem reference to user bos in surfaces
  * CVE-2023-5345
    - fs/smb/client: Reset password pointer to NULL
  * CVE-2023-39189
    - netfilter: nfnetlink_osf: avoid OOB read
  * CVE-2023-4244
    - netfilter: nft_set_rbtree: skip sync GC for new elements in this transaction
  * apparmor restricts read access of user namespace mediation sysctls to root
    (LP: #2040194)
    - SAUCE: apparmor: open userns related sysctl so lxc can check if restriction
      are in place
  * AppArmor spams kernel log with assert when auditing (LP: #2040192)
    - SAUCE: apparmor: fix request field from a prompt reply that denies all
      access
  * apparmor notification files verification (LP: #2040250)
    - SAUCE: apparmor: fix notification header size
  * apparmor oops when racing to retrieve a notification (LP: #2040245)
    - SAUCE: apparmor: fix oops when racing to retrieve notification
  * SMC stats: Wrong bucket calculation for payload of exactly 4096 bytes
    (LP: #2039575)
    - net/smc: Fix pos miscalculation in statistics
  * Support mipi camera on Intel Meteor Lake platform (LP: #2031412)
    - SAUCE: iommu: intel-ipu: use IOMMU passthrough mode for Intel IPUs on Meteor
      Lake
    - SAUCE: platform/x86: int3472: Add handshake GPIO function
  * CVE-2023-45898
    - ext4: fix slab-use-after-free in ext4_es_insert_extent(...

[Jef ZIegler (jziegler)]

Was this tested against RPi 5? On RPi 5 going from 6.5.0-1005-raspi to 6.5.0-1008-raspi breaks the USB boot?

https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/2047771

[Aurélien Leblond (blablack)]

It was only tested on 3B+ - unfortunately I don't have a Pi 5 at hand.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-raspi - 6.7.0-1001.1

---------------
linux-raspi (6.7.0-1001.1) noble; urgency=medium

  * noble/linux-raspi: 6.7.0-1001.1 -proposed tracker (LP: #2051136)

  * Packaging resync (LP: #1786013)
    - [Packaging] update Ubuntu.md
    - [Packaging] update update.conf
    - debian/dkms-versions -- update from kernel-versions (main/d2024.01.02)

  * Remove linux-modules-extra (LP: #2048862)
    - [Packaging] raspi: Remove linux-modules-extra package

  * Make dwc2 the default (LP: #2048861)
    - SAUCE: ARM: dts: bcm27xx: Make dwc2 the default

  * Raspberry Pi 3B+ doesnt boot from USB on 23.10 Mantic (LP: #2039786)
    - SAUCE: Revert "usb: misc: onboard-hub: add support for Microchip USB2514B
      USB 2.0 hub"

  * Missing overlays/README (LP: #1954757)
    - SAUCE: (no-up) Install overlays/README

  * [Raspberry Pi/lunar] systemd-oomd fails with
    "ConditionControlGroupController=memory was not met" (LP: #2017209)
    - SAUCE: Revert "cgroup: Disable cgroup "memory" by default"

  * Remove armhf support (LP: #2048864)
    - [Packaging] raspi: Remove armhf packages
    - [Packaging] raspi: Remove armhf ABI files

  * Miscellaneous Ubuntu changes
    - [Packaging] raspi: Initial import of debian.raspi from mantic:linux-raspi
      (6.5.0-1010.13)
    - [Packaging] raspi: Sync packaging files from debian.master
    - [Packaging] raspi: Initial version of linux-raspi for Noble
    - [Config] raspi: updateconfigs after rebase to Ubuntu-6.7.0-2.2
    - [Config] raspi: Include master annotations
    - [Packaging] raspi: Disable all ABI checks
    - SAUCE: Revert "iommu: Retire map/unmap ops"
    - [Packaging] raspi: Import of upstream raspberrypi patchset
    - [Config] raspi: updateconfigs after import of rpi-6.6.y patchset
    - [Config] raspi: Set SWIOTLB_DYNAMIC=n
    - SAUCE: arm64: dts: broadcom: Remove downstream dt overlay support
    - SAUCE: (no-up) ARM: dts: Disable unsupported Raspberry Pi DTBs
    - SAUCE: ARM: dts: Fix broken symlinks
    - SAUCE: ARM: dts: overlays: Fix file permissions
    - [Packaging] raspi: Update reconstruct script

  * Miscellaneous upstream changes
    - raspberrypi-firmware: Update mailbox commands
    - drm/vc4: Add FKMS as an acceptable node for dma ranges.
    - drm/atomic: Don't fixup modes that haven't been reset
    - drm/vc4: Allow setting the TV norm via module parameter
    - drm/vc4: Add firmware-kms mode
    - drm/vc4: Add support for gamma on BCM2711
    - drm/vc4: Add debugfs node that dumps the vc5 gamma PWL entries
    - drm/vc4: hvs: Force modeset on gamma lut change
    - drm/vc4: Relax VEC modeline requirements and add progressive mode support
    - drm/vc4: Make VEC progressive modes readily accessible
    - drm: Check whether the gamma lut has changed before updating
    - drm/vc4: Enable gamma block only when required.
    - drm/vc4: Only add gamma properties once.
    - drm/vc4: Validate the size of the gamma_lut
    - drm/vc4: Disable Gamma control on HVS5 due to issues writing the table
    - drm/dsi: Document the meaning and spec references for MIPI_DSI_MODE_*
    - drm/bridge: tc358762: Ignore EPROBE_DEFER when logging errors
    - vc4/drm: vc...