ubuntu server 20.04 LTS for ARM64 on Raspberry Pi4 could not handle more than 32 USB3 devices

[Ye He]

When I connect a 16 port USB3 hub to one of the USB3 ports on the Pi4 (4GB or 8GB ram model), and then plug in external powered hard drives into the hub, when I reach the sixth or seventh or maximum 8th drive, OS complains xHCI max limit of 32 has been reached. I have a keyboard, no mouse, I'm not sure why it's already 32 USB devices.

I was hoping I could connect 2 such 16port usb hubs to the 2 usb3 ports on the Pi4, and connect 32 externally powered hard drives to the Pi4.

[Ye He (yhebm)]

There is some comments online about having the system switch from xHCI to eHCI but none of the suggested methods worked.

[Juerg Haefliger (juergh)]

What kernel version? What does the output of 'lsusb' show when you get the message?

[Juerg Haefliger (juergh)]

You won't be able to connect 32 devices since there are are already internal devices and hubs.

[Po-Hsu Lin (cypressyew)]

Marked as incomplete for Juerg's information request.

[Ye He (yhebm)]

~$ uname -a
Linux Ubuntu1 5.4.0-1034-raspi #37-Ubuntu SMP PREEMPT Mon Apr 12 23:14:49 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

Imaged this through the Raspian Imager after selecting the 20.4.02. Server LTS for ARM64

~$ lsusb -t
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
    |__ Port 1: Dev 5, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 1: Dev 6, If 0, Class=Mass Storage, Driver=uas, 5000M
        |__ Port 2: Dev 9, If 0, Class=Mass Storage, Driver=uas, 5000M
        |__ Port 3: Dev 10, If 0, Class=Mass Storage, Driver=uas, 5000M
        |__ Port 4: Dev 7, If 0, Class=Hub, Driver=hub/4p, 5000M
            |__ Port 3: Dev 15, If 0, Class=Hub, Driver=hub/3p, 5000M
                |__ Port 1: Dev 16, If 0, Class=Mass Storage, Driver=uas, 5000M
            |__ Port 1: Dev 11, If 0, Class=Hub, Driver=hub/3p, 5000M
                |__ Port 1: Dev 12, If 0, Class=Mass Storage, Driver=uas, 5000M
            |__ Port 4: Dev 8, If 0, Class=Hub, Driver=hub/4p, 5000M
                |__ Port 2: Dev 19, If 0, Class=Hub, Driver=hub/3p, 5000M
                    |__ Port 1: Dev 20, If 0, Class=Mass Storage, Driver=uas, 5000M
                |__ Port 1: Dev 17, If 0, Class=Hub, Driver=hub/3p, 5000M
                    |__ Port 1: Dev 18, If 0, Class=Mass Storage, Driver=uas, 5000M
            |__ Port 2: Dev 13, If 0, Class=Hub, Driver=hub/3p, 5000M
                |__ Port 1: Dev 14, If 0, Class=Mass Storage, Driver=uas, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 30, If 0, Class=Hub, Driver=hub/4p, 480M
            |__ Port 4: Dev 31, If 0, Class=Hub, Driver=hub/4p, 480M
                |__ Port 1: Dev 34, If 0, Class=Hub, Driver=hub/3p, 480M
                |__ Port 2: Dev 38, If 0, Class=Hub, Driver=hub/3p, 480M
                |__ Port 3: Dev 42, If 0, Class=Hub, Driver=hub/3p, 480M
                |__ Port 4: Dev 32, If 0, Class=Hub, Driver=hub/4p, 480M
                    |__ Port 2: Dev 50, If 0, Class=Hub, Driver=hub/3p, 480M
                    |__ Port 1: Dev 46, If 0, Class=Hub, Driver=hub/3p, 480M

[Ye He (yhebm)]

So it looks like other than the internal devices that count towards this xHCI max limit of 32 (what are they? Audio? Bluetooth? Can they be disabled to free up space), the 16 port USB3 hub itself is probably daisy chained internally and takes up spots as well. I've read that by switching to the eHCI driver I would not be limited to this 32 but I would downgrade to USB2 speed instead, which I could live with. So if this limit of 32 on xHCI just cannot be solved with the 1 USB controller onboard and the xHCI driver, is there a workaround to load the eHCI driver instead?

[Juerg Haefliger (juergh)]

The 32 devices is an xhci limitation. No way around it (/me thinks). xhci only supports up to 96 endpoints and each USB device consumes at least 3 endpoints...

And then you have a tiered structure with hubs so the number of attachable 'real' devices is reduced even further.

Seriously? You want to run multiple HDDs over a single USB2 connection? And no, xhci is built-in and will always claim the USB host controller so there is no way to downgrade to USB2.

[Ye He (yhebm)]

As a minimal I'd like to utilize all the 16 ports on this usb hub. usb3 would be preferred but usb2 is acceptable. so this limit is not on the OS and on Windows I'd have the same issue with 1 single USB controller?

[Juerg Haefliger (juergh)]

Ok so the xhci spec allows for more endpoints but Intel's implementation seems to be limited to 96 endpoints. But the datasheet of the VL805 specifies 20h (32) for MAXSLOTS so 32 devices seems to be a HW limitation on the Pi's USB controller as well. There's no way around it.

[Ye He (yhebm)]

I understand there is no way around the 32 device limit on the VL805 using USB3. But I'm nowhere near 32 devices, yet. On the Pi4 I don't even have a keyboard plugged in. It has one 16port USB3 hub, and I'm only able to plug in 6 hard drives on it when the hub is on a USB3 port, or 12 hard drives on it when the hub is on a USB2 port. But dmesg shows that even on a USB2 port, xHCI is used, not eHCI.

Also, I was able to disable xHCI driver during loading by blacklisting it in cmdlines.txt, but then eHCI is not loaded in its place, so in that case I don't have usb capabilities at all. lsusb returns nothing for example. I'm trying to figure out how to load that manually first.

[Juerg Haefliger (juergh)]

The ehci driver does not support this controller. It's an XHCI controller so only the xhci driver works. You would have to modify the ehci driver to recognize it and you probably also need to disable the xhci driver completely but I'm not sure if this will work.

[Ye He (yhebm)]

I got all 16 USB ports on the hub recognized by plugging it into the usbC port. A few things needed to happen in advance for that to work. 1. Supply power through PoE (or GPIO on the Pi4). 2. Switch the USB C port to host mode, and using the dwc2 driver. 3. Reboot Pi4. It's down to USB2.0 speed though but it doesn't have the same limit as xHCI. This plus the USB2 port on the Pi4 which could support up to 13 devices on the same USB hub, I could connect a max of 29 devices. The USB2 port still uses xHCI and thus the 13 device limit.

[Launchpad Janitor (janitor)]

[Expired for linux-raspi (Ubuntu) because there has been no activity for 60 days.]