Thunderbolt devices don't work if connected during boot

[André Düwel]

I am using a Dell XP 15 9550, if my Dockingstation (Dell TB16) is connected during boot time, all the USB devices (like Keyboard, Ethernet, ...) are only working while in grub boot menu. After booting they stop working.

The two 4k screens which are connected to the dock, are working properly in boot menu and after booting. If booting into Windows everything is working, only linux is affected.

Disconnecting and reconnecting the dock solves this problem.

I have created the bug-report directly after boot, where the USB devices are not working and attached "lspci-vnvn_after_reconnect.log" after reconnecting the dock.

ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: linux-image-4.10.0-22-generic 4.10.0-22.24
ProcVersionSignature: Ubuntu 4.10.0-22.24-generic 4.10.15
Uname: Linux 4.10.0-22-generic x86_64
NonfreeKernelModules: nvidia_uvm nvidia_drm nvidia_modeset nvidia
ApportVersion: 2.20.4-0ubuntu4.1
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: andre_duewel 3163 F.... pulseaudio
CurrentDesktop: X-Cinnamon
Date: Mon Jun 12 11:17:26 2017
InstallationDate: Installed on 2017-04-12 (60 days ago)
InstallationMedia: Ubuntu 17.04 "Zesty Zapus" - Release amd64 (20170412)
MachineType: Dell Inc. XPS 15 9550
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.10.0-22-generic.efi.signed root=UUID=0ba04b8b-151f-4522-81a8-b6b01352ea35 ro quiet splash nvme_core.default_ps_max_latency_us=6000 vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-4.10.0-22-generic N/A
 linux-backports-modules-4.10.0-22-generic N/A
 linux-firmware 1.164.1
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/22/2016
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 1.2.19
dmi.board.name: 0N7TVV
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 9
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr1.2.19:bd12/22/2016:svnDellInc.:pnXPS159550:pvr:rvnDellInc.:rn0N7TVV:rvrA00:cvnDellInc.:ct9:cvr:
dmi.product.name: XPS 15 9550
dmi.sys.vendor: Dell Inc.

[Joseph Salisbury (jsalisbury)]

This change was made by a bot.

[Kai-Heng Feng (kaihengfeng)]

When the issue happen, does "sudo lspci -vv" or "sudo lsusb -v" workaround the issue?

[André Düwel (aduewel)]

saw, that I created the bug report with my normal user account :/ ... here is the lspci directly after reboot with usb devices not working

[André Düwel (aduewel)]

No it won't help.

[André Düwel (aduewel)]

I see errors during the sudo lsusb -v:

Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Couldn't open device, some information will be missing
...
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Couldn't open device, some information will be missing

these do not occur after reconnecting the dock.

[Mario Limonciello (superm1)]

This was raised recently and the root cause was recently discussed during the submission of the thunderbolt patch series by Mika Westerberg:
https://lkml.org/lkml/2017/5/25/288

It will need to be worked out with the PCIe subsystem maintainers yet.

[Mario Limonciello (superm1)]

Mika reproduced this on a a different TBT device too.
(so to be clear; this is not TB16 specific. This is a problem with Thunderbolt devices + Linux currently)

[André Düwel (aduewel)]

@Mario, thanks for this additional information. Could you image a way to influence this order, to let the pci_scan happen before ACPI is initialized?

[Mario Limonciello (superm1)]

Mika has already consulted with the ACPI guys and has an idea, but it's going to require changes outside of Linux first in ACPICA. So it shouldn't be a very quick fix.

>There is also another PCI/ACPI related issue that Mario reported where
>we execute _INI() methods before the initial PCI scan on boot when
>Thunderbolt device is connected causing Linux to accidentally find the
>upstream port of the Thunderbolt host controller before it is configured
>properly by the BIOS ACPI hotplug handler. I've discussed this with
>Rafael and he has an idea how we could fix it but it probably requires
>some changes to ACPICA first. Also on my todo list :)

https://lkml.org/lkml/2017/6/5/881

[Kai-Heng Feng (kaihengfeng)]

Thanks for the info, Mario.
Is it machine/chipset specific? I didn't see this issue on Latitude E7370 though.

[André Düwel (aduewel)]

I could reproduce this issue on my old TB15 dock with my XP15 9550, too.

@Kai-Heng Feng: Dell is replacing the TB15 with the TB16 for free, because of heat problems and other issues especially if you use two 4k monitors. But you need to call the service center and they do not want the old TB15 back. ;)

[Mario Limonciello (superm1)]

@Kai Heng:

It should not be machine/chipset specific, but may have some BIOS component to this due to BIOS sends the hotplug event. If the hotplug event comes too soon (before enumeration is done) you should see this behavior.

It's specific to PCIe tunnels of TBT devices. You won't for example see it if you connect a non-TBT type-C device to the the type-C port of the TB16 or a display to one of the display ports.

The USB hub and USB ports are however behind the PCIe tunnels though and should hit this issue.

[Kai-Heng Feng (kaihengfeng)]

@Mario

Thanks for the info.

@Andre

Does doing a PCI rescan work for you?

$ sudo -s
# echo 1 > /sys/bus/pci/rescan

[André Düwel (aduewel)]

nope, does not help. Only reconnecting helps :-/

[Kai-Heng Feng (kaihengfeng)]

Hmm, maybe try go to /sys/bus/pci/drivers/pcieport, then unbind & bind all Alpine Ridge devices?

[Zhenfang Wei (kopkop)]

Hi, i am using Dell XPS 9360 Developer Edition (with ubuntu 16.04 (kernel 4.4.0)) together with TB16 docking station. I have the same issue, if TB16 is plugged before booting, the peripherals such as the USB port, ethernet port, USB-C port, etc. works fine. But they are not detected once the TB16 is unplugged after booting and re-plugged again.

[André Düwel (aduewel)]

@Kai-Heng: How do I unbind/bind all devices? Never did this before :/

echo "0000:00:1d.6" > /sys/bus/pci/drivers/pcieport/unbind

Seems to unbind all usb devices, at least they stop working :D ... but

echo "0000:00:1d.6" > /sys/bus/pci/drivers/pcieport/bind

has no effect.

lspci | grep Alpine
06:00.0 PCI bridge: Intel Corporation DSL6340 Thunderbolt 3 Bridge [Alpine Ridge 2C 2015]
07:00.0 PCI bridge: Intel Corporation DSL6340 Thunderbolt 3 Bridge [Alpine Ridge 2C 2015]
07:01.0 PCI bridge: Intel Corporation DSL6340 Thunderbolt 3 Bridge [Alpine Ridge 2C 2015]
07:02.0 PCI bridge: Intel Corporation DSL6340 Thunderbolt 3 Bridge [Alpine Ridge 2C 2015]
08:00.0 System peripheral: Intel Corporation DSL6340 Thunderbolt 3 NHI [Alpine Ridge 2C 2015]
09:00.0 PCI bridge: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015]
0a:01.0 PCI bridge: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015]
0a:04.0 PCI bridge: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015]
0c:00.0 PCI bridge: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015]
0d:01.0 PCI bridge: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015]
0d:04.0 PCI bridge: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015]

─root@atlantis /sys/bus/pci/drivers/pcieport
╰─# tree
.
├── 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
├── 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
├── 0000:00:1c.1 -> ../../../../devices/pci0000:00/0000:00:1c.1
├── 0000:00:1d.0 -> ../../../../devices/pci0000:00/0000:00:1d.0
├── 0000:00:1d.4 -> ../../../../devices/pci0000:00/0000:00:1d.4
├── 0000:00:1d.6 -> ../../../../devices/pci0000:00/0000:00:1d.6
├── 0000:06:00.0 -> ../../../../devices/pci0000:00/0000:00:1d.6/0000:06:00.0
├── 0000:07:00.0 -> ../../../../devices/pci0000:00/0000:00:1d.6/0000:06:00.0/0000:07:00.0
├── 0000:07:01.0 -> ../../../../devices/pci0000:00/0000:00:1d.6/0000:06:00.0/0000:07:01.0
├── 0000:07:02.0 -> ../../../../devices/pci0000:00/0000:00:1d.6/0000:06:00.0/0000:07:02.0
├── 0000:09:00.0 -> ../../../../devices/pci0000:00/0000:00:1d.6/0000:06:00.0/0000:07:01.0/0000:09:00.0
├── 0000:0a:01.0 -> ../../../../devices/pci0000:00/0000:00:1d.6/0000:06:00.0/0000:07:01.0/0000:09:00.0/0000:0a:01.0
├── 0000:0a:04.0 -> ../../../../devices/pci0000:00/0000:00:1d.6/0000:06:00.0/0000:07:01.0/0000:09:00.0/0000:0a:04.0
├── 0000:0c:00.0 -> ../../../../devices/pci0000:00/0000:00:1d.6/0000:06:00.0/0000:07:01.0/0000:09:00.0/0000:0a:04.0/0000:0c:00.0
├── 0000:0d:01.0 -> ../../../../devices/pci0000:00/0000:00:1d.6/0000:06:00.0/0000:07:01.0/0000:09:00.0/0000:0a:04.0/0000:0c:00.0/0000:0d:01.0
├── 0000:0d:04.0 -> ../../../../devices/pci0000:00/0000:00:1d.6/0000:06:00.0/0000:07:01.0/0000:09:00.0/0000:0a:04.0/0000:0c:00.0/0000:0d:04.0
├── bind
├── new_id
├── remove_id
├── uevent
└── unbind

[Kai-Heng Feng (kaihengfeng)]

Something like this:

for i in $(lspci -nn | grep Alpine | awk '{print $1}'); do
    i="0000:$i"
    echo "$i" > /sys/bus/pci/drivers/pcieport/unbind
    echo "$i" > /sys/bus/pci/drivers/pcieport/bind
done

[André Düwel (aduewel)]

# for i in $(lspci -nn | grep Alpine | awk '{print $1}'); do
    i="0000:$i"; echo $i
    echo "$i" > /sys/bus/pci/drivers/pcieport/unbind
    echo "$i" > /sys/bus/pci/drivers/pcieport/bind
done
0000:06:00.0
0000:07:00.0
0000:07:01.0
0000:07:02.0
0000:08:00.0
echo: write error: no such device
echo: write error: no such device
0000:09:00.0
0000:0a:01.0
0000:0a:04.0
0000:0c:00.0
0000:0d:01.0
0000:0d:04.0

without success :/

[Kai-Heng Feng (kaihengfeng)]

Probably physical intervention is the only viable workaround now...

[dragon788 (dragon788)]

Kai-Heng, did you run the command as root? Otherwise you may want to `sudo su` and run the loop or "prime" your sudo by running `sudo echo` (to get prompted for you password once) and then use the same loop but instead of `> /path/to/file` use `| tee /path/to/file` and see if you get different results.

I will bookmark this to test after my next restart, as I'm seeing the same issue on boot, and I don't know that changing the system hardware test from Automatic to Thorough or Quick in the BIOS will make a difference.

[Kai-Heng Feng (kaihengfeng)]

Yea, I ran it as root.

[dragon788 (dragon788)]

I wonder if we need to sleep a second between the unbind and the rebind? I still haven't restarted but I will test that while monitoring with `journalctl -xf` in another terminal.

[Yehezkel Bernat (yehezkelshb)]

FYI, the patches that Mario mentioned above landed Linus' tree for v4.14-rc1 and hopefully will be backported to 4.13.
See details here:
https://github.com/01org/thunderbolt-software-user-space/issues/24#issuecomment-328011428

[Mario Limonciello (superm1)]

Those 3 patches were backported to 4.13.13.

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-4.13.y&id=dcd2b010f17d64f19ac6a1f7cc77d85faf29196a
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-4.13.y&id=d055d4161c9354cd53f33c38bc9699ceeaadf1b5
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-4.13.y&id=61380235fe7c76ee306565c0f4dd294256e25079

[Mario Limonciello (superm1)]

Looks like those made it to artful too:
https://bugs.launchpad.net/intel/+bug/1717431