"lsusb -v" as root freezes the kernel on Acer ES1-111; also freeze on shutdown

[Roland Weber]

reproducible: always

Steps to reproduce:
- open a terminal
- sudo bash
- lsusb -v

In a virtual terminal, the command prints the name of the first device it finds, then the system freezes completely. In a terminal of the desktop, I don't even get the name of the device, the system freezes immediately. Either way, I have to power off the machine. There is no kernel oops. Nothing is written to the log. The magic sysreq key yields no response, although I have verified that it works before triggering the freeze.

I've narrowed down the problem by calling "lsusb -v -s <device>". The system freezes on both devices of bus 3 (003:001 and 003:002). For all other devices, it prints the expected output and continues to work. When calling lsusb as a regular user, I get "Couldn't open device, some information will be missing", then the regular output, and the system keeps running. Therefore I assume that the problem is somewhere in the kernel, when the devices are accessed.

This is a regression. It used to work in 14.10, and fails to work since 15.04.
It also used to work in 14.04.2, and fails in 14.04.3.
It fails with mainline builds 4.2.0-rc6-unstable, 4.2.0-rc2-unstable, 4.1.0-rc2-vivid. (yeah, I've been trying for a while :-)

Incidentally, the shutdown sequence also freezes with the kernels that exhibit the problem; but I cannot be sure that the cause is the same. I decided to first chase the USB freeze, because it was easier to gather information about that.

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: linux-image-3.19.0-25-generic 3.19.0-25.26
ProcVersionSignature: Ubuntu 3.19.0-25.26-generic 3.19.8-ckt2
Uname: Linux 3.19.0-25-generic x86_64
ApportVersion: 2.17.2-0ubuntu1.2
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: rweber 1525 F.... pulseaudio
CurrentDesktop: XFCE
Date: Fri Aug 14 18:38:36 2015
HibernationDevice: RESUME=UUID=8a45266d-0232-428e-bb25-113d23caaf8a
InstallationDate: Installed on 2015-06-28 (46 days ago)
InstallationMedia: Xubuntu 14.10 "Utopic Unicorn" - Release amd64 (20141022.1)
MachineType: Acer Aspire ES1-111M
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.19.0-25-generic root=UUID=b8872e03-7295-4cbd-ac86-06b9be2f112f ro quiet splash crashkernel=384M-:128M vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-3.19.0-25-generic N/A
 linux-backports-modules-3.19.0-25-generic N/A
 linux-firmware 1.143.3
SourcePackage: linux
UpgradeStatus: Upgraded to vivid on 2015-07-05 (39 days ago)
dmi.bios.date: 08/20/2014
dmi.bios.vendor: Insyde Corp.
dmi.bios.version: V1.08
dmi.board.asset.tag: Type2 - Board Asset Tag
dmi.board.name: R2
dmi.board.vendor: Acer
dmi.board.version: Type2 - A01 Board Version
dmi.chassis.type: 10
dmi.chassis.vendor: Chassis Manufacturer
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnInsydeCorp.:bvrV1.08:bd08/20/2014:svnAcer:pnAspireES1-111M:pvrV1.08:rvnAcer:rnR2:rvrType2-A01BoardVersion:cvnChassisManufacturer:ct10:cvrChassisVersion:
dmi.product.name: Aspire ES1-111M
dmi.product.version: V1.08
dmi.sys.vendor: Acer

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

This issue appears to be an upstream bug, since you tested the latest upstream kernel. Would it be possible for you to open an upstream bug report[0]? That will allow the upstream Developers to examine the issue, and may provide a quicker resolution to the bug.

Please follow the instructions on the wiki page[0]. The first step is to email the appropriate mailing list. If no response is received, then a bug may be opened on bugzilla.kernel.org.

Once this bug is reported upstream, please add the tag: 'kernel-bug-reported-upstream'.

[0] https://wiki.ubuntu.com/Bugs/Upstream/kernel

[Roland Weber (r-o-l)]

Thanks for your advice!
I've posted a bug report upstream:
http://permalink.gmane.org/gmane.linux.usb.general/129790

I also checked out some older kernels.
The problem is present since 3.18, while 3.17.x was OK.
Or rather, had other, less dramatic problems on the machine.

[Roland Weber (r-o-l)]

As requested on the linux-usb mailing list, I've bisected the problem in the mainline git repository.

638139eb95d2d241781330a321e88c8dafe46078 is the first bad commit
commit 638139eb95d2d241781330a321e88c8dafe46078
Author: Petr Mladek <email address hidden>
Date: Fri Sep 19 17:32:24 2014 +0200

    usb: hub: allow to process more usb hub events in parallel

    It seems that only choose_devnum() was not ready to process more hub
    events at the same time.

    All should be fine if we take bus->usb_address0_mutex there. It will
    make sure that more devnums will not be chosen for the given bus and
    the related devices at the same time.

    Signed-off-by: Petr Mladek <email address hidden>
    Acked-by: Alan Stern <email address hidden>
    Signed-off-by: Greg Kroah-Hartman <email address hidden>

:040000 040000 6c3b464b8db2bec572cf7904c0aac317b41c1eec da3ee40957d0637f17895ae05aad4a6646377b2a M drivers

[Roland Weber (r-o-l)]

I've verified that the freeze on shutdown is caused by the same commit.

Contrary to my initial report here, the freeze on shutdown happens late in the sequence, not early.
I'm used to much longer shutdown times from my desktop :-)

[Valentine A. (antteor)]

I use Acer Aspire ES1-511. Installed Xubuntu 14.04.01 and he's work perfect.

But...
At the start of a LiveUSB and install version 4.14.02 or older (and 15.04) observed a problem with the power off and reboot the notebook.

After a BIOS update from version 1.03 to 1.09, the problem persists, but behaves differently. The system does not start. Even with LiveUSB. System boot hangs on this - http://cdn.img42.com/307609ac57589768dd3e0f7723b376c4.jpeg

Adding these commands at startup does not help:
              modprobe.blacklist=dw_dmac,i2c_designware_platform,i2c_designware_core

This is the case with any fresh Linux distributions, whether Manjaro or Fedora etc.
Google say these problems often occur on models Acer.

[Roland Weber (r-o-l)]

Valentine, your problem is almost certainly not related to the one I reported here.
You should seek advice in forums or on mailing lists for your computer and/or distribution.

[Roland Weber (r-o-l)]

As suggested on the linux.usb mailing list, I've added some additional debug print statements to the first bad kernel. They are not triggered by "lsusb -v" though. I'm attaching kernel debug output from the boot sequence of two kernels:
- the bad one, with additional print statements
- the last good one, without additional print statements
The difference in the startup behavior is remarkable.

[Roland Weber (r-o-l)]

The previous log for the "bad" kernel was incomplete, due to kernel buffer overflows caused by systemd output. I've switched that off as explained here: https://bugs.freedesktop.org/show_bug.cgi?id=76935#c1

[Roland Weber (r-o-l)]

As suggested by Alan Stone on the linux-usb mailing list, I've verified that the "USB cable is bad" messages are caused by the order in which the USB devices are initialized.

I've also found a new way of triggering the freeze on a bad kernel:
https://bugs.freedesktop.org/show_bug.cgi?id=76935#c1

[Roland Weber (r-o-l)]

Sorry, the link in comment #14 is nonsense, a paste error.
The new way to trigger a freeze is:

echo 0000:00:1d.0 >/sys/bus/pci/drivers/ehci-pci/unbind

[Roland Weber (r-o-l)]

For the records, here's a screenshot (photo) of the system after a freeze. Series of actions to get there:
1. enable kernel debug output
2. detach external mouse, to verify that messages appear
3. hit enter to get a new prompt line
4. trigger the freeze with "lsusb -v"
lsusb prints the name of the device it's going to work with. Then nothing anymore.

[Roland Weber (r-o-l)]

Current state of things: I've tracked down the code line where the freeze happens. It's in function ehci_halt within file ehci-hcd.c. The function is executed on shutdown as well as when the usb devices on the bus are about to be probed again. The latter is triggered by "lsusb -v" as root. Apparently, the hardware misbehaves by not responding to a "readl" call.

As a workaround, it is possible to compile kernel 4.2 with a patch that disables the unused EHCI USB bus. However, this is *not* a generic solution! The same chip may be built into other computers, and the EHCI bus may be used there.

[Roland Weber (r-o-l)]

I removed the bios-outdated tag. The guy who wrote about his BIOS update in comment #9 has a completely different problem and commented off-topic.

[Roland Weber (r-o-l)]

contents of /sys/kernel/debug/usb/ehci/0000:00:1d.0/registers before unbinding the bus which triggers the freeze:
bus pci, device 0000:00:1d.0
EHCI Host Controller
EHCI 1.00, rh state suspended
ownership 01000001 linux
SMI sts/enable 0xe0000000
structural params 0x00200008
capability params 0x00036881
status 1000 Halt
command 0010000 (park)=0 ithresh=1 period=1024 HALT
intrenable 37 IAA FATAL PCD ERR INT
uframe 0bbf
port:1 status 601085 0 ACK POWER sig=se0 SUSPEND PE CONNECT
port:2 status 501000 0 ACK POWER sig=se0
port:3 status 501000 0 ACK POWER sig=se0
port:4 status 501000 0 ACK POWER sig=se0
port:5 status 501000 0 ACK POWER sig=se0
port:6 status 501000 0 ACK POWER sig=se0
port:7 status 501000 0 ACK POWER sig=se0
port:8 status 501000 0 ACK POWER sig=se0
irq normal 18 err 0 iaa 8 (lost 0)
complete 18 unlink 1

/sys/bus/pci/devices/0000:00:1d.0/power/control: on
/sys/bus/pci/devices/0000:00:1d.0/power/runtime_status: active

transcript of on-screen output when unbinding the EHCI USB bus:
ehci-pci 0000:00:1d.0: remove, state 4
ehci-pci 0000:00:1d.0: roothub graceful disconnect
usb usb3: USB disconnect, device number 1
usb 3-1: USB disconnect, device number 2
usb 3-1: ep 81: release intr @ 8+64 (1.0+256) [1/0 us] mask 0001
usb_remove_hcd: calling stop
ehci_silence_controller: entry
ehci_halt: entry
ehci_halt: about to readl prematurely
ehci_halt: premature readl returned 10000
ehci_halt: after first ehci_writel
ehci_halt: before ehci_readl

[Roland Weber (r-o-l)]

contents of /sys/kernel/debug/usb/ehci/0000:00:1d.0/registers before unbinding:
bus pci, device 0000:00:1d.0
EHCI Host Controller
EHCI 1.00, rh state running
ownership 00000001
SMI sts/enable 0xc0080000
structural params 0x00200008
capability params 0x00036881
status 6008 Periodic Recl FLR
command 0010011 (park)=0 ithresh=1 Periodic period=1024 RUN
intrenable 37 IAA FATAL PCD ERR INT
uframe 2cb2
port:1 status 001005 0 ACK POWER sig=se0 PE CONNECT
port:2 status 001000 0 ACK POWER sig=se0
port:3 status 001000 0 ACK POWER sig=se0
port:4 status 001000 0 ACK POWER sig=se0
port:5 status 001000 0 ACK POWER sig=se0
port:6 status 001000 0 ACK POWER sig=se0
port:7 status 001000 0 ACK POWER sig=se0
port:8 status 001000 0 ACK POWER sig=se0
irq normal 17 err 0 iaa 8 (lost 0)
complete 17 unlink 0

There are no /sys/.../power/ files, because CONFIG_PM is disabled.
This kernel does not freeze, so the debug output on unbinding is contained in the attached dmesg log.

[Roland Weber (r-o-l)]

The cause of the freeze seems to be a BIOS problem, as Alan suggested on the linux-usb mailing list. When booting in UEFI mode (from a Xubuntu 15.04 installation medium), three of the devices listed by lspci are missing, compared to legacy BIOS mode. One of those is the device that caused the freezes. Looks like some of the hardware is unused, and Acer just hides it from the OS. But only in UEFI mode.

I'll try a re-installation of Xubuntu in UEFI mode.

[Roland Weber (r-o-l)]

Installing Xubuntu 15.04 to boot in UEFI non-secure mode has some challenges of it's own. But once I got the system working, it doesn't freeze anymore.

If you want to use legacy BIOS mode, you'll have to compile a kernel. However, it is not necessary to apply the patch above. Just disable the USB EHCI driver.