Unable to hotplug ExpressCard

[Po-Hsu Lin]

There is a similar bug 1091125, but this one is slightly different

Steps:
1. Install 12.04.4 + update (3.11.0-18)
2. Hotplug the APIOTEK usb 3.0 adapter ExpressCard
2.1 Hotplug a USB stick on the ExpressCard
3. Hotplug the APIOTEK usb 3.0 adapter ExpressCard with a USB on it

Expected Result:
* The ExpressCard should be hot-pluggable

Actual Result:
The following message will shows up in dmesg:
* plug the ExpressCard
[ 748.270195] pciehp 0000:00:15.3:pcie04: Card present on Slot(0-1)
* Connect a USB stick to the ExpressCard
(Nothing happened)
* unplug the ExpressCard
[ 756.122707] pciehp 0000:00:15.3:pcie04: Card not present on Slot(0-1)
* plug the ExpressCard with USB on it
[ 1199.593797] pciehp 0000:00:15.3:pcie04: Card present on Slot(0-1)

Cold plug works fine, and when it's cold-plugged, usb hotplug works as usual.

Debugging information:
acpiphp module cannot be found in lsmod
I can't load that module manually with this error message:
FATAL: Module acpiphp not found.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.11.0-18-generic 3.11.0-18.32~precise1
ProcVersionSignature: Ubuntu 3.11.0-18.32~precise1-generic 3.11.10.4
Uname: Linux 3.11.0-18-generic x86_64
NonfreeKernelModules: fglrx
AlsaVersion: Advanced Linux Sound Architecture Driver Version k3.11.0-18-generic.
ApportVersion: 2.0.1-0ubuntu17.6
Architecture: amd64
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 1: Generic_1 [HD-Audio Generic], device 0: CX20590 Analog [CX20590 Analog]
   Subdevices: 1/1
   Subdevice #0: subdevice #0
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: ubuntu 1622 F.... pulseaudio
 /dev/snd/controlC0: ubuntu 1622 F.... pulseaudio
CRDA:
 country TW:
  (2402 - 2472 @ 40), (3, 27)
  (5270 - 5330 @ 40), (3, 17), DFS
  (5735 - 5815 @ 40), (3, 30)
Card0.Amixer.info:
 Card hw:0 'Generic'/'HD-Audio Generic at 0xf0b44000 irq 47'
   Mixer name : 'ATI R6xx HDMI'
   Components : 'HDA:1002aa01,00aa0100,00100200'
   Controls : 7
   Simple ctrls : 1
Card0.Amixer.values:
 Simple mixer control 'IEC958',0
   Capabilities: pswitch pswitch-joined penum
   Playback channels: Mono
   Mono: Playback [on]
Card1.Amixer.info:
 Card hw:1 'Generic_1'/'HD-Audio Generic at 0xf0b40000 irq 16'
   Mixer name : 'Conexant CX20590'
   Components : 'HDA:14f1506e,17aa21ea,00100000'
   Controls : 20
   Simple ctrls : 9
Card29.Amixer.info:
 Card hw:29 'ThinkPadEC'/'ThinkPad Console Audio Control at EC reg 0x30, fw unknown'
   Mixer name : 'ThinkPad EC (unknown)'
   Components : ''
   Controls : 1
   Simple ctrls : 1
Card29.Amixer.values:
 Simple mixer control 'Console',0
   Capabilities: pswitch pswitch-joined penum
   Playback channels: Mono
   Mono: Playback [on]
Date: Sun Mar 9 22:08:19 2014
HibernationDevice: RESUME=UUID=f746ff69-0a94-4753-8729-e754d0d71a4c
InstallationMedia: Ubuntu 12.04.4 LTS "Precise Pangolin" - Release amd64 (20140204)
MachineType: LENOVO 120024U
MarkForUpload: True
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.11.0-18-generic root=UUID=adfbaf9f-a293-4508-a962-3c1d4590c5cf ro quiet splash initcall_debug vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-3.11.0-18-generic N/A
 linux-backports-modules-3.11.0-18-generic N/A
 linux-firmware 1.79.10
RfKill:
 0: phy0: Wireless LAN
  Soft blocked: no
  Hard blocked: no
SourcePackage: linux-lts-saucy
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 07/14/2011
dmi.bios.vendor: LENOVO
dmi.bios.version: 8IET33WW(1.06)
dmi.board.asset.tag: Not Available
dmi.board.name: 120024U
dmi.board.vendor: LENOVO
dmi.board.version: Not Available
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvr8IET33WW(1.06):bd07/14/2011:svnLENOVO:pn120024U:pvrThinkPadE525:rvnLENOVO:rn120024U:rvrNotAvailable:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 120024U
dmi.product.version: ThinkPad E525
dmi.sys.vendor: LENOVO

[Po-Hsu Lin (cypressyew)]

lspci information with the ExpressCard cold plugged:

06:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller [1033:0194] (rev 03) (prog-if 30 [XHCI])
 Physical Slot: 0-1
 Flags: bus master, fast devsel, latency 0, IRQ 19
 Memory at f0500000 (64-bit, non-prefetchable) [size=8K]
 Capabilities: [50] Power Management version 3
 Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+
 Capabilities: [90] MSI-X: Enable+ Count=8 Masked-
 Capabilities: [a0] Express Endpoint, MSI 00
 Capabilities: [100] Advanced Error Reporting
 Capabilities: [140] Device Serial Number ff-ff-ff-ff-ff-ff-ff-ff
 Capabilities: [150] Latency Tolerance Reporting
 Kernel driver in use: xhci_hcd

[Po-Hsu Lin (cypressyew)]

dmesg with the ExpressCard cold plugged:
ubuntu@201107-8258:~$ dmesg | grep pciehp
[ 0.940613] pciehp 0000:00:15.2:pcie04: HPC vendor_id 1022 device_id 43a2 ss_vid 1022 ss_did 0
[ 0.940645] pciehp 0000:00:15.2:pcie04: service driver pciehp loaded
[ 0.940657] pciehp 0000:00:15.3:pcie04: HPC vendor_id 1022 device_id 43a3 ss_vid 1022 ss_did 0
[ 0.940685] pciehp 0000:00:15.3:pcie04: service driver pciehp loaded
[ 0.940690] pciehp: PCI Express Hot Plug Controller Driver version: 0.4

Verified with older kernels:
3.2.0-60, 3.5.0-47, 3.8.0-37, none of them could work with ExpressCard hotplug.

Also verified with the latest mainline kernel, 3.14-rc5 trusty
Still not working.

[Gavin Guo (mimi0213kimo)]

I spent some times testing and found that the commits causing the regression is between v3.11 and v3.12-rc1. It still needs more times to do the bisect.

Mon May 26 19:02:09 gavin@gavin-w530:~/os/linux$ git log --pretty=oneline v3.11..v3.12-rc1 | wc -l
10153

Since there're still 10153 commits left to be identified. I'll try to test the related commits first which show up in the following:

Mon May 26 18:51:24 gavin@gavin-w530:~/os/linux$ git log --pretty=oneline v3.11..v3.12-rc1 drivers/acpi/pci*
a9238741987386bb549d61572973c7e62b2a4145 Merge tag 'pci-v3.12-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci
e89c33168aad32436da842ddda307dcc31c0c4e2 Merge branch 'pci/misc' into next
3dc48af310709b85d07c8b0d3aa8f1ead02829d3 PCI/ACPI: Fix _OSC ordering to allow PCIe hotplug use when available
39772038ea93e85ea4f1307ec9c1f48a063d89a0 PCI: Assign resources for hot-added host bridge more aggressively
928bea964827d7824b548c1f8e06eccbbc4d0d7d PCI: Delay enabling bridges until they're needed
be1c9de98d8904c75a5ab8b2a0d97bea0f7c07cc ACPI / PCI: Make bus registration and unregistration symmetric

[Gavin Guo (mimi0213kimo)]

I've built the following commits and it can work so have no clue to do the bisect within this commits.

Mon May 26 18:51:24 gavin@gavin-w530:~/os/linux$ git log --pretty=oneline v3.11..v3.12-rc1 drivers/acpi/pci*
a9238741987386bb549d61572973c7e62b2a4145 Merge tag 'pci-v3.12-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci
e89c33168aad32436da842ddda307dcc31c0c4e2 Merge branch 'pci/misc' into next
3dc48af310709b85d07c8b0d3aa8f1ead02829d3 PCI/ACPI: Fix _OSC ordering to allow PCIe hotplug use when available
39772038ea93e85ea4f1307ec9c1f48a063d89a0 PCI: Assign resources for hot-added host bridge more aggressively
928bea964827d7824b548c1f8e06eccbbc4d0d7d PCI: Delay enabling bridges until they're needed
be1c9de98d8904c75a5ab8b2a0d97bea0f7c07cc ACPI / PCI: Make bus registration and unregistration symmetric

Now, I'm on the way of bisecting from v3.11 to v3.12-rc1.
gavinguo@tangerine:~/os/linux$ git bisect log
# bad: [272b98c6455f00884f0350f775c5342358ebb73f] Linux 3.12-rc1
# good: [6e4664525b1db28f8c4e1130957f70a94c19213e] Linux 3.11
# good: [39772038ea93e85ea4f1307ec9c1f48a063d89a0] PCI: Assign resources for hot-added host bridge more aggressively
# good: [3dc48af310709b85d07c8b0d3aa8f1ead02829d3] PCI/ACPI: Fix _OSC ordering to allow PCIe hotplug use when available
git bisect start 'v3.12-rc1' 'v3.11' '39772038ea93e85ea4f1307ec9c1f48a063d89a0' '3dc48af310709b85d07c8b0d3aa8f1ead02829d3' '--'
# bad: [57d730924d5cc2c3e280af16a9306587c3a511db] Merge branch 'timers-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad 57d730924d5cc2c3e280af16a9306587c3a511db
# bad: [357397a14117f0c2eeafcac06a1f8412a02aa6af] Merge branch 'for-3.12' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/libata
git bisect bad 357397a14117f0c2eeafcac06a1f8412a02aa6af
# good: [91121c103ae93ef117e58712786864270d7f488e] staging: dwc2: make dwc2_core_params documentation more complete
git bisect good 91121c103ae93ef117e58712786864270d7f488e

[Gavin Guo (mimi0213kimo)]

I've completed the bisect and found 2d8b1d566a5f4874f4d92361f5cdbb50baa396f8 is the first bad commit. The "git bisect log" is also attached.

commit 2d8b1d566a5f4874f4d92361f5cdbb50baa396f8
Author: Mika Westerberg <email address hidden>
Date: Sat Jul 13 20:09:59 2013 +0300

    ACPI / hotplug / PCI: Get rid of check_sub_bridges()

    Now that acpiphp_check_bridge() always enumerates devices behind the
    bridge, there is no need to do that for each sub-bridge anymore like
    it is done in the current ACPI-based PCI hotplug (ACPIPHP) code.

    Given this we don't need check_sub_bridges() anymore, so drop that
    function completely.

    This also simplifies the ACPIPHP code a bit.

    Signed-off-by: Mika Westerberg <email address hidden>
    Signed-off-by: Rafael J. Wysocki <email address hidden>

:040000 040000 1f3f81404e06c92c01d6848ed7a6c5bffdb5f878 4e7ca3e4af42738f90555640cafbb6e6522ceedf M drivers

[Gavin Guo (mimi0213kimo)]

The bug was reported to https://bugzilla.kernel.org/show_bug.cgi?id=76901. Currently, Mika asked for the support to apply his patch, which adds more debug messages to the suspected function, and get the dmesg log. I've submitted the dmesg for his diagnosis.

[Gavin Guo (mimi0213kimo)]

Rafael submitted a patch to the https://bugzilla.kernel.org/show_bug.cgi?id=76901. I've tested and it works fine. Now, I've provided the dmesg for him to check and am waiting for Rafael to give a official commit.

[Gavin Guo (mimi0213kimo)]

I've tested with Rafael's cleaner patch on the Lenovo CID 201107-8258. It works fine. As it didn't work on the v3.13 generic.

[Gavin Guo (mimi0213kimo)]

The patch has been merged in the v3.16-rc1 as following:

commit 882d18a702c66404fcb62b84748f719f9b47441c
Author: Rafael J. Wysocki <email address hidden>
Date: Tue Jun 10 22:46:35 2014 +0200

    ACPI / hotplug / PCI: Add hotplug contexts to PCI host bridges

    After relatively recent changes in the ACPI-based PCI hotplug
    (ACPIPHP) code, the acpiphp_check_host_bridge() executed for PCI
    host bridges via acpi_pci_root_scan_dependent() doesn't do anything
    useful, because those bridges do not have hotplug contexts. That
    happens by mistake, so fix it by making acpiphp_enumerate_slots()
    add hotplug contexts to PCI host bridges too and modify
    acpiphp_remove_slots() to drop those contexts for host bridges
    as appropriate.

    Link: https://bugzilla.kernel.org/show_bug.cgi?id=76901
    Fixes: 2d8b1d566a5f (ACPI / hotplug / PCI: Get rid of check_sub_bridges())
    Reported-and-tested-by: Gavin Guo <email address hidden>
    Acked-by: Bjorn Helgaas <email address hidden>
    Reviewed-by: Mika Westerberg <email address hidden>
    Cc: 3.15+ <email address hidden> # 3.15+
    Signed-off-by: Rafael J. Wysocki <email address hidden>

I've tried to cherry-picked the commit. But it caused a lot of conflicts. And I tried to show the difference between v3.13 to v3.16-rc1:
Wed Jun 18 08:37:57 gavin@gavin-w530:~/os/ubuntu-trusty$ git log v3.13..v3.16-rc1 --pretty=oneline drivers/pci/hotplug/acpiphp_glue.c
7550cfab3d4053b54f16e2fe337affde71d1eb51 Merge tag 'pci-v3.16-changes-2' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci
882d18a702c66404fcb62b84748f719f9b47441c ACPI / hotplug / PCI: Add hotplug contexts to PCI host bridges
227f06470502c4fea3d93df1f12a77e3e37f6263 PCI: Merge multi-line quoted strings
d1a2523d2adc0b6910dbc2a9aed44c4217134db1 Merge branches 'pci/hotplug', 'pci/pci_is_bridge' and 'pci/virtualization' into next
c7a071f6a9f02c9fd2d0aa81d000c814a7adb914 PCI: acpiphp: Use pci_is_bridge() to simplify code
10874f5a00266343a06e95da680e8a5a383d9a80 PCI: Remove unnecessary __ref annotations
56a3d18279f00c7ccbcdc193ceaf1a2f88c51457 PCI: Remove unnecessary includes of <linux/init.h>
4b1779c2cf030c68aefe939d946475e4136c1895 Merge tag 'pci-v3.15-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci
d901188f002104905a7845298e1a7dc94189007a Merge branch 'acpi-pci-hotplug' into acpi-hotplug
b8a62d540240387c7e97a69127cab3ab9cadc70a ACPI / hotplug / PCI: Use pci_device_is_present()
be27b3dcb02335ec093b81053fc8c84b32d3106e ACPI / dock: Add .uevent() callback to struct acpi_hotplug_context
edf5bf34d40804fbef32f240a79b74ffc69a658b ACPI / dock: Use callback pointers from devices' ACPI hotplug contexts
3b52b21fa1f44c8956e21dfba645eda959111b5e ACPI / dock: Use ACPI device object pointers instead of ACPI handles
96075315c5e7077fc5a3ac54c9b9e97e376e66ed Merge branch 'acpi-pci-hotplug' into acpi-dock
59b42fa01fe2d84f3c3f28ee6f25510820ace35b ACPI / hotplug: Add .fixup() callback to struct acpi_hotplug_context
d7c7c0256b936901eb7a70fbc6d674a6bdfd463e ACPI / hotplug / PCI: Do not clear event callback pointer for docks
cc6254e00eb676dda6501655f8185aef7b761b4f ACPI / hotplug / PCI...

[Adam Lee (adam8157)]

linux-lts-saucy is EOL, moving to Utopic

[Adam Lee (adam8157)]

Utopic is actually fixed, since 882d18a702c66404fcb62b84748f719f9b47441c was merged already

[Adam Lee (adam8157)]

@Po-Hsu Lin,

since this is for N+1 certification, Utopic is also acceptable, please take a test an close this bug, thanks.

[Adam Lee (adam8157)]

Close this since Utopic has already fixed it.