Unable to hotplug ExpressCard on 12.04.3

[kaxing]

cid: 201205-11049

Cold plug works with my APIOTEK usb 3.0 adapter, however it is not hot pluggable.
And when it's cold-plugged, usb hotplug works as usual:

Dec 17 16:02:53 201205-11049 kernel: [ 853.327844] scsi9 : usb-storage 6-2:1.0
Dec 17 16:02:53 201205-11049 mtp-probe: checking bus 6, device 5: "/sys/devices/pci0000:00/0000:00:1c.2/0000:04:00.0/usb6/6-2"
Dec 17 16:02:53 201205-11049 mtp-probe: bus: 6, device: 5 was not an MTP device
Dec 17 16:02:54 201205-11049 kernel: [ 854.327533] scsi 9:0:0:0: Direct-Access ADATA USB Flash Drive 1.00 PQ: 0 ANSI: 6
Dec 17 16:02:54 201205-11049 kernel: [ 854.328560] sd 9:0:0:0: Attached scsi generic sg2 type 0
Dec 17 16:02:54 201205-11049 kernel: [ 854.328800] sd 9:0:0:0: [sdb] 30869504 512-byte logical blocks: (15.8 GB/14.7 GiB)
Dec 17 16:02:54 201205-11049 kernel: [ 854.328995] sd 9:0:0:0: [sdb] Write Protect is off
Dec 17 16:02:54 201205-11049 kernel: [ 854.329003] sd 9:0:0:0: [sdb] Mode Sense: 23 00 00 00
Dec 17 16:02:54 201205-11049 kernel: [ 854.329201] sd 9:0:0:0: [sdb] Write cache: disabled, read cache: disabled, doesn't support DPO or FUA
Dec 17 16:02:54 201205-11049 kernel: [ 854.331605] sdb: sdb1
Dec 17 16:02:54 201205-11049 kernel: [ 854.332523] sd 9:0:0:0: [sdb] Attached SCSI removable disk
Dec 17 16:06:44 201205-11049 wpa_supplicant[942]: WPA: Group rekeying completed with 68:7f:74:2f:f1:0c [GTK=CCMP]
Dec 17 16:12:17 201205-11049 kernel: [ 1417.375484] usb 6-2: USB disconnect, device number 5

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-34-generic 3.2.0-34.53
ProcVersionSignature: Ubuntu 3.2.0-34.53-generic 3.2.33
Uname: Linux 3.2.0-34-generic x86_64
NonfreeKernelModules: nvidia wl
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
ApportVersion: 2.0.1-0ubuntu15
Architecture: amd64
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 0: PCH [HDA Intel PCH], device 0: STAC92xx Analog [STAC92xx Analog]
   Subdevices: 1/1
   Subdevice #0: subdevice #0
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: ubuntu 1557 F.... pulseaudio
 /dev/snd/controlC0: ubuntu 1557 F.... pulseaudio
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Card0.Amixer.info:
 Card hw:0 'PCH'/'HDA Intel PCH at 0xf7730000 irq 54'
   Mixer name : 'IDT 92HD93BXX'
   Components : 'HDA:111d76df,1028053e,00100203'
   Controls : 19
   Simple ctrls : 10
Card1.Amixer.info:
 Card hw:1 'NVidia'/'HDA NVidia at 0xf6080000 irq 17'
   Mixer name : 'Nvidia GPU 42 HDMI/DP'
   Components : 'HDA:10de0042,1028053e,00100100'
   Controls : 12
   Simple ctrls : 2
Date: Mon Dec 17 16:12:49 2012
HibernationDevice: RESUME=UUID=99e407d3-5f30-472b-8141-fe37a95dd820
InstallationMedia: Ubuntu 12.04.1 LTS "Precise Pangolin" - Release amd64 (20120823.1)
MachineType: Dell Inc. Precision M4700
MarkForUpload: True
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-34-generic root=UUID=1daafbbf-ff63-4e89-b9e9-3cf7097ee3af ro quiet splash initcall_debug pciehp_force=1 pciehp_poll_mode=1 vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-34-generic N/A
 linux-backports-modules-3.2.0-34-generic N/A
 linux-firmware 1.79.1
SourcePackage: linux
StagingDrivers: mei
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/21/2012
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A01
dmi.board.name: 0DK7DT
dmi.board.vendor: Dell Inc.
dmi.board.version: X03
dmi.chassis.type: 9
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvrA01:bd06/21/2012:svnDellInc.:pnPrecisionM4700:pvr01:rvnDellInc.:rn0DK7DT:rvrX03:cvnDellInc.:ct9:cvr:
dmi.product.name: Precision M4700
dmi.product.version: 01
dmi.sys.vendor: Dell Inc.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.7 kernel[0] (Not a kernel in the daily directory) and install both the linux-image and linux-image-extra .deb packages.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.7-raring/

[kaxing (kaxing)]

@jsalisbury

I've tested latest 3.7 kernel(030700.201212102335) , and this bug does not fix.
Besides bcmwl is not able to install with mainline kernel, #1023145 , where wireless become not working.

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

The mainline v3.8-rc2 kernel is now available. Can you also test that kernel:

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.8-rc2-raring/

[kaxing (kaxing)]

@Joseph

I've tried kernel 3.8-rc2, and it results same as 3.7 , only cold-plug works, and if you tried to re-plug it's not working even when it is work when it cold-plugged.

[Joseph Salisbury (jsalisbury)]

This issue appears to be an upstream bug, since you tested the latest upstream kernel. Would it be possible for you to open an upstream bug report[0]? That will allow the upstream Developers to examine the issue, and may provide a quicker resolution to the bug.

Please follow the instructions on the wiki page[0]. The first step is to email the appropriate mailing list. If no response is received, then a bug may be opened on bugzilla.kernel.org.

[0] https://wiki.ubuntu.com/Bugs/Upstream/kernel

[kaxing (kaxing)]

I've figured that there is a similar bug already in upstream tracker.

https://bugzilla.kernel.org/show_bug.cgi?id=43192

[Gavin Guo (mimi0213kimo)]

@Yung

Could you please help to print the dmesg when you remove the PCI express card after the successfully cold boot? Thanks

[kaxing (kaxing)]

@Gavin Guo

There is no dmesg output while I removing Express Card.

[Gavin Guo (mimi0213kimo)]

@Yung

As we checked the lsmod output before, acpiphp was not loaded. Maybe we can try to load the acpiphp module to see if it will work.

[Gavin Guo (mimi0213kimo)]

status:
Asking the upstream <email address hidden> about how to add the configuration file to the module-init-tools to get approved.

[kaxing (kaxing)]

@Gavin

manually loading acpiphp works, I've also tried to add acpiphp into /etc/modules ,
and please check attachment for acpiphp logs.

[Gavin Guo (mimi0213kimo)]

@Yung

I think adding to the configuration file is ok, now waiting for the upstream's reply to see how to get approved.

[Gavin Guo (mimi0213kimo)]

I created a bug and sent a patch to the debian bug checking system to fix the bug.
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=700993

[James M. Leddy (jm-leddy)]

This bug was not present on enabled systems in Oneiric. If we can test Oneiric on these systems, it would be a good data point to have, to see if it was something that is a regression or a workaround on the iso that fixed it.

[Daniel Manrique (roadmr)]

Yung, could you please test 11.10 to see if this bug is present there?

Thanks!

[kaxing (kaxing)]

Hey Daniel,

Tried on 11.10, It's the same as precise, User still have to manually 'modprobe acpiphp' to make hot-plug work.

[Gavin Guo (mimi0213kimo)]

Upstream maintainer rejected to accept the solution, adding acpiphp in the /etc/modules. He suggested that the fix should be in the kernel side. I'll try to find out the point in the kernel.

[Gavin Guo (mimi0213kimo)]

The bug is possible to solve by building in the acpiphp modules, it needs to discuss with senior Kernel team members.

[Gavin Guo (mimi0213kimo)]

After discussing with Ike, we came up with 2 methods, either to write a driver, which like acpiphp_ibm.c, to depend on acpiphp.ko or to simulate the acpiphp then to write a driver under drivers/platform/x86/. Now, I'm consulting the upstream for the suggestion.

[Gavin Guo (mimi0213kimo)]

I found recent patches from linux-pci maintainer had made the acpiphp only statically build-in to avoid sub-driver dependency issue, and will test the driver in the latest kernel. If passing the test, the patches will be backported from linux-next tree of linux-pci maintainer.

[James M. Leddy (jm-leddy)]

Thanks for the status update Gavin, please keep us abreast of your progress.

[Gavin Guo (mimi0213kimo)]

The acpiphp can't work after testing the pci maintainer's, Bjorn Helgaas, next tree. Now, I'll submitting the dmesg and some error description, before testing the Linux-next again.

[Gavin Guo (mimi0213kimo)]

upstream maintainers aggressively help to solve the bug. I'm helping to test the patches updated by them.

[Gavin Guo (mimi0213kimo)]

Upstream solved the bug and I plan to submit a patch to enable the configuration as to build in the acpiphp module in the Quantal and Raring for the reason that the modules will only can be built-in in the Linux 3.10 merge window. However, backporting from helgaas/next will cover a lot of commits thus it might not be a good solution.

[Gavin Guo (mimi0213kimo)]

To save debugging log for future reference. The commit id at the current helgaas/next is d4f09c5d7fbabd1389a5f03f5c9329d790f544e3. Yinghai Lu provided 2 patches and the v1 fail but the v2 works perfectly. The patch is attached.

[Gavin Guo (mimi0213kimo)]

Submitting a patch using the built-in modules but rejected by kernel team because of building in the acpiphp modules before v3.10-rc1, which includes the Jiang's patch, will cause the ordering issue, the issue that other sub-driver related to acpiphp should link before the acpiphp.

There are many lines in drivers/pci/hotplug/Makefile has useful information:

# native drivers should be linked before acpiphp in order to allow the
# native driver to attempt to bind first. We can then fall back to
# generic support.

[James M. Leddy (jm-leddy)]

Hi Gavin,

What's the next step here? Can we break out the built-in drivers for the next release so that we no longer have this problem?

[Gavin Guo (mimi0213kimo)]

@James,

In 3.10, we just need to build in the acpiphp driver. There are many newly added commits related to the acpiphp modules from raring kerne to the v3.10-rc2 as following, and I'm wondering if kernel team would receive those many commits if backporting those from mainline kernel:

gavin@gavin-virtual-machine:~/os/ubuntu-quantal$ git log Ubuntu-3.8.0-19.29..v3.10-rc2 --pretty=oneline drivers/pci/hotplug/acpiphp_[cg]*
3d54a3160fb6ba877324ffffa5d301dec8038fd9 PCI: acpiphp: Protect acpiphp data structures from concurrent updates
ad41dd9dd0c8ca1876f30b62c5c79625ffe83174 PCI: acpiphp: Use normal list to simplify implementation
3b63aaa70e1ccc4b66d60acc78da09700706a703 PCI: acpiphp: Do not use ACPI PCI subdriver mechanism
6037a803b05eef9943fb64982e19964007fb7478 PCI: acpiphp: Convert acpiphp to be builtin only, not modular
ce15d873d05ebf3bf38579c4e0252140e28f1781 PCI: acpiphp: Replace local macros with standard ACPI macros
3a0e40beefc20852191ed65d53e1b82d95ac11b8 PCI: acpiphp: Remove all functions even if function 0 doesn't exist
d65eba6a7a769bb939303969267cbf1c916358f5 PCI: acpiphp: Use list_for_each_entry_safe() in acpiphp_sanitize_bus()
556f12f602ac0a18a82ca83e9f8e8547688fc633 Merge tag 'pci-v3.9-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci
3757b94802fb65d8f696597a74053cf21738da0b ACPI / hotplug: Fix concurrency issues and memory leaks
be6d2867b4f68a575c78fa368abd3ad49980c514 PCI: acpiphp: Remove dead code for PCI host bridge hotplug
2ca344e8c798127ae01e656838b9b06ed41d1461 PCI: acpiphp: Create companion ACPI devices before creating PCI devices
bfee26dba0f373ebe4e6f0b293d078b02f9f7f69 ACPI / scan: Make it clear that acpi_bus_trim() cannot fail
d59f53bc9bd80ee62072dea590fc623c67cb84a8 PCI: acpiphp: Keep driver loaded even if no slots found
668192b678201d2fff27c6cc76bb003c1ec4a52a PCI: acpiphp: Move host bridge hotplug to pci_root.c
92d8aff3a317fcd6f78ed9ac13dbbaeae8cb11ed PCI/ACPI: acpiphp: Rename alloc_acpiphp_hp_work() to alloc_acpi_hp_work()
1f96a965e30d097a25818196e33d2dce923973a7 PCI: acpiphp: Add is_hotplug_bridge detection
b8bd759acd05281abf88cddef30c57313c109697 ACPI / scan: Drop acpi_bus_add() and use acpi_bus_scan() instead
ae281795ec92d35dd1631401829124acab965b1f ACPI / scan: Drop the second argument of acpi_bus_trim()
0cd6ac52b333f66ee64e50ed216ec99231092dcd ACPI: Make acpi_bus_scan() and acpi_bus_add() take only one argument
636458de36f1fb4cdd318387d2f45604e451b17a ACPI: Remove the arguments of acpi_bus_add() that are not used
02f57c67a8677ae55dcdd256a2a7abaf41e4cc1f ACPI: Remove acpi_start_single_object() and acpi_bus_start()

[James M. Leddy (jm-leddy)]

Gavin,

We may want to do that. Another option (if it is easier) is to hold off until 12.04.3, and backport to that kernel. I'll discuss with the certification team.

[James M. Leddy (jm-leddy)]

We've decided to wait on 12.04.3 for this bug.

[Gavin Guo (mimi0213kimo)]

After discussing with Jmleddy, the bug will be waiting until 3.10.

[Gavin Guo (mimi0213kimo)]

Yung found that some platforms use the pciehp driver instead of acpiphp one. And I tried to find what's the difference. The answer is the platform using pciehp uses irq 41, the PME pciehp, and the one using acpiphp uses irq 9, the acpi default irq. So, what reason make the difference? The platform uses acpi irq #9 as shared irq because of irq's scarcity.

[Po-Hsu Lin (cypressyew)]

Also affects 201202-10559 Dell Latitude E6330

[Ara Pulido (ara)]

Yung, Po-Hsu: can you check if this is actually fixed with the 3.10 kernel?

[kaxing (kaxing)]

@Ara

we've tried kernel 3.10rc7[0], hot plug still not working, but received following message,

plug:
<7>[ 448.136826] _handle_hotplug_event_root: Bus check notify on \_SB_.PCI0
unplug:
<7>[ 449.837717] _handle_hotplug_event_root: Bus check notify on \_SB_.PCI0

and talked with gavin, he will help us on checking if the patch had been successfully build into the kernel.

[0]http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.10-rc7-saucy/

[Gavin Guo (mimi0213kimo)]

I've tested with latest v3.10 released today. It works, the difference is that it needs to enable the CONFIG_HOTPLUG_PCI_ACPI=y.

[Anthony Wong (anthonywong)]

Hi Gavin, what's our next step here to have it fixed on Saucy?

[Gavin Guo (mimi0213kimo)]

@Anthony, the next step will send a patch, which enable the CONFIG_HOTPLUG_PCI_ACPI=y, to the Ubuntu mainline when I have bandwidth.

[Gavin Guo (mimi0213kimo)]

the configuration was already enabled in the

commit f3ce39309d9416473ba79035267a42d4dac0a11d
Author: Tim Gardner <email address hidden>
Date: Tue May 14 13:45:10 2013 -0600

    UBUNTU: rebase to v3.10-rc1

    Signed-off-by: Tim Gardner <email address hidden>

-CONFIG_HOTPLUG_PCI_ACPI=m
+CONFIG_HOTPLUG_PCI_ACPI=y

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "acpiphp_check_host_bridge_v2.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Anthony Wong (anthonywong)]

@kaxing
According to comment #42 this bug should already been fixed in Saucy, could you grab a copy of saucy ISO from http://cdimage.ubuntu.com/daily-live/current/ and give it a test?

[kaxing (kaxing)]

@Anthony
Verified with kernel 3.10.0-6 #17, hotplug now work as expected.

[Gavin Guo (mimi0213kimo)]

This bug can't be fixed in Raring. As the feature is also available in the 12.04.4, users can upgrade their kernel to use the feature.

[Po-Hsu Lin (cypressyew)]

Verified on 201307-13942 Dell Latitude E5540 with 14.04 (3.13.0-32)
And it does not work.

this message will appear in dmesg when you try to hotplug the expresscard
ACPI: Bus check notify on \_SB_.PCI0

But it works with upstream utopic kernel 3.17.0

[Po-Hsu Lin (cypressyew)]

Verified with 201404-14986 with 14.04.3 (3.19 kernel), expresscard hotplug works! Thanks!