PCIe ASPM disabled even when enabled in FADT and pcie_aspm=force / setpci used

[Chris Cheney]

This is somewhat similar in issue to lp#1377596.

I have a "Lenovo ThinkCentre m900 Tiny" with an "Intel 600p" NVMe drive in it.

As far as I can tell from dissembling the FADT PCIe ASPM is not set to disabled but the ports are not enabled in UEFI during boot.

I also notice that the PCIe ASPM debug files are not in sysfs like they normally are ie: link_state

I tried running the following manually but I think it must not be enough as the NVMe drive does not run any cooler as it does in a system with known working ASPM.

setpci -s 00:1b.0 0x50.B=0x3
setpci -s 00:1c.0 0x50.B=0x3
setpci -s 01:00.0 0x80.B=0x3
setpci -s 02:00.0 0x50.B=0x3

I noticed via lspci that the root ports say ASPM not supported even though the FADT doesn't have them disabled. Is that the reason its still not working and how do I work around that issue?

Also is there something specific I can tell Lenovo to do to fix this issue on their end, if it can't be worked around in Linux? I have an open case with them via the UEFI engineering team but so far its been slow going since I don't have anything concrete to tell them to fix.

===

00:1b.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Root Port #17 (rev f1) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 122
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 0000f000-00000fff
        Memory behind bridge: df100000-df1fffff
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #17, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s <1us, L1 <16us
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
                        Slot #0, PowerLimit 25.000W; Interlock- NoCompl+
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
                        Changed: MRL- PresDet- LinkState+
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABC, TimeoutDis+, LTR+, OBFF Not Supported ARIFwd+
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
                         EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
                Address: fee00218 Data: 0000
        Capabilities: [90] Subsystem: Lenovo Sunrise Point-H PCI Root Port
        Capabilities: [a0] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
        Capabilities: [140 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Capabilities: [220 v1] #19
        Kernel driver in use: pcieport
        Kernel modules: shpchp

ProblemType: Bug
DistroRelease: Ubuntu 17.04
Package: linux-image-4.10.0-19-generic 4.10.0-19.21
ProcVersionSignature: Ubuntu 4.10.0-19.21-generic 4.10.8
Uname: Linux 4.10.0-19-generic x86_64
ApportVersion: 2.20.4-0ubuntu4
Architecture: amd64
AudioDevicesInUse:
 Cannot stat file /proc/2050/fd/24: Stale file handle
 Cannot stat file /proc/2050/fd/25: Stale file handle
                      USER PID ACCESS COMMAND
 /dev/snd/controlC0: ubuntu-mate 2046 F.... pulseaudio
CasperVersion: 1.380
Date: Fri Apr 14 04:10:41 2017
LiveMediaBuild: Ubuntu-MATE 17.04 "Zesty Zapus" - Release amd64 (20170412)
MachineType: LENOVO 10FMCTO
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/casper/vmlinuz.efi debug ignore_loglevel pcie_aspm.policy=powersave pcie_aspm=force file=/cdrom/preseed/hostname.seed boot=casper quiet splash ---
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-4.10.0-19-generic N/A
 linux-backports-modules-4.10.0-19-generic N/A
 linux-firmware 1.164
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 03/07/2017
dmi.bios.vendor: LENOVO
dmi.bios.version: FWKT6CA
dmi.board.name: 30D0
dmi.board.vendor: LENOVO
dmi.board.version: SDK0K17763 WIN 1801925111491
dmi.chassis.type: 3
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnLENOVO:bvrFWKT6CA:bd03/07/2017:svnLENOVO:pn10FMCTO:pvrLenovoProduct:rvnLENOVO:rn30D0:rvrSDK0K17763WIN1801925111491:cvnLENOVO:ct3:cvrDefaultstring:
dmi.product.name: 10FMCTO
dmi.product.version: Lenovo Product
dmi.sys.vendor: LENOVO

[Chris Cheney (ccheney)]

The setpci commands I used modified the LnkCtl option but not the LnkCap values. I'm not sure but suspect that the LnkCap values are what are causing the problem.

--

LnkCap: Port #17, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s <1us, L1 <16us
        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkCap: Port #7, Speed 8GT/s, Width x1, ASPM not supported, Exit Latency L0s <1us, L1 <16us
        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

vs

LnkCap: Port #17, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s unlimited, L1 <16us
        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk-
        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

LnkCap: Port #7, Speed 8GT/s, Width x1, ASPM not supported, Exit Latency L0s unlimited, L1 <16us
        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk-
        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-

[Brad Figg (brad-figg)]

This change was made by a bot.

[Chris Cheney (ccheney)]

Talking to Matthew Garrett on irc he said that he thinks this may only be fixable via an UEFI change.

04:33 < ccheney> anyone know if it is possible to change a setting in LnkCap on a pci device? My pcie ports apparently have 'ASPM not supported' set and I need to turn it on somehow
04:33 < ccheney> its on a skylake system
04:35 < mjg59> ccheney: It's set by either the firmware or the chipset itself
04:35 < ccheney> the FADT doesn't have aspm disabled but its not enabled on the ports or the devices, setpci seems to be able to turn it on for LnkCtl but LnkCap seems to be overriding it?
04:35 < mjg59> lnkcap represents what the link is capable of
04:35 < ccheney> mjg59: is there anyway to force enable it afterwards via initrd (similar to acpi replacement tables) or something like that?
04:35 < mjg59> ccheney: Nope
04:35 < ccheney> ugh :(
04:35 < mjg59> Not that I know of

[Chris Cheney (ccheney)]

I sent a follow up email to Lenovo but don't expect to hear back from them until sometime after next Monday.

[Kai-Heng Feng (kaihengfeng)]

Is cooling the NVMe is your goal? 4.10.0-19 already has NVMe APST support. No sure if PCIe ASPM matters though.

[Chris Cheney (ccheney)]

Kai-Heng,

Yes, but with ASPM disabled unfortunately even ASPT doesn't help enough. :-(