Skylake processor never reaches low power states on X1 Carbon gen 4 with NVMe drive

[Dave Chiluk]

Reproducer
1. Find a skylake machine. - in my case Lenovo x1 carbon gen 4. In my case with an NVME drive.
2. run powertop, and switch to idle stats tab
3. do nothing. *(as in let the machine idle till the cores regularly reach low power states.
4. Watch the watts fly. On my machine the package never gets into any power state other than PC2.

Expected results: PC3-PC10 power states should all show some usage.

I also tried the mainline build of 4.6-rc7 with no benefits.

Public conversations about the issue
https://mjg59.dreamwidth.org/42156.html
https://mjg59.dreamwidth.org/41713.html

As far as I can tell having the nvme drive is part of the overall problem here. Please keep this case to skylake + nvme. Skylake + sata appears to have a number of fixes in the 4.6 and newer kernels that enable some of the higher sleep states, but I have not tested with a sata drive. Those fixes as of testing 4.6-rc7 do not resolve this issue with users who have nvme drives.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-22-generic 4.4.0-22.39
ProcVersionSignature: Ubuntu 4.4.0-22.39-generic 4.4.8
Uname: Linux 4.4.0-22-generic x86_64
ApportVersion: 2.20.1-0ubuntu2
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: chiluk 3139 F.... pulseaudio
CurrentDesktop: Unity
Date: Mon May 9 15:55:03 2016
HibernationDevice: RESUME=UUID=4f4ea88e-642e-4be5-bdf0-bbc7f47f5628
InstallationDate: Installed on 2016-04-25 (14 days ago)
InstallationMedia: Ubuntu 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.1)
Lsusb:
 Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
 Bus 001 Device 003: ID 04f2:b531 Chicony Electronics Co., Ltd
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: LENOVO 20FBCTO1WW
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-22-generic.efi.signed root=UUID=249f91bb-7789-41cc-9cc0-8ba25d7f55bb ro quiet splash pcie_aspm=force vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-22-generic N/A
 linux-backports-modules-4.4.0-22-generic N/A
 linux-firmware 1.157
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 03/15/2016
dmi.bios.vendor: LENOVO
dmi.bios.version: N1FET37W (1.11 )
dmi.board.asset.tag: Not Available
dmi.board.name: 20FBCTO1WW
dmi.board.vendor: LENOVO
dmi.board.version: SDK0J40709 WIN
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: None
dmi.modalias: dmi:bvnLENOVO:bvrN1FET37W(1.11):bd03/15/2016:svnLENOVO:pn20FBCTO1WW:pvrThinkPadX1Carbon4th:rvnLENOVO:rn20FBCTO1WW:rvrSDK0J40709WIN:cvnLENOVO:ct10:cvrNone:
dmi.product.name: 20FBCTO1WW
dmi.product.version: ThinkPad X1 Carbon 4th
dmi.sys.vendor: LENOVO

[Brad Figg (brad-figg)]

This change was made by a bot.

[Tim Gardner (timg-tpi)]

Dave - I wonder if https://bugs.launchpad.net/bugs/1579278 has any bearing on your problem ?

[Buzzing Robot (buzzingrobot)]

Seeing much the same with 16.04 on an Intel NUC with a Skylake i5 6260U, and a Samsung 950 NVME, with the default kernel or 4.6-rc7. (I've been tracking the 4.6 RC's and this behavior has been apparent in each.)

Running powertop with no tuning option shows all usage confined to C2 (pc2).

Running powertop with the autotune option ("powertop --auto-tune") results in usage spread out among pc2, pc3, and, occasionally, pc7. PC8 and lower are never reached.

[Dave Chiluk (chiluk)]

I think it's unrelated, but in the off chance this is firmware related, I have also tried the 1.14 version of Lenovo's bios with no improvement

[Dave Chiluk (chiluk)]

Per request. fwts cstates

[Colin Ian King (colin-king)]

Can you run:

powerstat -za 1 480

..and paste the output into the bug report as an attachment.

[Colin Ian King (colin-king)]

And also:

sudo powerstat -Rag 1 480

..and again paste the output into the bug report as an attachement.

[Dave Chiluk (chiluk)]

powerstat -za 1 480

When the machine from 7 -> 4 watts usage is roughly when the display went to sleep.

[Dave Chiluk (chiluk)]

sudo powerstat -Rag 1 480.

[Dave Chiluk (chiluk)]

Here's a screen cap of powertop. Notice that the package never reaches any powerstate above pc2.

[Colin Ian King (colin-king)]

Thanks Dave, that data clearly shows that the CPUs are getting into the deeper C states yet the processor package is indeed showing that it is not going into a deep power state.

[Colin Ian King (colin-king)]

Dave, this is interesting to read:

http://www.mouser.com/pdfdocs/4thgencorefamilydesktopvol1datasheet.pdf

Section 4.2.5, pages 54 through to 58. See the C2 package state, it maybe that the package is not able to transition to C3 because of those constraints.

[Dave Chiluk (chiluk)]

So I recently had to run a certain Microsoft OS while debugging another issue. I ran a number updates during that session. The most important of which was an IMEI firmware update. I can't prove for certain that the IMEI firmware update resolved this issue, but it is the mostly likely considering every other update was unrelated to firmware bios or power.

[Dave Chiluk (chiluk)]

Sorry hit post too quickly.

Anyhow, so it looks like upgrading the IMEI firmware now allows my amt/vpro capable laptop to reach low-power power pc-states.

[Dave Chiluk (chiluk)]

IMEI firmware update definitely appears to be the fix as it is now verified independently.
https://bugzilla.kernel.org/show_bug.cgi?id=116591

[Dave Chiluk (chiluk)]

Closing as invalid, as the root cause was clearly the IMEI firmware in my case, and has since been verified.

[David Jao (djao)]

I applied the Intel IMEI update and now I can reach pc6 and pc7 pretty consistently but never pc8, pc9, or pc10. Does this agree with your observations? Using a T460s with NVMe and kernel 4.4.0-36-generic #55 on 16.04, no powertop tuning. All combinations of AC/battery and display on/off were tested.

[Francois Thirioux (fthx)]

I hit this exact bug running Zesty, kernel 4.9 or 4.10 (or 4.8...).

[Dave Chiluk (chiluk)]

@djao, sorry for the slow response. I hit pc8 briefly, but that's with most everything tuned, and running nothing. Lots of applications seem to be pretty power hungry for me. That's going to have to get tackled by each upstream project before we integrate solutions into the distro.

@fthx. Welcome to the club. Did you try updating the IMEI firmware?

[Francois Thirioux (fthx)]

My Dell BIOS is up-to-date. The IMEI firmware is included in Dell's BIOS updates, there is no separate package.
I'll try to check IMEI version anyway.

PS : I do use NVMe drive, before that my Skylake Xeon 1535M was eating very few watts (~4.5W), now it's ~8.2W ; I don't know what to think about that, NVMe missing APST for now or IMEI.

[Francois Thirioux (fthx)]

FYI i'm back to 4.23 W (that's damn good, TLP still used) with Ubuntu 4.11 RC1 kernel (including APST commit by default).

APST is the key for my problem, since I reach PC8 (powertop) at the same time.
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1664602