High CPU usage of CFS

[Dave Johansen]

It appears that there is an issue with the Completely Fair Scheduler (CFS) not intereacting well with sleep commands. I have written up a test program to help demonstrate the issue and it can be found at:
www.github.com/daveisfera/test_sleep/
The test outputs wall time (CPU clock), user time, and system time, but usually only the wall time and user time outputs are meaningful because the system time is so small and usually within the standard deviation for all tests.

I have also attached the results from running this test on an HP Pavilion dm4 and I believe that the gap at the low number of threads is caused by either the processor going into a power saving state because of the sleeping and/or a poor interaction with HyperThreading (scheduling the 2 threads on logical cores rather than separate physical cores).

Because of this, I then ran 4 instances of:
schedtool -a <CPU#> -e nice -19 yes > /dev/null
so that all of the cores would be kept busy during the sleeping and this had an unexpected effect of causing the wall time to increase significantly. I am not sure what the cause of the wall time increase is, but it does help the user time plot display the gap between the sleep methods and the none sleep methods that I believe is a symptom/display of the unexpectedly high CPU usage when the process should be sleeping.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-31-generic 3.2.0-31.50
ProcVersionSignature: Ubuntu 3.2.0-31.50-generic 3.2.28
Uname: Linux 3.2.0-31-generic x86_64
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
ApportVersion: 2.0.1-0ubuntu13
Architecture: amd64
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 0: PCH [HDA Intel PCH], device 0: STAC92xx Analog [STAC92xx Analog]
   Subdevices: 1/1
   Subdevice #0: subdevice #0
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: dlj 1778 F.... pulseaudio
Card0.Amixer.info:
 Card hw:0 'PCH'/'HDA Intel PCH at 0xc0800000 irq 50'
   Mixer name : 'Intel CougarPoint HDMI'
   Components : 'HDA:111d76e0,103c1793,00100102 HDA:80862805,103c1793,00100000'
   Controls : 22
   Simple ctrls : 10
Date: Fri Sep 28 20:57:35 2012
InstallationMedia: Ubuntu 12.04.1 LTS "Precise Pangolin" - Release amd64 (20120823.1)
MachineType: Hewlett-Packard HP Pavilion dm4 Notebook PC
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-31-generic root=UUID=162256F02256D479 loop=/hostname/disks/root.disk ro quiet splash vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-31-generic N/A
 linux-backports-modules-3.2.0-31-generic N/A
 linux-firmware 1.79.1
SourcePackage: linux
StagingDrivers: rts_pstor mei
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 01/17/2012
dmi.bios.vendor: Hewlett-Packard
dmi.bios.version: F.08
dmi.board.asset.tag: Base Board Asset Tag
dmi.board.name: 1793
dmi.board.vendor: Hewlett-Packard
dmi.board.version: 41.1C
dmi.chassis.asset.tag: Chassis Asset Tag
dmi.chassis.type: 10
dmi.chassis.vendor: Hewlett-Packard
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnHewlett-Packard:bvrF.08:bd01/17/2012:svnHewlett-Packard:pnHPPaviliondm4NotebookPC:pvr068E100002204710000622100:rvnHewlett-Packard:rn1793:rvr41.1C:cvnHewlett-Packard:ct10:cvrChassisVersion:
dmi.product.name: HP Pavilion dm4 Notebook PC
dmi.product.version: 068E100002204710000622100
dmi.sys.vendor: Hewlett-Packard

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.6 kernel[0] (Not a kernel in the daily directory) and install both the linux-image and linux-image-extra .deb packages.

Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. Please only remove that one tag and leave the other tags. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.6-quantal/

[Dave Johansen (davejohansen)]

I ran it with the 3.6 kernel and it gave basically identical results. I didn't attach the plots because they didn't show anything meaningful or new, but I can if you would like.

[Dave Johansen (davejohansen)]

I installed the kernel with the BFS scheduler and other CK modifications from https://launchpad.net/~chogydan/+archive/ppa and ran the test.

The performance at the low number of threads shows the same gap, but once the thread count hits the number of cores, the gap basically closes completely, so it appears that this kernel does not have the same issue with the sleep methods not behaving properly.

Also, the issue when running schedtool doesn't show the same degradation in performance but the sleep methods do take a hit at the higher thread counts which is probably to be expected and is still relatively minor, especially when compared to the performance of the standard kernel.

My general assessment would be that the kernel with the BFS scheduler and other CK modifications either reduces or eliminates the issue that is causing the unexpectedly high CPU usage of the sleep methods.

[penalvch (penalvch)]

Dave Johansen, could you please provide the full computer model as noted on the sticker (ex. HP Pavilion dm4-2015dx Entertainment Notebook PC)?

[Dave Johansen (davejohansen)]

I no longer have access to this hardware to get the full computer model, but the referenced test programs should be able to reproduce the issue on most, if not all machines (it's worked on everyone I've tried it on to date).