CFS in 2.6.24 kernel needs cpu_share adjustment for "niced" processes

[Daniel Hahler]

I have BOINC running in the background, which uses the remaining (idle) CPU cycles.

With 2.6.24-2, other processes now get max. 50% of the available CPU cycles! ~50% of the cycles still go to boinc, although it's running at niceness 19.

Additionally, non-nice load does not change the cpu frequency (from 1.0 to 1.8ghz in my case), when the dynamic cpufreq governor is being used.

I've tested normal cpu load with "cat /dev/urandom > /dev/null", which is expected to max out, but does not so with 2.6.24-2.

This has been confirmed by somebody else with the same CPU (Athlon64 3000+), using the amd64 kernel.
I'm using the x86-generic kernel myself.

[Daniel Hahler (blueyed)]

Confirmed because of duplicate.

[Dana Goyette (danagoyette)]

[Comments copied from other bug report:]

I usually run two instances of folding@home, using schedtool to assign each to a different CPU core, and both to nice +19 and SCHED_IDLEPRIO.

Since booting the 2.6.24 -generic kernels, my system has become severely sluggish -- it will take somewhere between 1/4 and 1/2 second (subjectively) for a character to appear on-screen after hitting the key. This happens with both Metacity and Compiz-Fusion (using git version). Switching to the NV driver reduces this sluggishness slightly, but it's still subjectively worse than the 2.6.22 kernel ever was at its worst.

Stopping my 'niced' folding@home processes immediately alleviates this severe sluggishness, so it seems that somehow these 'nice' tasks are being given a too high priority. I believe this most likely has something to do with the new CFS scheduler in the 2.6.24 kernel.

I tried to reproduce this using a simple busy loop in bash, and with 'yes', and by using cat /dev/zero or cat /dev/urandom > /dev/null, each niced to +19 and SCHED_IDLEPRIO, but for some reason, these did not create the same sluggishness that folding@home creates. In addition, these loads did not show up as 'nice' in my Gnome system monitor panel applet; Instead, these processes showed up as 'system' load, and they also sped up my CPU despite cpufreq being set to ignore 'nice' loads.

[Bryan Quigley (bryanquigley)]

I am affected by this bug as well. I run BOINC and Hardy on two computers and occasionally transfer files between them. Network performance degraded quite badly. Was getting as low as 300KB/s on Ethernet (10/100).

I did testing using:
time md5sum /dev/urandom & pid=$! ; sleep 10 ; killall md5sum
time nice md5sum /dev/urandom & pid=$! ; sleep 10 ; killall md5sum

Results:
bryan@homer:~$ time md5sum /dev/urandom & pid=$! ; sleep 10 ; killall md5sum
[1] 7822
bryan@homer:~$ bash: line 1: 7823 Terminated md5sum /dev/urandom

real 0m10.116s
user 0m0.112s
sys 0m8.589s

[1]+ Exit 143 time md5sum /dev/urandom
bryan@homer:~$ time nice md5sum /dev/urandom & pid=$! ; sleep 10 ; killall md5sum
[1] 7857
bryan@homer:~$ bash: line 3: 7858 Terminated nice md5sum /dev/urandom

real 0m10.028s
user 0m0.120s
sys 0m9.893s

[Daniel Hahler (blueyed)]

Re-assigning the bug task to boinc:
it's related to CONFIG_FAIR_USER_SCHED: each user gets the same amout of CPU slices/cycles and therein the niceness is applied. Therefore, the boinc user gets half of the CPU.
This can be adjusted using the following command, which will give "2" cpu_share units to the boinc user (the default is 1024 and root gets 2048). Caution: assigning 1 there locked my (single core) system!
$ echo 2 | sudo tee /sys/kernel/uids/`id -u boinc`/cpu_share

I will provide a patch for the boinc-client init script.

I've posted about this to http://lkml.org/lkml/2008/1/19/63, but without a reply until now.

[Dana Goyette (danagoyette)]

Well, I don't run my Folding@Home as a different user (actually, I ran it as root, though I suppose I should change that now), and I have noticed the issue still applies even to other non distributed-computing cpu loads:

"I tried to reproduce this using a simple busy loop in bash, and with 'yes', and by using cat /dev/zero or cat /dev/urandom > /dev/null, each niced to +19 and SCHED_IDLEPRIO, but for some reason, these did not create the same sluggishness that folding@home creates. In addition, these loads did not show up as 'nice' in my Gnome system monitor panel applet; Instead, these processes showed up as 'system' load, and they also sped up my CPU despite cpufreq being set to ignore 'nice' loads."

Therefore, this is not just a boinc issue.

[Dana Goyette (danagoyette)]

Oh wait, I just misinterpreted my own comment. What I mean is that issues with 'nice' handling (namely, 'nice' loads not showing up as 'nice') seem to be more complicated than just that user factor.

[Daniel Hahler (blueyed)]

Dana, root has even twice the cpu cycles assigned, therefore when using boinc or something similar as "root" it gets even worse for the "user herself".
You can also create load using something like "python -c 'while 1: i = 1; i += i'".

Maybe just taking SCHED_IDLEPRIO out of the "cfs main loop" would be a good idea?

[dhaval (dhaval-giani)]

Hi,

this is not a regression. The CFS has the group scheduling feature enabled, which is utilized using the Fair User interface. What this means is that the root has the ability to set the importance of every user. Those tunables are accessible at /sys/kernel/uids/<uid>/cpu_share.

You can also have udev scripts which can be written as per http://lkml.org/lkml/2007/10/9/115 .

[By default, root has twice the share of the other user]
[Also this is a propotional bandwidth scheduler. That is if you have two users, giving user A 100 shares and User b 200 shares is the same as giving User A 1000 shares and User B 2000 shares.]

Hope that helps

Thanks
Dhaval

[dhaval (dhaval-giani)]

I am setting this as invalid, as it is *not* a regression.

[Bryan Quigley (bryanquigley)]

I ran my tests as the same user so group scheduling should have had nothing to do with it.

As far as I can tell nice has not been depreciated so it should still function as desired.

Please mark this as valid, again.
Thank you.

[Dana Goyette (danagoyette)]

Hmm, it seems like the only thing being marked as 'invalid' is the 'boinc' assignment; the 'linux' assignment seems to have mysteriously been replaced by the confusing statement: "This bug is not recorded as needing to be fixed in linux (Ubuntu). <Also Needs Fixing Here>"

I sure hope it is being left as valid for 'linux'; the behavior feels like a regression subjectively, even if there is a good reason behind it.

[dhaval (dhaval-giani)]

On 1/20/08, gQuigs <email address hidden> wrote:
> I ran my tests as the same user so group scheduling should have had
> nothing to do with it.
>

I seem to be missing something. I am not sure how scheduling makes a
difference to your test case, unless you are doing something alongside
it. If your script is sleeping for 10seconds, how much time do you
expect time to return?

> As far as I can tell nice has not been depreciated so it should still
> function as desired.
>

nice behaviour though has changed, it maintains its old behaviour only
within a group.

> Please mark this as valid, again.
> Thank you.
>

Thanks
Dhaval

[dhaval (dhaval-giani)]

On 1/20/08, DanaGoyette <email address hidden> wrote:
> Hmm, it seems like the only thing being marked as 'invalid' is the
> 'boinc' assignment; the 'linux' assignment seems to have mysteriously
> been replaced by the confusing statement: "This bug is not recorded as
> needing to be fixed in linux (Ubuntu). <Also Needs Fixing Here>"
>
> I sure hope it is being left as valid for 'linux'; the behavior feels
> like a regression subjectively, even if there is a good reason behind
> it.
>

Hi,

This is not a regression. Ubuntu is configuring the kernel with
CONFIG_FAIR_USER_SCHED, which changes the behaviour. You can utilize
the feature more effectivly with uevent scripts (to which I have
linked in an earlier comment)

CONFIG_FAIR_CGROUP_SCHED will allow finer control of group scheduling,
with the option not to use by not making any groups and putting
everything in just one group.

Thanks,
Dhaval

[I seem to be missing something, I've marked the bug as invalid, does
that mean something other than this bug is not a bug?]

[Dana Goyette (danagoyette)]

Well, if this is by design, then I guess I need a new method of running folding@home -- whether I run it as root or as a new user, even at IDLEPRIO and +19, the folding@home process still slows down other things on the system. The input lag is no longer present, and the process does show up as 'nice', but Compiz drops to half its framerate. It seems odd to me that a process cannot simply yield its CPU cycles to other processes regardless of user. Will I now have to arbitrarily assign a CPU time percentage to the user?

I also figured out the "is not in Linux" issue: my marked-as-duplicate bug had been attached to linux, I made a 'freudian slip' and reached this bug by the /+source/linux/ path, which is not attached to this 'boinc' bug at all.

[dhaval (dhaval-giani)]

On 1/20/08, DanaGoyette <email address hidden> wrote:
> Well, if this is by design, then I guess I need a new method of running
> folding@home -- whether I run it as root or as a new user, even at
> IDLEPRIO and +19, the folding@home process still slows down other things
> on the system. The input lag is no longer present, and the process does
> show up as 'nice', but Compiz drops to half its framerate. It seems odd
> to me that a process cannot simply yield its CPU cycles to other
> processes regardless of user. Will I now have to arbitrarily assign a
> CPU time percentage to the user?
>

If you run it as a different user, you can reduce the weight that
group has by changing the value of /sys/kernel/uids/<uid of
folding@home>/cpu_share

nice should retain behaviour within a group.

Dhaval

[Daniel Hahler (blueyed)]

The boinc task isn't invalid: I'll look into fixing it. I've patched my local init script already (and send the patch to Debian), let's see how it goes.
The "linux" task "went away", because I've assigned it to boinc instead.

[dhaval (dhaval-giani)]

On Jan 21, 2008 5:22 AM, dAniel hAhler <email address hidden> wrote:
> The boinc task isn't invalid: I'll look into fixing it. I've patched my local init script already (and send the patch to Debian), let's see how it goes.

Hi,

Please do not use init scripts. Please use the uevents interface. That
is the preferred interface.

> The "linux" task "went away", because I've assigned it to boinc instead.
>
> ** Changed in: boinc (Ubuntu)
> Assignee: (unassigned) => dAniel hAhler (blueyed)
> Status: Invalid => Confirmed
>
> ** Summary changed:
>
> - 2.6.24-2: Regression with idle cpu cycle handling
> + CFS in 2.6.24 kernel needs cpu_share adjustment for "niced" processes
>

Wrong. FAIR_USER_SCHED needs cpu_share adjustment. We prefer distros
to use FAIR_CGROUP_SCHED which gives much more control. To give fair
user scheduling, we prefer using a daemon which can be configured.

Dhaval

[onyxrev (entp)]

One temporary solution to avoid ditching boinc or folding for those who ran two processes on a dual core system is to run one process. You'll lose some efficiency, yes, but the system doesn't become a hog.

[Daniel Hahler (blueyed)]

Marking it as invalid, not really a bug with boinc.

[Daniel Hahler (blueyed)]

I've filed a new bug 188226 for linux: "Kernel should use CONFIG_FAIR_CGROUP_SCHED".
I'll upload a workaround for boinc to adjust the cpu_share to 2 in the init script until this gets resolved for the kernel.

[Launchpad Janitor (janitor)]

This bug was fixed in the package boinc - 5.10.30-5ubuntu1

---------------
boinc (5.10.30-5ubuntu1) hardy; urgency=low

  * debian/patches/ubuntu_temp_cfs_fix.patch:
    Temporary workaround for the new CFS Linux scheduler,
    by adjusting the "boinc" user's cpu_share to the minimum (2)
    in the init script (LP: #177713)
  * Modify Maintainer value to match the DebianMaintainerField
    specification.

 -- Daniel Hahler <email address hidden> Sat, 02 Feb 2008 02:48:01 +0100

[Dana Goyette (danagoyette)]

I'm adding an assignment to linux-source-2.6.24, because working around the bug in boinc does not fix the underlying cause.

In addition to the folding@home issues, I've also noticed dropouts / skipping in PulseAudio upon doing things such as closing my laptop lid; under the 2.6.22 kernel, I never (or at least rarely) had such issues. I am in the pulse-rt group, so it should be running with realtime priority.

[dhaval (dhaval-giani)]

On Feb 2, 2008 1:01 PM, DanaGoyette <email address hidden> wrote:
> I'm adding an assignment to linux-source-2.6.24, because working around
> the bug in boinc does not fix the underlying cause.
>
> In addition to the folding@home issues, I've also noticed dropouts /
> skipping in PulseAudio upon doing things such as closing my laptop lid;
> under the 2.6.22 kernel, I never (or at least rarely) had such issues.
> I am in the pulse-rt group, so it should be running with realtime
> priority.
>

Let me get it clear. You are seeing pulseaudio skip/dropout in 2.6.24
or 2.6.22. If it is on 2.6.24 please report to the LKML.

Thanks
Dhaval

[Daniel Hahler (blueyed)]

Dana, the "linux" task has been invalidated before.
Please refer to bug 188226 to improve the default kernel behaviour.
I just wanted to provide the workaround for boinc, which I'm using myself.
Dhaval has asked me to use the uevents interface instead, so I'll do a new upload for boinc later, which works around it by providing a udev rules file:
SUBSYSTEM=="kernel", DEVPATH=="/kernel/uids/*", ACTION=="add", RUN+="/usr/lib/boinc-client/udev-usr_share"
and a script to check if it's the boinc user's uid and assign the cpu_share accordingly (/usr/lib/boinc-client/udev-usr_share).

Like said before, this is only meant as a workaround for now and no option for "downloaded programs" like e.g. folding@home. Let's wait for a decision from the ubuntu-kernel team (other bug). Thanks.

[Dana Goyette (danagoyette)]

Aah, okay. Perhaps this bug just needs to be renamed back to being specific to 'boinc' again, and not marked fixed yet since it's not yet a 'proper' fix.

I can confirm, by installing the old .22 kernel, that the audio dropouts only happen with the new .24 kernel. It's especially noticeable with timidity (not as daemon) -- when PulseAudio drops out, the application itself stalls. In addition, with the old kernel, my folding tasks no longer slow down my compiz-fusion desktop. Should I add comments to the linked bug 188226 ?

[JuRiJ (juraj-kvasnica)]

Hi guys, I had same problem with CFS + BOINC + pulseaudio on my Pentium 4 HT 2.8GHz. The trouble was caused by using schedtool to run BOINC. After I removed it (sudo aptitude purge schedtool), there was no pulseaudio stuttering (despite pulseaudio is using 20% of my CPU... which is IMHO pretty high). This bug is 100% reproducible on Debian with 2.6.26 kernels. I have default pulseadio config (and changed only number of default-output-channels from 2 to 6), and while listening to music (running audacious audio player with pulse output), and running BOINC at the same time I have no stuttering. Then if I install schedtool (sudo aptitude install schedtool) and restart BOINC (sudo /etc/init.d/boinc-client restart) the music starts to lag, skip etc... then removing schedool (sudo aptitude purge schedtool) and restarting BOINC again makes the music run fine (e.g. no stuttering and skips).
I'm not 100% sure this will help you guys with Ubuntu systems, but since Debian and Ubuntu are pretty close, I think it could be the same problem...
Schedtool isn't installed by default on Debian(same for Ubuntu), but it happened that I used it to run BOINC, before 2.6.24 kernel with CFS was released...

PS: this bug isn't related to CONFIG_FAIR_USER_SCHED, where all users have same priority and it slows systems with BOINC running, because BOINC is running under boinc user, which has 50% of CPU share. It's for those who got sound issues, even after using kernels with CONFIG_FAIR_USER_SCHED turned off...