boinc does not use low-priority processes

[mokabar]

boinc seems to run its jobs with a niceness of 19 as user `boinc'.

when i am running 4 processes (gcc) on a 4-core machine at a niceness of 5, i'd expect the boinc process to be idle (because it has a lower niceness). but in fact the boinc process takes one full CPU, while the user jobs are restricted to the other CPUs. it seems that the scheduler interprets the niceness not globally, but per-user (maybe related to cgroups)?

iac, it would be great, if this could be addressed.

[Andris (andris-redhat-bugs)]

Description of problem:

I noticed that on 2 different systems upgraded to Fedora 15pre x86_64 BOINC projects does not obey nice settings when system has another load. Both systems
have dual core CPUs.

Version-Release number of selected component (if applicable):

boinc-client-6.10.58-3.r22930.fc15.x86_64
kernel 2.6.38.5-24.fc15.x86_64

How reproducible:

Always

Steps to Reproduce:
1. start BOINC client and have some BOINC projects running
2. start some other process with the default priority (I used 2 copies
    of simple C++ program that wastes CPU time like 'while(true){}'

Actual results:

top shows that boinc projects are getting about same amount of CPU time
as ather processes running with the default priority (see snapshot of top
output below)

Expected results:

BOINC projects are expected to get several % of CPU time when there are
processes with normal priority requiring enough CPU time.

Additional info:

1) snapshot for top on one of these Fedora 15 x86_64 systems:
top - 20:06:58 up 30 min, 7 users, load average: 3.84, 3.53, 2.69
Tasks: 179 total, 5 running, 174 sleeping, 0 stopped, 0 zombie
Cpu(s): 51.7%us, 3.0%sy, 45.3%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 4024908k total, 2202032k used, 1822876k free, 42092k buffers
Swap: 4395004k total, 0k used, 4395004k free, 937132k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 6807 andris 20 0 11828 688 572 R 48.5 0.0 0:11.55 wastecpu
 6806 andris 20 0 11828 688 572 R 46.5 0.0 0:12.88 wastecpu
 2700 boinc 39 19 216m 211m 2660 R 44.6 5.4 19:32.91 einsteinbinary_
 3213 boinc 39 19 267m 261m 1356 R 43.6 6.7 15:39.84 einstein_S5GC1H
 1721 root 20 0 187m 30m 12m S 3.0 0.8 1:40.41 X

Tried to renice boinc project executable to 10. In this case it got about
70% of CPU time of 1 core.

nice seems to work OK between several my processes

So it seems that nice 19 for boinc processes acts as nice 0 of normal user
process.

This is clear regression in comparisson with Fedora 14 x86_64 (and earliers)
where I have necer observed similar behaviour.

[Andris (andris-redhat-bugs)]

The problem is how to configure BOINC to run as real background task when systemd and cgroups are being used. boinc-client package should be configured in that way, so after installation (or upgrade) it would run really in background instead of getting the same cpu time share as all some other normal user processes together.

[Andris (andris-redhat-bugs)]

Workarounded problem by
1) adding backgromg CGROUP

--- /etc/cgconfig.conf-20110526 2011-05-26 08:49:07.794303903 +0300
+++ /etc/cgconfig.conf 2011-05-26 08:49:49.666429924 +0300
@@ -24,3 +24,8 @@
        net_cls = /sys/fs/cgroup/net_cls;
 }

+group background {
+ cpu {
+ cpu.shares = 1;
+ }
+}

2) telling boinc-client service to use cpu:/background group

--- /etc/sysconfig/boinc-client-20110526 2011-05-26 08:51:06.530832500 +0300
+++ /etc/sysconfig/boinc-client 2011-05-26 08:51:37.530188219 +0300
@@ -40,3 +40,5 @@
 #BOINCOPTS="--check_all_logins"
 # The default is no options.
 #BOINCOPTS=
+
+CGROUP_DAEMON="cpu:/background"

After that BOINC client seems to work OK in background (tested on 2
systems: both Fedora 15 x86_64)

I guess it would be good idea if this would happen automatically without need
to additionally hack config files.

[Andris (andris-redhat-bugs)]

It seems that setup mentioned above is highly unreliable
- starting cgconfig.service often fails at boot time and as result
   cpu:/background group is not created
- it is often impossible to start cgconfig.service also manually (it simply
   fails, no matter what I tried:
      - systemctl start cgconfig.service
      - service cgconfig start
      - /etc/init.d/cgconfig start
   also with attempting to stop it at first

In these cases boinc-client service does not start (well I'm getting message
that it is started, but 'systemctl status boinc-client.service' says that it has
exited with return code 0)

So I reverted modifications to /etc/cgconfig.conf and put the following
into /etc/sysconfig/boinc-client:

------------------------------------------------------------
if ! [ -d /sys/fs/cgroup/cpu/background ] ; then
    mkdir -p /sys/fs/cgroup/cpu/background
    echo 1 >/sys/fs/cgroup/cpu/background/cpu.shares
fi

CGROUP_DAEMON="cpu:/background"
------------------------------------------------------------
(the idea is taken from 'man cgconfig.conf')

[Milos (milos-redhat-bugs)]

Andris,

thank you very much for your detailed bugreport and sorry for the delay. I'm about to release a boinc update that solves this issue (using more or less your solution, except for some details -- minimum allowed cpu.share is 2 and I use the cg* tools to manipulate cgroups + configuration is in the init script so that it can be overridden by user in sysconfig file).

[Fedora (fedora-redhat-bugs)]

boinc-client-6.12.35-1.r24014svn.fc16 has been submitted as an update for Fedora 16.
https://admin.fedoraproject.org/updates/boinc-client-6.12.35-1.r24014svn.fc16

[Fedora (fedora-redhat-bugs)]

boinc-client-6.12.35-1.r24014svn.fc15 has been submitted as an update for Fedora 15.
https://admin.fedoraproject.org/updates/boinc-client-6.12.35-1.r24014svn.fc15

[Fedora (fedora-redhat-bugs)]

Package boinc-client-6.12.35-1.r24014svn.fc16:
* should fix your issue,
* was pushed to the Fedora 16 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing boinc-client-6.12.35-1.r24014svn.fc16'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/boinc-client-6.12.35-1.r24014svn.fc16
then log in and leave karma (feedback).

[Fedora (fedora-redhat-bugs)]

boinc-client-6.12.35-1.r24014svn.fc16 has been pushed to the Fedora 16 stable repository. If problems still persist, please make note of it in this bug report.

[Fedora (fedora-redhat-bugs)]

boinc-client-6.12.35-1.r24014svn.fc15 has been pushed to the Fedora 15 stable repository. If problems still persist, please make note of it in this bug report.

[Bryan Quigley (bryanquigley)]

Red hat bug says it should be fixed in the next version of BOINC.. It should be fixed in Precise at least.

[Daniel Hahler (blueyed)]

The Red Hat bug says that the upload to Fedora fixed it, not that it has been fixed upstream.

From skimming the report it seems that they've added cgroup configuration to fix this.

[mokabar (tim-klingt)]

fwiw, it seems that nice does not have any effect on systems which have sched_autogroup enabled ...

[Gianfranco Costamagna (costamagnagianfranco)]

so are you suggesting to use a cgroup patch?

I can provide it

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Bryan Quigley (bryanquigley)]

I see setting the scheduler to ignore nice load when ramping to be more important: /sys/devices/system/cpu/cpufreq/ondemand/ignore_nice_load

Should I report a new bug on this or is this related to this one?

[Gianfranco Costamagna (costamagnagianfranco)]

Please try this ppa and boinc 7.8.1, and let me know if the problem still occurs by setting back the status to "new"
https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/boinc/

[Gianfranco Costamagna (costamagnagianfranco)]

We might need to address this again in systemd service script

[Bryan Quigley (bryanquigley)]

This was fixed in systemd service file everywhere.