RT_GROUP_SCHED kernel option makes JACK and Rtkit unusable in Ubuntu Natty

[Ronan Jouchet]

After talking with adi in #ffado and las in #jack, it appears that Natty's kernel needs adjustment to be usable for audio work. Currently the kernel flavors shipped by Ubuntu have the RT_GROUP_SCHED option activated, which makes JACK unusable (las put up http://jackaudio.org/linux_group_sched to sum up the situation).

*Until this problem is solved, JACK is unusable under Natty*. Las mentioned several solutions:
  0. Disabling RT_GROUP_SCHED for all kernels
  1. Disabling RT_GROUP_SCHED for the kernel targeting audio users (-lowlatency, currently living in abogani's PPA)
  2. Set up a cgroup and invoke JACK and every client from a shell script that joins the cgroup first
  3. Use cgrules.conf to at least put processes into the right group by default
  4. Use Ubuntu's specific mechanism to put user tasks into a RT-enabled cgroup

Quoting las: "(1) is the desirable option; (2) would be next and (3) would be hard, because it would enforce 'application <foobar> always run in the <bazbar> cgroup' which is not really very useful or correct". (0) may also be considered since this kernel option is apparently very specialized for embedded systems and doesn't make sense for a desktop distro.

Which path should be followed is beyond my knowledge. I just hope this discussion will pave the way to a usable audio stack in Natty.
IRC resources: #jack, #ubuntustudio, #ubuntustudio-devel. Full IRC log here: http://pastebin.com/2yw7aPhr

[Ronan Jouchet (ronj)]

Mailing list thread:
https://lists.ubuntu.com/archives/ubuntu-studio-devel/2010-December/002881.html

[David Henningsson (diwic)]

$ grep rtkit /var/log/syslog

Dec 14 08:38:29 ubuntu rtkit-daemon[3429]: Successfully called chroot.
Dec 14 08:38:29 ubuntu rtkit-daemon[3429]: Successfully dropped privileges.
Dec 14 08:38:29 ubuntu rtkit-daemon[3429]: Successfully limited resources.
Dec 14 08:38:29 ubuntu rtkit-daemon[3429]: Running.
Dec 14 08:38:29 ubuntu rtkit-daemon[3429]: Failed to make ourselves RT: Operation not permitted
Dec 14 08:38:29 ubuntu rtkit-daemon[3429]: Failed to make ourselves RT: Operation not permitted
Dec 14 08:38:29 ubuntu rtkit-daemon[3429]: Watchdog thread running.
Dec 14 08:38:29 ubuntu rtkit-daemon[3429]: Canary thread running.
Dec 14 08:38:29 ubuntu rtkit-daemon[3429]: Failed to make ourselves RT: Operation not permitted
Dec 14 08:38:31 ubuntu rtkit-daemon[3429]: last message repeated 13 times
Dec 14 08:40:34 ubuntu rtkit-daemon[3429]: Failed to make ourselves RT: Operation not permitted
Dec 14 08:40:34 ubuntu rtkit-daemon[3429]: last message repeated 10 times
Dec 14 08:40:34 ubuntu rtkit-daemon[3429]: Failed to make ourselves RT: Operation not permitted
Dec 14 08:40:35 ubuntu rtkit-daemon[3429]: last message repeated 13 times
Dec 14 08:40:35 ubuntu rtkit-daemon[3429]: Warning: Reached burst limit for user '999', denying request.
Dec 14 09:20:49 ubuntu rtkit-daemon[3429]: Failed to make ourselves RT: Operation not permitted
Dec 14 09:20:50 ubuntu rtkit-daemon[3429]: last message repeated 10 times
Dec 14 09:20:50 ubuntu rtkit-daemon[3429]: Failed to make ourselves RT: Operation not permitted
Dec 14 09:20:50 ubuntu rtkit-daemon[3429]: Failed to make ourselves RT: Operation not permitted
Dec 14 09:20:50 ubuntu rtkit-daemon[3429]: Failed to make ourselves RT: Operation not permitted
Dec 14 09:20:50 ubuntu rtkit-daemon[3429]: Failed to make ourselves RT: Operation not permitted
Dec 14 09:20:50 ubuntu rtkit-daemon[3429]: Failed to make ourselves RT: Operation not permitted
Dec 14 09:20:50 ubuntu rtkit-daemon[3429]: last message repeated 9 times
Dec 14 09:20:50 ubuntu rtkit-daemon[3429]: Warning: Reached burst limit for user '999', denying request.

[Trulan Martin (trulanm)]

Alessio's latest lowlatency kernel build (2.6.37-9.22~ppa1) has RT_GROUP_SCHED turned off. This takes care of this problem for me.
https://launchpad.net/~abogani/+archive/ppa

[Ronan Jouchet (ronj)]

Confirmed.

Leaving the decision to mark as resolved to a kernel maintainer, since maybe a better solution can be implemented, preserving this features but implementing proper cgroups management...

[Ronan Jouchet (ronj)]

Lots of things are happening on the JACK-devel mailing list, and among them, a proper patch. This bug should definitely remain open to track inclusion of these patches.

[David Henningsson (diwic)]

Seems like the current recommended fix, as of http://lists.jackaudio.org/private.cgi/jack-devel-jackaudio.org/2011-January/006752.html is (quoting for those of you who can't access the above) :

Its quite straightforward, the config file at /etc/sysconfig/cgconfig should say
CREATE_DEFAULT=no as opposed to CREATE_DEFAULT=yes

...in Debian/Ubuntu, this is /etc/default/cgconfig

I'll test that solution shortly.

[David Henningsson (diwic)]

On 2011-01-02 05:18, Dhaval Giani wrote:
> On Sun, Jan 2, 2011 at 12:56 AM, David Henningsson
> <email address hidden> wrote:
>> On 2010-12-24 10:08, Lennart Poettering wrote:
>>>
>>> On Fri, 24.12.10 09:53, Dhaval Giani (<email address hidden>) wrote:
>>>
>>>>> I do believe that the short time fix for this problem is that
>>>>> libcgroup/Ubuntu should stop sorting processes into any non-root cgroup
>>>>> hierarchy by default.
>>>>
>>>> Ubuntu needs to fix that.
>>>
>>> Yes, they do. David?
>>
>> Guess that's me. I'm not really a scheduling expert and had never heard of
>> cgroups before this bug showed up, but I'll try to grab the right people at
>> Canonical's next sprint, which is 10-14 January. That is, unless somebody
>> fixes it sooner.
>>
>
> Its quite straightforward, the config file at /etc/sysconfig/cgconfig should say
> CREATE_DEFAULT=no as opposed to CREATE_DEFAULT=yes

Hmm, it doesn't seem to be that simple unfortunately. There is no such
file present. There is a package called "cgroup-bin" that adds something
similar into /etc/default/cgconfig, but that package isn't installed
either. Not even the cgroup library package is installed.

http://git.debian.org/?p=collab-maint/libcgroup.git;a=blob;f=debian/cgroup-bin.cgconfig.default;h=5a61bf6e9878250557f3c4993376c7c408135b89;hb=HEAD

I also tried manually adding a /etc/sysconfig/cgconfig file with
"CREATE_DEFAULT=no" in it, but it didn't help either.

--
David Henningsson, Canonical Ltd.
http://launchpad.net/~diwic

[David Henningsson (diwic)]

It might be that Ubuntu is bitten by this instead:
https://lkml.org/lkml/2011/1/10/52

We indeed added the autogroup patch (or some version of it) to 2.6.37
(for testing only - 11.04 will use 2.6.38) so it seems reasonable.

// David

[Paul Davis (paul-linuxaudiosystems)]

On Mon, Jan 10, 2011 at 4:28 PM, David Henningsson
<email address hidden> wrote:
> It might be that Ubuntu is bitten by this instead:
> https://lkml.org/lkml/2011/1/10/52
>
> We indeed added the autogroup patch (or some version of it) to 2.6.37 (for
> testing only - 11.04 will use 2.6.38) so it seems reasonable.

that's not a different bug. the idea of turning off "auto cgroup" is
to leave tasks in the root cgroup with has RT bandwidth already
allocated.

[Jeremy Foshee (jeremyfoshee)]

Hi Ronan,

Please be sure to confirm this issue exists with the latest development release of Ubuntu. ISO CD images are available from http://cdimage.ubuntu.com/daily/current/ . If the issue remains, please run the following command from a Terminal (Applications->Accessories->Terminal). It will automatically gather and attach updated debug information to this report.

apport-collect -p linux 690010

Also, if you could test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

    [This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

[David Henningsson (diwic)]

@JFo, so current status is
 * The autogroup stuff, aka the "magic responsiveness patch", breaks RT scheduling
 * We brought this patch in with 2.6.37 whereas upstream doesn't bring it in until 2.6.38; that's why we're seeing this before anybody else
 * Following the lkml thread, they're still discussing the right way to solve it

[David Henningsson (diwic)]

According to https://lkml.org/lkml/2011/1/17/105 a patch has now been applied. It would be nice to see if this solves our problem as well, I'm hopeful about that.

[Ronan Jouchet (ronj)]

Is this patch in a PPA yet, or is it currently required to compile it?

[David Henningsson (diwic)]

Tested http://kernel.ubuntu.com/~kernel-ppa/mainline/v2.6.38-rc1-natty/
The bug is present in that version. The fix didn't make it into 2.6.38, but is committed as http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=f44937718ce3b8360f72f6c68c9481712517a867 so I hope it will be in 2.6.38-rc2.

[David Henningsson (diwic)]

Confirmed fixed in v2.6.38-rc2. Soon coming to a kernel near you :-)

[Conor Curran (cjcurran)]

Excellent, thx guys.

[David Henningsson (diwic)]

Confirmed fixed in 2.6.38-1-generic

[Brian Bergstrom (boilingbergstrom)]

Which solution path was followed?

[David Henningsson (diwic)]

@Brian, this commit fixed the bug: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=f44937718ce3b8360f72f6c68c9481712517a867

Ubuntu is taking the upstream version, which includes this fix.