Kernel 2.6.24-23-rt locks up hard under heavy rt load

Bug #324275 reported by Matthias Lüscher on 2009-02-02
14
Affects Status Importance Assigned to Milestone
linux-rt (Ubuntu)
Undecided
Unassigned
Hardy
Medium
Alessio Igor Bogani
Jaunty
Undecided
Unassigned

Bug Description

Description: Ubuntu 8.04.2
Release: 8.04
linux-rt:
  Installed: 2.6.24.23.25
  Candidate: 2.6.24.23.25
  Version table:
 *** 2.6.24.23.25 0
        500 http://ch.archive.ubuntu.com hardy-updates/multiverse Packages
        100 /var/lib/dpkg/status
     2.6.24.22.24 0
        500 http://security.ubuntu.com hardy-security/multiverse Packages
     2.6.24.16.18 0
        500 http://ch.archive.ubuntu.com hardy/multiverse Packages

The latest rt-kernel locks up hard under heavy load:
The problem can be reproduced easily by using the cyclictest program (http://git.kernel.org/?p=linux/kernel/git/tglx/rt-tests.git) with the following command line options:
sudo ./cyclictest -p99 -t10 -n -i250

Additional observations:
- The problem got introduced with the upgrade to the rt21 patch - it did not happen with older kernel versions based on previous rt patches.
- The lockup is really hard: there is no output on a netconsole and even the magic sysrequest keys to reboot the system (Alt+SysRQ+b) do not work anymore.
- It also happens to a vanilla kernel with the rt21 patch (I did not try any newer patches yet)
- with a lower realtime priority the system seems to run at least more stable (e.g. sudo ./cyclictest -p95 -t10 -n -i250)
- the problem seems to appear only on multicore systems

Matthias Lüscher (m-luescher) wrote :

The above mentioned problem can also be reproduced with vanilla 2.6.24.7 with the latest rt patch (rt26).

Matthias Lüscher (m-luescher) wrote :

The lockup is currently being discussed on linux-rt-users (http://marc.info/?t=123360942700006&r=1&w=2). It looks like the issue is related to a patch that was introduced with 2.6.24.7-rt13.

Matthias Lüscher (m-luescher) wrote :

The fix for the crash can be found here:
http://marc.info/?l=linux-rt-users&m=123393014501532&w=2

The described patch should get included into the quilt series of the rt kernel:
[ubuntu/ubuntu-hardy.git]/debian/binary-custom.d/rt/patchset/

Matthias Lüscher (m-luescher) wrote :

The above mentioned problem might also crash ubuntu studio (jackd, http://marc.info/?l=linux-rt-users&m=123323476722881&w=2, 2.6.26.7-rt13 had the same problem).

The attached patch has made it into 2.6.24.7-rt27.

Changed in linux-rt:
assignee: nobody → abogani
status: New → In Progress
Changed in linux-rt:
importance: Undecided → Medium
status: In Progress → Fix Committed
Alessio Igor Bogani (abogani) wrote :

The fix will be available with first kernel after 2.6.24-24.50.

Martin Pitt (pitti) wrote :

Please clarify the situation in Jaunty and close the task if it is already fixed.

tags: added: verification-needed
Martin Pitt (pitti) wrote :

Accepted linux into hardy-proposed; please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Alessio Igor Bogani (abogani) wrote :

Jaunty is not affected.

Changed in linux-rt:
assignee: nobody → abogani
importance: Undecided → Medium
status: New → Fix Committed
status: Fix Committed → Invalid
assignee: abogani → nobody
importance: Medium → Undecided
stryd_one (stryd-one) wrote :

After installing these updates my onboard sound, virtual midi, lirc and USB TV tuner failed. Rolled back to .50 and all is well. Errors in the logs are as attached.

Alessio Igor Bogani (abogani) wrote :

@stryd_one,

Do you have tested 51-generic also?

Thanks!

stryd_one (stryd-one) wrote :

Sorry Alessio... it took me a few hours to get it back to working order (my first experience with rolling back kernel updates), so I haven't found such a window of opportunity since then... I'll try it ASAP.

stryd_one (stryd-one) wrote :

Short version: Still no good with .51-rt, but okay with .51-generic. I'm back to 2.6.24-23-rt for the time being.

Long version: Using 2.6.24-24.51-rt, I still have logs full of those symbol errors (same thing repeated for all the devices involved). I didn't attach logs because it's the same thing with new timestamps, but I'm happy to if you need them.
Using 2.6.24-24.51-generic, I didn't see any symbol errors, but I did have some troubles with my TV Tuner, which works OK with 2.6.24-24.50-rt and -generic.

I suspected this to be related to the DKMS build of v4l-dvb I'm using and not directly related to this patch. I did notice that libc was updated too though, so perhaps that has a part to play in this? I gather that the DKMS system builds the kernel module automatically when the kernel is updated. It would have used libc to do so, so perhaps this explains the malfunctional driver. I wanted to be complete, so I thought it worth a mention. Otherwise the .51-generic kernel works OK, sound and midi and lirc are behaving.

I'm too linux-newb to know what this indicates... Is there anything I can do to assist, or is this more likely to be a problem with my particular machine?

stryd_one (stryd-one) wrote :

Any word on this? There is an update I'm being alerted to today, to 2.6.24-23.52, but I'm concerned about installing that update as that is now my only working kernel version... Sorry to nag!

stryd_one (stryd-one) wrote :

Don't suppose you'd like to know what was wrong with the package?

Rolf Leggewie (r0lf) wrote :

Hardy has seen the end of its life and is no longer receiving any updates. Marking the Hardy task for this ticket as "Won't Fix".

Changed in linux-rt (Ubuntu Hardy):
status: Fix Committed → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Bug attachments