/var/log fills up with "all normal" messages @ about 575/sec fill up the available space

[Taras]

Forum thread is here

http://ubuntuforums.org/showthread.php?t=1284691&page=1

System Karmic Beta i386

Machine Dell Inspiron 1000 Laptop
Celeron 2.4 running 256MB RAM 40 GB HDD

Partitioning:
SDA1 100 MB with /boot
SDA2 764 MB swap
SDA3 6.4 GB with /
SDA4 29.5 GB /home

Problem:

After the initial install from alternate cd, and system update to -14 kernel version, the computer could not boot in to gnome or xterm.

“The configuration defaults for GNOME Power Manager have not been installed correctly. Please contact your computer administrator."

further investigation showed that the disk is full and in particular the /var/log directory occupied all available space for root. These particular files are a problem:

sys.log is 1.7 GB

kern.log is 1.6 GB

messages are 700 mb

copies are available if needed.

Dmesg is attached.

[lavinog (lavinog)]

Although there is another issue here, the size of the log files made it difficult for the user to get help. Even with the logs compressed the forums wouldn't permit uploading the file. Logs this size hinder troubleshooting, and can mask other problems by using up all available storage space.

I recommend setting a default size limit using logrotate.
Ubuntuforums has a limit of 976.6Kb for attachments. Email typically has a limit of 10M.

Another idea would be to implement a notification for when the log size is abnormally high. Currently the only notification that something was wrong was that there was no free space on the drive.

[Michael Vogt (mvo)]

What does the log contain? I do not want it attached here, just to get a idea if its duplicating a lot of data or if its full of "useful" information.

[Taras (tiras-dude)]

messages (600 some megs)

Oct 15 00:42:34 DELL-1000 kernel: [ 26.814694] CPU0: Temperature/speed normal

8238668 instances @ about 575 times a second

syslog (1.6 gigs

Oct 15 00:42:31 DELL-1000 kernel: [ 23.568556] CPU0: Temperature above threshold, cpu clock throttled (total events = 13710)
Oct 15 00:42:31 DELL-1000 kernel: [ 23.569662] CPU0: Temperature/speed normal

16478398 instances at about the same frequency

 kern.log

Showed much the same situation as in syslog, with ocasionall intermix of the following below:

Oct 15 00:42:27 DELL-1000 kernel: [ 19.676242] b43-pci-bridge 0000:02:00.0: enabling device (0000 -> 0002)
Oct 15 00:42:27 DELL-1000 kernel: [ 19.676270] CPU0: Temperature above threshold, cpu clock throttled (total events = 11589)
Oct 15 00:42:27 DELL-1000 kernel: [ 19.676302] b43-pci-bridge 0000:02:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
Oct 15 00:42:27 DELL-1000 kernel: [ 19.676343] b43-pci-bridge 0000:02:00.0: setting latency timer to 64
Oct 15 00:42:27 DELL-1000 kernel: [ 19.680520] ssb: Sonics Silicon Backplane found on PCI device 0000:02:00.0
Oct 15 00:42:27 DELL-1000 kernel: [ 19.681424] CPU0: Temperature/speed normal
Oct 15 00:42:27 DELL-1000 kernel: [ 19.681880] CPU0: Temperature above threshold, cpu clock throttled (total events = 11590)
Oct 15 00:42:27 DELL-1000 kernel: [ 19.683063] CPU0: Temperature/speed normal

So I am not sure is that pc was melting or not =)

more details with actual logs (via rapidshare) can be found on the tread:

http://ubuntuforums.org/showthread.php?t=1284691&page=3

[Michael Vogt (mvo)]

There are two bugs here, one that the kernel should not support this at this rate. The other that rsyslog should deal with repeated messages (better?) and do not write them out at this speed. I reassign.

[Michael Terry (mterry)]

I've been handling rsyslog this cycle, but I can't look at rsyslog today. I can next week. If someone wants to peek before then, good on ya.

[Michael Vogt (mvo)]

This diff should turn on the duplication detection in rsyslog:

diff -u rsyslog-4.2.0/debian/rsyslog.conf rsyslog-4.2.0/debian/rsyslog.conf
--- rsyslog-4.2.0/debian/rsyslog.conf
+++ rsyslog-4.2.0/debian/rsyslog.conf
@@ -35,6 +35,9 @@
 #
 $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

+# Filter duplicated messages
+$RepeatedMsgReduction on
+
 #
 # Set the default permissions for all log files.
 #
diff -u rsyslog-4.2.0/debian/changelog rsyslog-4.2.0/debian/changelog
--- rsyslog-4.2.0/debian/changelog
+++ rsyslog-4.2.0/debian/changelog
@@ -1,3 +1,10 @@
+rsyslog (4.2.0-2ubuntu6) karmic; urgency=low
+
+ * enable $RepeatedMsgReduction to avoid bloating the syslog
+ file (LP: #453444)
+
+ -- Michael Vogt <email address hidden> Fri, 23 Oct 2009 17:28:10 +0200
+
 rsyslog (4.2.0-2ubuntu5) karmic; urgency=low

   Upstart fixups; LP: #430220

[Michael Terry (mterry)]

Looks reasonable. Please push, mvo, and thanks for looking into this one!

[Steve Langasek (vorlon)]

Has anyone tested that $RepeatedMsgReduction works sensibly? The actual change is small and makes sense, but I have no idea about the precise semantics of this option and what other bugs this might cause.

I think this is best left for an SRU, where we can test it without rushing.

[Michael Vogt (mvo)]

Thanks Steve,

sorry that I have not added more background to the fix. I tested this change with the attached test program. Before the change each line is printed, afterwards the behaviour is similar to the old syslog that prints " last message repeated 62 times".

[Michael Vogt (mvo)]

The upstream documentation for the option is here http://www.rsyslog.com/doc-rsconf1_repeatedmsgreduction.html

[Amit Kucheria (amitk)]

Commit b417c9fd8690637f0c91479435ab3e2bf450c038 in Linus' tree fixes the problem of the kernel spamming the logs.

This commit is in the ubuntu kernel now.

[Ariel Faigon (ariel.faigon)]

Seeing the same problem on my desktop.

Another side effect is 'logcheck' never being able to catch up with the logs so multiple logcheck processes accumulate over time and the system becomes overloaded with logcheck processes taking all CPUs at close to 100% utilization.

Seems like pretty high severity to me.

[Ariel Faigon (ariel.faigon)]

Sorry to have to add one more piece of info.

The suggested rsyslog.conf duplication mitigation is not very effective because when you have many CPUs the messages are not identical dups and so they aren't being suppressed.

e.g. after adding
    $RepeatedMsgReduction on
to /etc/rsyslog.conf, and rebooting.

I'm still getting alternating CPU0/CPU4 messages, multiple times per sec.

Oct 31 01:13:03 xx kernel: [ 1932.745006] CPU0: Temperature/speed normal
Oct 31 01:13:03 xx kernel: [ 1932.745007] CPU4: Temperature/speed normal
Oct 31 01:13:03 xx kernel: [ 1932.916289] CPU0: Temperature/speed normal
Oct 31 01:13:03 xx kernel: [ 1932.916291] CPU4: Temperature/speed normal
Oct 31 01:13:03 xx kernel: [ 1933.036344] CPU0: Temperature/speed normal
Oct 31 01:13:03 xx kernel: [ 1933.036346] CPU4: Temperature/speed normal

IOW: the kernel fix is pretty urgent here. Thanks.

[Scott Ritchie (scottritchie)]

I had this one affect me. A fun workaround I discovered was to prevent the CPU from getting hot by placing my laptop on top of a cooling rack. No heating -> no overheating -> no excessive logging -> no full disk -> program no longer crashes due to lack of tmp file space.

[sKwa (setarckos)]

the same problem to me - no log rotation and I getting CPU messages too (patch didn't help me)

$ uname -a
Linux ubuntu 2.6.31-14-generic #48-Ubuntu SMP Fri Oct 16 14:05:01 UTC 2009 x86_64 GNU/Linux

$ lsb_release -a
...
Release: 9.10
Codename: karmic

$ df -ha
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 9.9G 9.4G 0 100% /
...

$ ls -lh /var/log/ | grep -r "M\|G"
total 6.6G
-rw-r----- 1 syslog adm 2.8G 2009-11-03 03:20 kern.log
-rw-r----- 1 syslog adm 1.1G 2009-11-03 03:20 messages
-rw-r----- 1 syslog adm 705M 2009-11-03 03:20 syslog
-rw-r----- 1 syslog adm 2.1G 2009-11-02 08:07 syslog.1

[Adrian Tritschler (ajft)]

All I can say is it is affecting my system too; P4 Shuttle system that has about 4G of logs saying, endlessly:

Nov 4 08:38:22 fafnir kernel: [841531.434500] CPU1: Temperature/speed normal
Nov 4 08:38:22 fafnir kernel: [841531.434503] CPU0: Temperature/speed normal
Nov 4 08:38:22 fafnir kernel: [841531.651247] CPU0: Temperature/speed normal
Nov 4 08:38:22 fafnir kernel: [841531.651251] CPU1: Temperature/speed normal
Nov 4 08:38:22 fafnir kernel: [841531.660400] CPU0: Temperature/speed normal
Nov 4 08:38:22 fafnir kernel: [841531.660405] CPU1: Temperature/speed normal

[Peter Wagner (o-petris)]

Hello,

just a useful hint on how to get into the system: After some time of working with the quite slow Karmic version (now I know why) and a regular shutdown, the next day I couldn't run Ubuntu any more. After entering the password and some minutes of intensive working, the system returned to the login form and so on. The login form was without colours and without selection bars at the bottom. In the recovery mode I could get up to a command line and was able to see files.

This bug showed me, what's going on (thanks!). After some hesitation (I'm no expert) I have backuped the huge files "syslog", "messages" and "kern.log" and replaced them by tall or empty ones (e.g. sudo cp syslog.1 syslog ). The second start after this was completely successful.

Now the system is still slow and after some time it gives warnings about stuffed root disk --> shutdown, clean up the logs and re-run.

In my case the message about the temperature toggles between OK and NON-OK, like in a children's rhyme with the flower: she loves me, she doesn't, ....
Nov 4 14:30:28 peter-desktop kernel: [ 1281.883345] CPU0: Temperature above threshold, cpu clock throttled (total events = 645050)
Nov 4 14:30:28 peter-desktop kernel: [ 1281.884631] CPU0: Temperature/speed normal
Nov 4 14:30:28 peter-desktop kernel: [ 1281.884657] CPU0: Temperature above threshold, cpu clock throttled (total events = 645051)
Nov 4 14:30:28 peter-desktop kernel: [ 1281.885943] CPU0: Temperature/speed normal

[Taras (tiras-dude)]

As a temporary fix I can suggest setting up a separate partition for /var/log. Say about 300 megs to a gig depending on the hdd size. Now this will probably require non-primary partitions to be setup, but at least it will keep the logs at bay until the patch is out.

The other solution is more involved, solving the overheating issues, in my case resulted in a disassembly of the laptop, removal of 3 mm thick layer of dust from the heat sink and replacement of factory heat pads with thermal grease. Now running 30 Degrees C cooler (down to 54C from 85C). No more monster logs.

Unfortunately in the process I killed the keyboard. Anyone has one for Dell Inspiron 1000 lying around?

[Michael Vogt (mvo)]

The kernel messages are not de-duplicated because they are not exactly the same. The timestamp of the msg is always different. A option is to remove the timestamp when we read the message from the kernel (because syslog writes a timestap as well). This does loose information however because the kerneltimetamp is more accurate and its also useful to correlate this uptime format to potentional issues.

[Martin Pitt (pitti)]

Enabling the option in an SRU seems a bit risky to me, since it wasn't exposed to testing during the karmic cycle. Michael tested this, and enabling it fixes a regression to syslog, so let's accept this and bake in -proposed for a while.

I veto against removing kernel timestamps in an SRU, too much string fiddling. It seems that the main cause for this were the temperature messages, which have been fixed already. Excessive logging from the kernel should be fixed at the root (in the kernel itself).

[Martin Pitt (pitti)]

Accepted rsyslog into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Michael Vogt (mvo)]

Thanks Martin, I'm fine with ignoring the kernel timestamps for now, its not a regression. As for the kernel, I still get reports that its not fixed there, I will ask the kernel team about it (Scott Ritchie mentioned it on irc today).

[perspectoff (perspectoff)]

There are three problems:

1) the libsensors4 package does not contain the drivers for certain motherboard sensors. The bug is that It returns an "out of range" error repeatedly (many times per minute) instead of a "no sensors found" error, which ought to trigger completely different behavior vis a vis CPU frequency scaling. This is bug #1: there should be a way to turn this loop off if there are no driver for the hardware. I experienced this on 3 different machines, none of which have drivers for the motherboard sensors.

2) the CPU throttling loop (CPU frequency scaling) is triggered by the aberrant sensor error messages. Fortunately, this is easy to turn off using the debian runtime utility rcconf, where CPU frequency ONDEMAND can be unticked.

3) the frequent kernel error messages being written to the log from the aberrant libsensor errors. This is tricky, as it is not necessarily desirable to prevent errors from being written to the log. it is better to fix the root problems.

[Patrick Waks (wazpys)]

Enabled karmic-proposed yesterday, and made an update. I saw that it added the new rsyslog package, but I am still getting all these lines in three logs, kern.log, syslog as well as messages.

Mind you, I'm a bit of a noob when it comes to this, so please tell me if you want me to issue some commands to see if the fix actually got applied etc.

some quick info about the machine:
$ uname -a
14-generic #48-Ubuntu SMP Fri Oct 16 14:04:26 UTC 2009 i686 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 9.10
Release: 9.10
Codename: karmic

$ ls -lh /var/log/ | grep -r 'M\|G'
total 20G
-rw-r----- 1 syslog adm 14G Nov 6 09:48 kern.log
-rw-r----- 1 syslog adm 4.9G Nov 6 09:48 messages
-rw-r----- 1 syslog adm 1.8G Nov 6 09:48 syslog

[Ariel Faigon (ariel.faigon)]

Sorry to not have any good news.

I just upgraded to rsyslog (proposed).

From /var/log/apt/term.log:
    Setting up rsyslog (4.2.0-2ubuntu5.1) ...
    rsyslog start/running, process 26509

The problem still exists. i.e. when load goes up, CPU gets hot and rsyslog starts spewing at a high rate, alternating between high temp and normal temp which makes the CPU-util problem even worse.

Brief excerpt from the log:
Nov 6 11:21:28 ze kernel: [497797.855109] CPU1: Temperature above threshold, cpu clock throttled (total events = 18517766)
Nov 6 11:21:28 ze kernel: [497797.856109] CPU5: Temperature/speed normal
Nov 6 11:21:28 ze kernel: [497797.856112] CPU1: Temperature/speed normal
...
Rate of printing under load is 2400 lines/second (this is a 8-core machine)
    $ grep '^Nov 6 11:21:28' /var/log/syslog | wc -l
   2400

The only difference I see is this regression in my logs:

Nov 6 09:06:59 ze kernel: imklog 4.2.0, log source = /var/run/rsyslog/kmsg started.
Nov 6 09:06:59 ze rsyslogd: rsyslogd's groupid changed to 102
Nov 6 09:06:59 ze rsyslogd: rsyslogd's userid changed to 101
Nov 6 09:06:59 ze rsyslogd-2039: Could no open output file '/dev/xconsole' [try http://www.rsyslog.com/e/2039 ]

(never seen the /dev/xconsole error before)
My system has no /dev/xconsole

Will gladly provide more data, just ask.
Thanks.

[Martin Pitt (pitti)]

Ah, it's not actually v-failed. As already explained, rsyslog can't auto-merge kernel messages. Reopening kernel task, since apparently the fix did not yet make it in.

[caseroj (caserojj)]

I have the same problem. I'm using ubuntu 9.10 on a fujitsu N5010 laptop with a P4 at 3.2 Ghz. Normal temperature swings generate thousands of messages in /var/log/messages about the CPU temperature....like so

Nov 8 15:48:10 bonsai kernel: [ 4428.945041] CPU0: Temperature/speed normal

The processes that seem to be consuming all the cpu resources are rsyslogd and dd. They are pegged at near 100% of cpu load and effectively seize the computer as I am not able to do anything else with it. If you ask me this is pretty severe.

Juan

[taka jedna (taka-jedna)]

Same here. Pentium 4 1,5GHz, some old HP motherboard. I'm laic in hardware, but I don't know it has any sensors at all.
The repeated line in var/log/syslog is:
Nov 9 12:19:10 Rencznie-Inkrustowany-Kartofel kernel: [ 2527.403083] CPU0: Temperature above threshold, cpu clock throttled (total events = 880863)
Nov 9 12:19:10 Rencznie-Inkrustowany-Kartofel kernel: [ 2527.404096] CPU0: Temperature/speed normal
The same in kern.log:
Nov 9 12:20:57 Rencznie-Inkrustowany-Kartofel kernel: [ 2634.262246] CPU0: Temperature above threshold, cpu clock throttled (total events = 951122)
Nov 9 12:20:57 Rencznie-Inkrustowany-Kartofel kernel: [ 2634.263447] CPU0: Temperature/speed normal
(500000 in two minutes?)
dd and rsyslogd are using all free CPU, so normal using of PC is almost impossible. Killing, deleting and changing priority don't work for too long. It's more common problem, when I searched help on #ubuntu few people complained about dd and rsyslogd, or about disappearing HDD space, just didn't came to diagnose.

[caseroj (caserojj)]

I get the same errors that taka jedna gets. The exact same messages about "Temperature above threshold, cpu clock throttled", etc. I mean thousands of these messages per minute. This machine has always run like this. I has always run hot and when it gets too hot the cpu is throttled down and the rpms of the internal cpu fan are ramped up. The machines originally came with Windows XP Home but I have even put windows XP Professional on it in the past with no problems. I tried disabling acpi by passing the switches "noapic nolapic acpi=off" on the grub kernel parameter line but it does not fix the problem. All that happens is that now hyperthreading (which is turned on in the bios) seems not to be recognized by the kernel. I have all the latest ubuntu 9.10 patches installed by the way. It is definitely dd and rsyslogd that are consuming cpu resources. If I tail the /var/log/messages file you can see a huge stream of these temperature cpu throttling messages flying past at a fantastic rate. I know this motherboard is supposed to work this way because it has software in the bios to shutdown the machine if the cpu temperature reaches dangerous levels. This happened once a few years ago when the copper heat exchange veins used to cool the cpu got clogged up with dust. Back then I opened the bottom of the case, cleaned out the veins and the machine started to work like normal again. Last night I checked the veins again and they were all clean and so far the machine has not shut down unexpectedly on me. All of which suggest that it is operating with normal temperature ranges.

Juan

[mobrien118 (mobrien118)]

I was getting these messages filling up my root hard drive very quickly.

I discovered that my heat sink was all clogged up with dust and cleaning it fixed the errors, but it shouldn't be doing this anyway. Eventually my heat sink is going to get gunked up again and the only way I'm going to know is when my computer crashes and I take time to dig into the error logs and find that my HD is full of this crap again.

Another issue is on 3 out of 5 computers I use don't update the CPU temperature in sensors anyway. 2 of them say that it is 40C all the time, and another stays at 20C. At first, I thought that this may be a fault in the motherboard as the 2 that stay at 40C are both cheap ECS mobos, from relatively the same time period, about 2-3 years ago, but one is an INTEL and the other AMD. The one that's stuck on 20C is another manufacturer, but even older.

BUT, since the lernel seemed to know that the chip was overheating, I guess it DID know what the temp was. I wonder if there is some connection between all of these "happenings".

Good luck!

[Raphael Campos (optgeek)]

Please excuse me if I'm wrong, but libsensors4 was mentioned before in this page.
I don't think this bug has any relationship with libsensors4 or 3. I tried purging libsensors3 and rebooting, but the problem persists.

Also, it seems that this bug is triggered only when the temperature goes up - for example, from a cold boot. When the temperature becomes stable, it stops flooding the logs. It starts flooding again if you run any proccess that constantly uses a considerable bit of the CPU. To reproduce this, you can 'tail -f /var/logs/syslog', wait for it to stop flooding if it's not stopped already and run glxgears in another instance. It will start flooding again.

Were anyone able to find at least a workaround for this?

[caseroj (caserojj)]

I agree with Raphael Campos. I do not see the symptoms from a cold boot. In my case if I do a find for example and search for a file in my entire file system the cpu temperature starts to go up rapidly and then this starts triggering the dd and rsyslogd load spikes. The problem is that it's a feedback loop. Once those two processes kick off it all goes to hell because they start to generate lots of disk writes which in turn causes the cpu to work harder which eventually leads to a higher cpu temperatures.

Juan

[caseroj (caserojj)]

Interesting. I applied all the updates from the ubuntu site just now....apparently there were some new ones today and now I cannot reproduce the problem on my Fujitsu N5010. In fact I am composing this message from that laptop after heaving searched my entire file system several times trying to see if I can trigger the events. I wonder if someone provided some sort of patch to solve it. I recommend everyone apply the latest updates from the ubuntu servers.

Juan

[caseroj (caserojj)]

Scratch that last comment.... :-). It took a while for the machine to heat up (from a cold boot) but I can now see the cpu load spike at 100% for dd and rsyslogd. It looks like the problem persists.

Juan

[caseroj (caserojj)]

Here you go. This was triggered when I went to www.adobe.com using firefox and the flash videos started playing on their site...

Nov 10 15:40:03 bonsai kernel: [ 1560.857817] CPU0: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.857822] CPU1: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.860614] CPU0: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.860610] CPU1: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.861255] CPU0: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.861262] CPU1: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.863378] CPU0: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.863387] CPU1: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.866821] CPU0: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.866831] CPU1: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.869609] CPU0: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.869618] CPU1: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.871730] CPU0: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.871739] CPU1: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.875830] CPU0: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.875838] CPU1: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.878620] CPU1: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.878620] CPU0: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.882067] CPU1: Temperature/speed normal
Nov 10 15:40:03 bonsai kernel: [ 1560.882071] CPU0: Temperature/speed normal

op - 15:41:25 up 27 min, 3 users, load average: 1.79, 1.05, 0.57
Tasks: 164 total, 2 running, 162 sleeping, 0 stopped, 0 zombie
Cpu(s): 29.3%us, 62.3%sy, 0.0%ni, 8.2%id, 0.0%wa, 0.2%hi, 0.0%si, 0.0%st
Mem: 2059724k total, 728384k used, 1331340k free, 472k buffers
Swap: 4080468k total, 0k used, 4080468k free, 341780k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  444 root 20 0 1848 544 448 R 100 0.0 3:58.81 dd
  480 syslog 20 0 35744 1860 984 S 68 0.1 2:47.87 rsyslogd
 1039 root 20 0 227m 26m 8744 S 6 1.3 1:07.43 Xorg
 1588 juan 20 0 38564 13m 9676 S 4 0.7 1:04.10 gnome-terminal
 1686 juan 20 0 305m 81m 28m S 4 4.1 1:16.42 firefox
 1887 juan 20 0 2468 1188 884 R 1 0.1 0:00.13 top
   22 root 15 -5 0 0 0 S 0 0.0 0:00.32 ata/0
 1492 juan 20 0 102m 12m 9860 S 0 0.6 0:04.90 metacity
    1 root 20 0 2528 1488 1120 S 0 0.1 0:01.63 init
    2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
    3 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0
    4 root 15 -5 0 0 0 S 0 0.0 0:00.15 ksoftirqd/0
    5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
    6 root RT -5 0 0...

[Leann Ogasawara (leannogasawara)]

The patch for the kernel side of things to quit spamming the logs should be coming in the next 2.6.31.6 upstream stable patch set that we'll pull into Karmic . I'll try to put together a test kernel with this patch for people to test. Thanks.

ogasawara@emiko:~/linux-2.6$ git show b417c9fd8690637f0c91479435ab3e2bf450c038
commit b417c9fd8690637f0c91479435ab3e2bf450c038
Author: Ingo Molnar <email address hidden>
Date: Tue Sep 22 15:50:24 2009 +0200

    x86: mce: Fix thermal throttling message storm

[X Y (throwaway-xy+ubuntu)]

Just want to note that I am experiencing this also, to near-disastrous results.

Symptoms: log spew seen in 'dmesg' quickly filling up logs in '/var/log';
top shows a 'dd' process pegging a CPU, running as root, which I believe
is doing the writing. This has caused my system to lock up hard, and also
take a couple of tries to reboot (that's pretty bad). No problems with this
ever in the past.

# dmesg | tail
[ 2160.946120] CPU0: Temperature above threshold, cpu clock throttled (total events = 1058701)
[ 2160.946774] CPU0: Temperature/speed normal
[ 2160.947276] CPU0: Temperature above threshold, cpu clock throttled (total events = 1058702)
[ 2160.948600] CPU0: Temperature/speed normal
[ 2160.950248] CPU0: Temperature above threshold, cpu clock throttled (total events = 1058704)
[ 2160.950901] CPU0: Temperature/speed normal
[ 2160.951451] CPU0: Temperature above threshold, cpu clock throttled (total events = 1058705)
[ 2160.952109] CPU0: Temperature/speed normal
[ 2160.952716] CPU0: Temperature above threshold, cpu clock throttled (total events = 1058706)
[ 2160.953369] CPU0: Temperature/speed normal

How did this garbage get into the kernel in the first place?

[Leann Ogasawara (leannogasawara)]

Test kernels (amd64 and i386) are available at the following. Please let us know your results. Thanks.

http://people.canonical.com/~ogasawara/lp453444/

[Raphael Campos (optgeek)]

Thanks, Ogasawara. I'll be testing it in few minutes.

Those who for some reason do not want to install the test kernel above can just disable syslog from running on system startup. You can do this by commenting the uncommented lines in /etc/init/syslog.conf.
Note that it will prevent syslog from flooding the log files and dd from consuming the CPU, but valuable messages could not be logged. This is in no way a fix, just a workaround to wait for the next official kernel patch, and you should uncomment the lines you commented when the patch is out.

[Denis (denis-chalon)]

Thanks Ogasawara,

I have the same problem than others in the list with my Quad core Q6600 processor.

I tested your kernel patch for i386 given above in #39 post. It refuses to launch GDM to login on. The login screen is the console one. And the display of login screen in text mode was blinking...

More information on what happens with kernel Linux yoda 2.6.31-14-generic #48-Ubuntu SMP Fri Oct 16 14:04:26 UTC 2009 i686 GNU/Linux installed by Karmic:
- Currently my CPU is not too hot.
- When I tail my /var/log/message, I got a lot of CPU0: Temperature/speed normal at the same timestamp
For example:
At Nov 11 15:18:55, I got 65 identical messages
At Nov 11 15:19:25, I got 19 identical messages
...

[caseroj (caserojj)]

Ogasawara -

I will your test kernels a try when I get home tonight and post the results here.

Cheers,
Juan

[Peter Wagner (o-petris)]

my nomination for jaunty was an error, soory

[perspectoff (perspectoff)]

Again, I think the core problem the problem is not intrinsically with rsyslogd.

While I agree that there ought to be a flooding failsafe in rsyslogd, the base problem here is the CPU temperature sensors algorithm in the kernel and the reporting of errant messages.

I earlier thought this was due to lm-sensors/libsensors3/libsensors4, but those aren't loaded at the initial install.

HAL now does those functions, so it seems to me that a module such as hal-addon-acpi is the culprit for the aberrant message stream.

The root problem is the enforced checking of motherboard/CPU sensors. Because there are is many motherboards/CPUs that do not have sensor drivers in Linux for their fan speeds and temperature sensors, the sensor checking algorithm must be optional.

It is not, in the current kernel, and instead of a single message that sensors are unable to be polled (due to lack of drivers), the kernel module instead returns repeated "temperature out of range" messages. This is the bug.

Jaunty did not rely on HAL in this manner (and Jaunty did not mandate CPU sensor monitoring).

This serious bug is giving Linux (and by extension Ubuntu) a big black eye with the Karmic release. This should be upgraded to urgent.

[bugpics (nicolas-lescuyer)]

When can we hope a fix ?

use karmic with this bug is impossible !

Nico

[bugpics (nicolas-lescuyer)]

I find a solution with my laptop that have this problem and a very long boot time (about 15 minutes) : sony vaio pcg-grt716s

I edit /boot/grub/grub.cfg and i add "noapic nomce" options to line :
linux /boot/vmlinuz-2.6.31-15-generic root=UUID=xxxxxxxx ro splash quiet noapic nomce

I find that "noapic nomce" are boot options linked to bios bugs. I can't explain why it works but if this tip can help you.

Best regards
Nico

[mitsos (desmitse)]

After my last rebooting the problem seemed to be taken out ...
That means that neither dd or rsylogd 'destroy' the system .
In the /var/log directory I found that kern.log , messages , syslog files appear double :

ls -s command lists :
0 kern.log 4 messages 4 syslog 24 syslog.2.gz
2387572 kern.log.1 591716 messages.1 241604 syslog.1 168364 syslog.3.gz

Any idea why have been created those files ?
Now the new logs will be placed in the new log files ?

[robytrevi (robytrevi)]

I've got a friend with the same bug on Ubuntu karmic (toshiba satellite M167)
CPU0: Temperature above threshold, cpu clock throttled (total events = 4220841)
Either with kernel 2.6.31.14 or 2.6.32.4

[MarkJBobak (mark-bobak)]

Hi all,

This bug has been a real pain in my backside. I think I have a reasonable, short-term workaround. I'm *not* suggesting this is a permanent fix for anyone, nor should this be applied to any source trees anywhere.

This is strictly for someone who, like me, is suffering because of this, and wants a slightly better solution than simply killing off logging systemwide till a proper patch is available.

Here's my hack:
--- /etc/init/rsyslog-kmsg.conf.orig 2009-11-17 01:40:07.000000000 -0500
+++ /etc/init/rsyslog-kmsg.conf 2009-11-17 01:42:36.000000000 -0500
@@ -18,7 +18,7 @@
     chown syslog:syslog /var/run/rsyslog/kmsg
 end script

-exec dd bs=1 if=/proc/kmsg of=/var/run/rsyslog/kmsg
+exec dd bs=1 if=`grep -v "CPU[0-7]: Temperature" /proc/kmsg` of=/var/run/rsyslog/kmsg

 post-stop script
     rm /var/run/rsyslog/kmsg

Or, for people who don't like dealing w/ patch files, go to /etc/init, make a copy of rsyslog-kmsg.conf. Then, edit rsyslog-kmsg.conf, find the line near the end of the file that looks like:
dd bs=1 if=/proc/kmsg of=/var/run/rsyslog/kmsg

and replace it with:
exec dd bs=1 if=`grep -v "CPU[0-7]: Temperature" /proc/kmsg` of=/var/run/rsyslog/kmsg

So, all this does is filter out *all* warnings referring to CPU temperature fluctuations (for boxes up to 8 CPUs). Don't look at me if you don't see the warnings and end up flaming out your CPU. :-)

Hope this helps someone until such time that a proper fix is available.

-Mark

[bmarsh (bmarsh-bmarsh)]

I have been bothered by this problem repeatedly so I finally took the following steps:

ln -s /dev/null syslog

ln -s /dev/null kern.log

That should take care of most of the problem. Why is it taking so long to fix this and why did it happen in the first place?

ARRRGGGGG!!

[Matt (slower-shamble)]

I was hopeful that MarkJBobak's suggestion to filter out CPU Temperature messages before passing them to rsyslog would work, but it actually made my system unbootable. On boot, my computer would halt after mounting and checking the disk partitions, even in recovery mode. I found the following related messages in /var/log/syslog:

Nov 19 16:46:29 arethusa kernel: imklog: Cannot open proc file system, 4.
Nov 19 16:46:29 arethusa rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="847" x-info="http://www.rsyslog.com"] (re)start
Nov 19 16:46:29 arethusa kernel: imklog: Cannot open proc file system, 4.
Nov 19 16:46:29 arethusa rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="1691" x-info="http://www.rsyslog.com"] (re)start
Nov 19 16:46:29 arethusa rsyslogd: rsyslogd's groupid changed to 102
Nov 20 06:50:54 arethusa kernel: imklog: Cannot open proc file system, 4.
Nov 20 06:55:18 arethusa kernel: imklog: Cannot open proc file system, 4.

Once I changed the /etc/init/rsyslog-kmsg.conf file back to its original state, I could boot normally again. Just a word of caution in case this bites someone else.

[ubername (ubername)]

I had the same symptoms as Matt, I haven't had a chance to check the syslog messages yet to see if it is exactly the same.

To cheekily repeat my question on the forum thread for this bug

it says that the milestone for this is Ubuntu karmic-updates which has an expected date of 2011-04-09!
Does this mean we have to wait until then for the fix?

[Raphael Campos (optgeek)]

Ubername, I don't belive it will take that long, as Ogasawara already pre-released the required patched kernel images that fixes this bug.

Those who are still trying to workaround this bug (and flooding the bug report, I must say), should try installing the kernel that Leann uploaded. I've been using it since he kindly released it, and it's working flawlessly for me. I'm looking forward for the official release.

[ubername (ubername)]

Installed the kernel Leann uploaded and it's working fine. Thanks

[caseroj (caserojj)]

Ogasawara -

I finally got around to installing the patched kernel and headers you provided and they appear to be working. My Fujitsu N5010 P4 laptop is no longer being bogged down by excessive cpu temperature messages to /var/log/messages. I tried taxing the system by running a full file system scan several times and never once saw dd or rsyslogd show up in the top list of cpu resource consumers. In fact I can feel the fan of my cpu throttling up and down as it normally should be doing. So I am confident your patch fixed the problem.

Thanks,
Juan

[Ariel Faigon (ariel.faigon)]

FWIW:

Tried to install the test kernels from Leann, and it looks like they are not playing nice with the nvidia drivers
(I know, they are not free, but they did work with all officials kernels so far)

If I'm doing anything wrong, please let me know.

Here's my session:

$ sudo dpkg -i linux-image-2.6.31-16-generic_2.6.31-16.51~lp453444_amd64.deb
(Reading database ... 372866 files and directories currently installed.)
Preparing to replace linux-image-2.6.31-16-generic 2.6.31-16.51~lp453444 (using linux-image-2.6.31-16-generic_2.6.31-16.51~lp453444_amd64.deb) ...
Done.
Unpacking replacement linux-image-2.6.31-16-generic ...
Running postrm hook script /usr/sbin/update-grub.
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.31-16-generic
Found initrd image: /boot/initrd.img-2.6.31-16-generic
Found linux image: /boot/vmlinuz-2.6.31-15-generic
Found initrd image: /boot/initrd.img-2.6.31-15-generic
Found linux image: /boot/vmlinuz-2.6.31-14-generic
Found initrd image: /boot/initrd.img-2.6.31-14-generic
Found linux image: /boot/vmlinuz-2.6.28-16-generic
Found initrd image: /boot/initrd.img-2.6.28-16-generic
Found linux image: /boot/vmlinuz-2.6.28-15-generic
Found initrd image: /boot/initrd.img-2.6.28-15-generic
Found linux image: /boot/vmlinuz-2.6.28-14-generic
Found initrd image: /boot/initrd.img-2.6.28-14-generic
Found linux image: /boot/vmlinuz-2.6.28-13-generic
Found initrd image: /boot/initrd.img-2.6.28-13-generic
Found linux image: /boot/vmlinuz-2.6.28-11-generic
Found initrd image: /boot/initrd.img-2.6.28-11-generic
Found memtest86+ image: /boot/memtest86+.bin
Found Ubuntu 9.04 (9.04) on /dev/sdb1
done
Setting up linux-image-2.6.31-16-generic (2.6.31-16.51~lp453444) ...
Running depmod.
update-initramfs: Generating /boot/initrd.img-2.6.31-16-generic
Not updating initrd symbolic links since we are being updated/reinstalled
(2.6.31-16.51~lp453444 was configured last, according to dpkg)
Not updating image symbolic links since we are being updated/reinstalled
(2.6.31-16.51~lp453444 was configured last, according to dpkg)
Running postinst hook script /usr/sbin/update-grub.
Generating grub.cfg ...
Found linux im...

[Martin Baláž (embecka)]

I wrote a file /etc/rsyslog.d/10-temperature.conf with the following content:

:msg,contains,"Temperature/speed normal" ~
:msg,contains,"Temperature above threshold" ~

All messages containing "Temperature/speed normal" and "Temperature above threshold" are discarded.
I restarted rsyslog and it looks like the problem is solved.

I hope this will help until the problem will be fixed in the kernel.

[Raphael Campos (optgeek)]

Ariel, I suggest you to put all (IIRC) three packages in the same folder and
run # dpkg -i *
The nVidia module may be missing the kernel headers.

[Ariel Faigon (ariel.faigon)]

Thanks so much Raphael,

My installation issue is gone after following your helpful suggestion.

Sorry for the false alarm.

I've not yet verified that this new kernel has solved the logging problem for me,
but I can confirm at this point that there's no issue with 64-bit and/or nvidia drivers during installation.

Thanks again, Raphael and Leann!

[Roman Yepishev (rye)]

The issue with the kernel side is definitely fixed by:
3967684006f30c253bc6d4a6604d1bad4a7fc672 - cleanup commit first and
b417c9fd8690637f0c91479435ab3e2bf450c038 - for the actual storm.

Tested on Intel(R) Celeron(R) CPU 2.80GHz on 2.6.31-14-generic.

Re: kernel posted earlier - looks like I reinvented the wheel...

[Matt (slower-shamble)]

Thanks embecka, your conf file fix worked for me.

The messages stopped cold after writing the new conf file and restarting rsyslog (they had been streaming in at the usual alarming rate just prior to the fix). Just to be sure, I ran a disk scan of a 40 GB partition that took about 5 minutes. So far, so good.

[Paillomams (aymeric-pallottini)]

I had this issue when transfering a Sony Vaio laptop with a P4 processor from XP to Ubuntu. Starting Ubuntu with "noapic option solve the problem but I don't thing log file should be allowed to grow to more than 150Mo.
It is of no use and it is possible to lock the computer. Size limit should be imposed via logrotate or a message should be displayed to the user before it is too late to do anything about it and not beeing able to boot the PC.

[leo (leo-china-leo)]

Thanks embecka, your conf file fix worked for me too.

everything now is ok!

[Ariel Faigon (ariel.faigon)]

Confirming that the new kernel totally solves the problem for me. I'm using the 64-bit one.

I see an occasional "CPU1: Temperature above threshold, cpu clock throttled" and then "CPU1: Temperature/speed normal" but no flooding of the logs occurs. Instead the number of occurrences appears as in "(total events = 254123)" on one line only.

Haven't seen any regressions in the past 2 days.

"Ship it" :)

[Félix (felix-rauch)]

Bug still present for me on ubuntu server with kernel 2.6.31-15-generic-pae (package linux-image-2.6.31-15-generic-pae, version 2.6.31-15.50).

If the CPU is under high load and over heating message is repeated very often (100 times/second).
The trick from embecka to discard these messages works fine but is not satisfying.

[Rob Traders (rob-traderspit)]

Looks like, that primary P4 systems have the problem
(I've not registered this issue before karmic and the syslog replacement called rsyslog).
My P4 3.0 GHz is running since the first day with a temperature 70...75°C, its not a fan issue or cooling problem.. its designed by Intel.

I've investigated now ~5 days with the problem with no acceptable solution (at least for me).
The Bug is partial fixed with the additional rsyslog temperature conf file to prevent the massive flood of various logfiles and filling up /var with messages like that

[11400.257971] CPU0: Temperature above threshold, cpu clock throttled (total events = 1972881)
[11400.258565] CPU0: Temperature/speed normal

But if you check the process you'll see a neverending stongly cpu consuming dd process (invoked by rsyslogd)
root 4443 1 11 21:25 ? 00:00:01 dd bs=1 if=/proc/kmsg of=/var/run/rsyslog/kmsg

The only way to have a little bit silence is, to stop the rsyslog proces until a fix is available
Don't forget, to stop again after reboot.
# sudo service rsyslog stop

[Ariel Faigon (ariel.faigon)]

To clarify: when I said the new kernel solves the problem for me I meant the kernel that was pre-released by Leann (see comment #39) - not the one in the repositories.

My kernel is:

2.6.31-16-generic #51~lp453444 SMP Wed Nov 11 07:45:32 UTC 2009 x86_64 GNU/Linux

And it definitely solves the problem.

[maxstirner (philipp-d)]

The fix above in comment #57 helped right away, no reboot required. In my case, the syslogd process itself was actually stopping the temperature from coming down in this fanless unit..

[Fabio Marzocca (thesaltydog)]

Is this fixed in Karmic?

[maxstirner (philipp-d)]

No it isnt and its really annoying.. It also tends to just generally flood the screen with those messages during boot.

[Fabio Marzocca (thesaltydog)]

In the heading of this page I see "Fix Committed"... when should it become "Released"?

[avcascade (andrew-villeneuves)]

I think this problem has affected me, bigtime. I recently tried to hibernate running Karmic, and got this message, over and over again. Here is an example:

Dec 1 08:02:42 compname kernel: [35031.545470] CPU0: Temperature/speed normal
Dec 1 08:02:42 compname kernel: [35031.545638] CPU0: Temperature above threshold, cpu clock throttled (total events = 236944)

These messages just kept running down the screen, and Ubuntu would not shut down. I finally killed power manually.

Not long after I noticed that something was eating up almost all my disk space. I investigated, and - whaddaya know - the log folder was taking up half the space on the hard drive! Pretty soon Ubuntu reported 0 bytes of free space. Whole machine seized up, most Ubuntu functions refused to work due to lack of disk space. I restarted in the hope that some log files might be compressed or deleted automatically; no such luck. I booted into the Live CD and sudo'ed into Nautilus, going into /var/logs/ then picked the biggest log file I could find, which was syslog, and moved it off the hard disk. I then rebooted and Ubuntu came back up okay. syslog was of course automatically recreated. But now it has a filesize of 1.25 MB instead of 1.0GB.

Now trying to figure out how to configure logrotate so i can stop /var/logs from totally eating up my disk space again. Wish logrotate had a GUI....

[caseroj (caserojj)]

I installed the kernels from comment #39 by Osagawara and it does solve the problem. However, when Karmic installs updated kernels from the repository the problem resurfaces. Any idea if and when these fixes will make it into the mainstream kernel? I would have thought this would have already made it into the mainline updates for karmic. This bug makes the machine totally useless. I even switched to OpenSuSE 11.2 for a shortwhile because of it.

Thanks,
Juan

[lcn_mustard (lcn-mustard)]

I have the same problem. The log files are huge. I do not receive messages on the cpu but normal messages but that is repeated every second and increase the use of just the cpu and hd space.
 CRON[2730]: pam_unix(cron:session): session opened for user brain by (uid=0)

My / etc / syslog.conf this blank.
How do I disable the logs.
How to fix this bug.
What is better to wait for the bug fix or return to the version I used before: hardy heron?

Does following the page below resolve?
https://wiki.kubuntu.org/DebuggingProgramCrash

[lcn_mustard (lcn-mustard)]

I could not put the other because I had to log excui them to reactivate the system

[lcn_mustard (lcn-mustard)]

I could not put the other because I had to log excui them to reactivate the system

[Raphael Campos (optgeek)]

I'm sad to report that the latest Ubuntu's kernel update broke Ogasawara's package.
We've just ran the update manager in some machines, and all of them asked to update the kernel. After that, many modules stopped being loaded, including snd-hda-intel.

All these machines used a custom Ubuntu ISO (remastered by me) wich had all packages related to kernel removed (linux-image*, linux-generic*...), then Leann Ogasawara's kernel was installed. Unfortunately, as it kept the same package name, update manager tried to update it, therefore breaking the kernel package installation.

Everyone using the kernel published in comment #39 must be aware that updating the kernel can lead to a broken kernel package. I suggest you manually remove (or purge) the installed ~lp453444 kernel packages before updating the system.

You can see what are the related packages installed by running the following command in a terminal...
$ dpkg -l | grep lp453444
...and purge them using aptitude.

[Martin Baláž (embecka)]

for all unhappy people who reinstalled their loved operating system only because of this bug:

there exists a very simple temporal solution!!!

1) stop rsyslog service (#service rsyslog stop)
2) (backup and) delete all huge files in /var/log
3) copy the file 10-temperature.conf attached in the post #57
4) start rsyslog service (#service rsyslog start)

what is going on:

The effect of this bug is so serious that there is no time to wait for a fix in the kernel. But it is easy to configure the syslog to discard all unwanted messages, i.e. messages containing "Temperature/speed normal" and "Temperature above threshold". This is what the file 10-temperature.conf is all about. When the fix will be released, just delete this file and reload the service rsyslog. I am using this fix few weeks and I haven't noticed another unwanted messages.

[caseroj (caserojj)]

Does anyone know when Osagawaras' fix will make into into the mainline kernels for ubuntu 9.10? It seems like the answer should be straightforward but no one seems to want to comment on it.

Juan

[Martin Pitt (pitti)]

Accepted linux into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[ubername (ubername)]

Seems fixed in 2.6.31-17

[dlgehrt (dlg-inanity)]

The I noticed effect of this bug this morning for the first time and after screwing up the bug reporting process a few times I note this bug is apparently fixed in -17, but my version is -16, and I can see no upgrade to -17 in the update manager, nor do I see any version beyond -16 in the synaptics list.

What us the solution?

dlg

[Ariel Faigon (ariel.faigon)]

-17 is in proposed, not in the official repo yet.

To install it you would need to add karmic-proposed to your list of supported repositories.

You can do it either via a GUI (e.g. synaptic) or by directly adding the following line to your /etc/apt/sources.list

     deb http://us.archive.ubuntu.com/ubuntu/ karmic-proposed restricted main multiverse universe

Hope this helps.

[Launchpad Janitor (janitor)]

This bug was fixed in the package rsyslog - 4.2.0-2ubuntu5.1

---------------
rsyslog (4.2.0-2ubuntu5.1) karmic-proposed; urgency=low

  * debian/rsyslog.conf:
    - enable $RepeatedMsgReduction to avoid bloating the syslog
      file (LP: #453444)
 -- Michael Vogt <email address hidden> Fri, 23 Oct 2009 17:28:10 +0200

[Martin Pitt (pitti)]

Copied to lucid, should be tested there a little first.

[Martin Pitt (pitti)]

rsyslog message repeat suppression works fine here:

Dec 16 13:03:30 tick rtkit-daemon[1644]: last message repeated 3 times

[Launchpad Janitor (janitor)]

This bug was fixed in the package rsyslog - 4.2.0-2ubuntu5.1

---------------
rsyslog (4.2.0-2ubuntu5.1) karmic-proposed; urgency=low

  * debian/rsyslog.conf:
    - enable $RepeatedMsgReduction to avoid bloating the syslog
      file (LP: #453444)
 -- Michael Vogt <email address hidden> Fri, 23 Oct 2009 17:28:10 +0200

[Ryan Thompson (rct86)]

Maybe this isn't the place to ask, but is there a bug report or fix for the underlying CPU throttling problem that is the source of all these log messages? I can't use Karmic because it throttles my CPU to the point that of complete unusability.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 2.6.31-17.54

---------------
linux (2.6.31-17.54) karmic-proposed; urgency=low

  [ John Johansen ]

  * SAUCE: AppArmor: Fix oops after profile removal
    - LP: #475619
  * SAUCE: AppArmor: Fix Oops when in apparmor_bprm_set_creds
    - LP: #437258
  * SAUCE: AppArmor: Fix cap audit_caching preemption disabling
    - LP: #479102
  * SAUCE: AppArmor: Fix refcounting bug causing leak of creds
    - LP: #479115
  * SAUCE: AppArmor: Fix oops there is no tracer and doing unsafe
    transition.
    - LP: #480112

  [ Leann Ogasawara ]

  * Revert "[Upstream] (drop after 2.6.31) usb-storage: Workaround devices
    with bogus sense size"
    - LP: #461556
  * Revert "[Upstream] (drop after 2.6.31) Input: synaptics - add another
    Protege M300 to rate blacklist"
    - LP: #480144

  [ Tim Gardner ]

  * [Config] udeb: Add squashfs to fs-core-modules
    - LP: #352615

  [ Upstream Kernel Changes ]

  * Revert "e1000e: swap max hw supported frame size between 82574 and
    82583"
    - LP: #461556
  * Revert "drm/i915: Fix FDI M/N setting according with correct color
    depth"
    - LP: #480144
  * Revert "agp/intel: Add B43 chipset support"
    - LP: #480144
  * Revert "drm/i915: add B43 chipset support"
    - LP: #480144
  * Revert "ACPI: Attach the ACPI device to the ACPI handle as early as
    possible"
    - LP: #327499, #480144
  * SCSI: Retry ADD_TO_MLQUEUE return value for EH commands
    - LP: #461556
  * SCSI: Fix protection scsi_data_buffer leak
    - LP: #461556
  * SCSI: sg: Free data buffers after calling blk_rq_unmap_user
    - LP: #461556
  * ARM: pxa: workaround errata #37 by not using half turbo switching
    - LP: #461556
  * tracing/filters: Fix memory leak when setting a filter
    - LP: #461556
  * x86/paravirt: Use normal calling sequences for irq enable/disable
    - LP: #461556
  * USB: ftdi_sio: remove tty->low_latency
    - LP: #461556
  * USB: ftdi_sio: remove unused rx_byte counter
    - LP: #461556
  * USB: ftdi_sio: clean up read completion handler
    - LP: #461556
  * USB: ftdi_sio: re-implement read processing
    - LP: #461556
  * USB: pl2303: fix error characters not being reported to ldisc
    - LP: #461556
  * USB: digi_acceleport: Fix broken unthrottle.
    - LP: #461556
  * USB: serial: don't call release without attach
    - LP: #461556
  * USB: option: Toshiba G450 device id
    - LP: #461556
  * USB: ipaq: fix oops when device is plugged in
    - LP: #461556
  * USB: cp210x: Add support for the DW700 UART
    - LP: #461556
  * USB: Fix throttling in generic usbserial driver
    - LP: #461556
  * USB: storage: When a device returns no sense data, call it a Hardware
    Error
    - LP: #400652, #461556
  * arm, cris, mips, sparc, powerpc, um, xtensa: fix build with bash 4.0
    - LP: #461556
  * intel-iommu: Cope with broken HP DC7900 BIOS
    - LP: #461556
  * futex: Detect mismatched requeue targets
    - LP: #461556
  * futex: Fix wakeup race by setting TASK_INTERRUPTIBLE before queue_me()
    - LP: #461556
  * tpm-fixup-pcrs-sysfs-file-update
    - LP: #461556
  * TPM: fix pcrread
    - LP: #461556
  * Bluetooth: Disconnect HIDRAW devices on disconnect
    - LP...

[Brad Figg (brad-figg)]

This bug was filed against a series that is no longer supported and so is being marked as Won't Fix. If this issue still exists in a supported series, please file a new bug.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.