ACPI + kworker high interrupts.

[Pieter]

Installed new kernel on ubuntu 10.10
$ uname -a
Linux pieters-laptop 2.6.37-020637-generic #201101050908 SMP Wed Jan 5 09:09:44 UTC 2011 x86_64 GNU/Linux

Now in top, kworker using 1cpu all the time
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    4 root 20 0 0 0 0 R 91 0.0 4:37.71 kworker/0:0

Powertop
Top causes for wakeups:
  80.7% (2486.2) [acpi] <interrupt>

[Pieter (diepes)]

Tried the latest kernel i could find, and the problem is gone.

$ uname -a
Linux pieters-laptop 2.6.38-020638rc4-generic #201102081010 SMP Tue Feb 8 10:12:40 UTC 2011 x86_64 GNU/Linux

[Pieter (diepes)]

Just upgraded to Natty-Beta2.
$ uname -a
Linux pieters-laptop 2.6.38-8-generic #41-Ubuntu SMP Tue Apr 5 19:30:49 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

From top, kworker is keeping one cpu busy.

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22580 root 20 0 0 0 0 R 88 0.0 1:21.93 kworker/0:1

[Pieter (diepes)]

From powertop

Top causes for wakeups:
  81.7% (5326.6) [acpi] <interrupt>

[Pieter (diepes)]

Here is one more interesting fact.

After reboot, ACPI interrupts is low.

When i close the laptop screen and the laptop sleeps and i wake it up again, the ACPI interrupts shoot up to 5k+/sec and one of the kworker threads uses close to 100% of one of the cpu's.

$ uname -a
Linux pieters-laptop 2.6.38-8-generic #41-Ubuntu SMP Tue Apr 5 19:30:49 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

[Chandan (compcode)]

The exact issue is also with my Ubuntu (Natty)

uname -a
Linux C-UBUNTU 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:50 UTC 2011 i686 i686 i386 GNU/Linux

[rod_h (rod-h)]

always present
uname -a
Linux huguets-EasyNote-SB85 2.6.38-9-generic #43-Ubuntu SMP Thu Apr 28 15:25:15 UTC 2011 i686 i686 i386 GNU/Linux

[Andre Rossouw (rossouwap)]

After resume from suspend high cpu usage. Possibly related to:
https://bugzilla.kernel.org/show_bug.cgi?id=29722

powertop:
Top causes for wakeups:
  92.4% (5531.1) [acpi] <interrupt>
   1.5% ( 90.5) [kernel scheduler] Load balancing tick

top:
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 4593 root 20 0 0 0 0 R 87 0.0 1:03.12 kworker/0:1

uname -a:
Linux pe-andrer-01 2.6.38-8-server #42-Ubuntu SMP Mon Apr 11 03:49:04 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

[Roland (roland-breedveld)]

The bug is not present in 2.6.38-7-generic #39-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux

The new kernel 2.6.38-8 is trouble.

so booting the old one, solved it for now for me.

[rssaddict (simoneau-louis)]

I have this problem. I tried upgrading the kernel to 2.6.39-0 and that doesn't fix it. ~30% CPU usage by the kworker process at all times.

[Jorge Eduardo (jorge-birck)]

Same here.

[Alexander Thole (a.t)]

I have this problem on one machine, too.

That one that have it is an Lenovo Y60p with an Intel i7-2630QM and an ATI Readeon 5000 Series grafics.

That one that doesn't have it is an self build PC with an AMD Phenom II 1090T and an NVIDIA Geforce GTS 450.

[rssaddict (simoneau-louis)]

Given that the constant CPU usage lowers battery life significantly, this pretty much stops me from being able to use Ubuntu on my laptop. Anything I can do to help debug?

[IKT (ikt)]

Hi rssaddict, looking at the bug report and patch this should not be affecting you as the patch was rolled into an update to the kernel.

Just to confirm, do you have all the latest updates installed for Natty?

[kaphar (captain-harry)]

On 07/18/2011 01:25 PM, IKT wrote:
> Hi rssaddict, looking at the bug report and patch this should not be
> affecting you as the patch was rolled into an update to the kernel.
>
> Just to confirm, do you have all the latest updates installed for Natty?
>
Well, I guess it depends on how many of these things have to be running.
I am up to date with the patches as far as I know. I update daily.

Thanks
Harry

[Taylor Heffernan (taylor-heffernan)]

I'm running 2.6.38-10 from the official release channel and I still have problems with kworker threads eating cpu. It happens for the first 5 minutes after a cold boot.

[Krisztian Poos (rocky-poos)]

Hello everyone, I also have the issue on a Thinkpad T500. Rolling back to 2.6.35 did not solved it, but on 2.6.34 I don't have the issue. upgrading to 2.6.39-02063903 also did not solve the issue. however on my laptop it is quite better than the with the default kernel. The system was unusable with the default one

I would be glad to provide any logs and information you need for resolving the issue.
By the way I have a bug uploaded also about the same topic: it can be related to this I guess: Bug #772704

I may think that it may somehow related also to the wifi card/driver, as usually during these periods when kworker eats up my cpu (it uses more than 50% of the cputime) the system often dropped from the wifi access point as well.

If anyone has a solution, it would be great to have, as this is a really annoying issue, and makes working impossible on the system.

Thanks and regards,
RockY

[kaphar (captain-harry)]

On 7/26/2011 12:53 PM, Krisztian Poos wrote:
> Hello everyone, I also have the issue on a Thinkpad T500. Rolling back
> to 2.6.35 did not solved it, but on 2.6.34 I don't have the issue.
> upgrading to 2.6.39-02063903 also did not solve the issue. however on my
> laptop it is quite better than the with the default kernel. The system
> was unusable with the default one
>
> I would be glad to provide any logs and information you need for resolving the issue.
> By the way I have a bug uploaded also about the same topic: it can be related to this I guess: Bug #772704
>
> I may think that it may somehow related also to the wifi card/driver, as
> usually during these periods when kworker eats up my cpu (it uses more
> than 50% of the cputime) the system often dropped from the wifi access
> point as well.
>
> If anyone has a solution, it would be great to have, as this is a really
> annoying issue, and makes working impossible on the system.
>
> Thanks and regards,
> RockY
>
I had to drop back to the thing that is unmentionable to keep going. I
only have the one Linux system and it is down because of this bug.
Would really like a solution.
Harry

[ultimatebuster (ultimatebusta)]

2.6.39.4 doesn't work as well.

Mine is even worse:

99.5% (36133.4) [acpi] <interrupt>
0.4% (137.4) [kernel scheduler] Load balancing tick

It didn't happen before, seem to have happened after upgrading to fglrx 11.7... but even with roll back, it doesn't work.

[ultimatebuster (ultimatebusta)]

I don't see anything about kworker though in my top... though

Does anyone know why this is happening and, most importantly, how to work around it?

[Arkadiy Kulev (eth-ethaniel)]

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26912 root 20 0 0 0 0 S 16 0.0 2:36.55 kworker/3:2
  428 root 20 0 0 0 0 S 11 0.0 0:06.37 kworker/3:0

Same here, I have 3-4 processes, each eating around 15% CPU every 5 seconds.

Using kernel 3.0.0-7

[ultimatebuster (ultimatebusta)]

Found something in dmesg:

[Firmware Bug]: ACPI(PEGP) defines _DOD but not _DOS

Does this relate?

[ultimatebuster (ultimatebusta)]

Actually, here's my full dmesg.

[gonemusic (gonemusic)]

Also have this problem on a samsung r530. Did compile the 2.6.39.4 kernel, but the problem still is there. Interupts every minute or so, ubuntu 11.04 is unusable this way. Before that I did run a 32 bit version and didnt suffer this problem. Maybe it is a 64 bit problem?

[Tim Cole (tcole)]

I'm having this problem (after suspend and resume) on a Toshiba Satellite L305, with kernel 2.6.38-10-generic.

[Krisztian Poos (rocky-poos)]

Is there any solution so far? I still have the issue... and it is really annoying... :(

[kaphar (captain-harry)]

On 09/27/2011 08:51 AM, Krisztian Poos wrote:
> Is there any solution so far? I still have the issue... and it is really
> annoying... :(
>
It is a very real issue for me. I still have over 20 of these damn
things running. I had to stop using the Linux sys until this is fixed. I
hated to do that but had no choice.

Kap Har

[Krisztian Poos (rocky-poos)]

11.10 Beta is also affected... I upgraded with big hope... the issue is still here with kernel 3.0.0.8 also unfortunately... :(

[Joseph Salisbury (jsalisbury)]

Would it be possible for someone affected by this to test the latest upstream kernel? It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

[Daniel (hackie)]

Same problem with

$ uname -a
Linux elrond 3.2.0-030200rc4-generic #201112012035 SMP Fri Dec 2 01:44:19 UTC 2011 i686 i686 i386 GNU/Linux

under Ubuntu 11.10 oneiric

as installed from http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2-rc4-oneiric/linux-image-3.2.0-030200rc4-generic_3.2.0-030200rc4.201112012035_i386.deb

$ top
top - 13:59:04 up 6 min, 1 user, load average: 1.35, 0.93, 0.42
Tasks: 130 total, 3 running, 127 sleeping, 0 stopped, 0 zombie
Cpu(s): 9.0%us, 80.1%sy, 0.0%ni, 7.6%id, 3.2%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 2047744k total, 1895768k used, 151976k free, 53812k buffers
Swap: 0k total, 0k used, 0k free, 1717424k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  690 root 20 0 0 0 0 R 77 0.0 1:17.94 kworker/1:3
 1226 root 20 0 0 0 0 S 75 0.0 0:41.20 kworker/0:3
 2151 daniel 20 0 21832 19m 1164 D 22 1.0 0:33.42 unison

It is a HP Mini 1502 with an Atom N450 @ 1.66GHz

[Daniel (hackie)]

Also tried the most recent nightly build from linus' tree:

Linux elrond 3.2.0-999-generic #201112250405 SMP Sun Dec 25 09:13:50 UTC 2011 i686 i686 i386 GNU/Linux

Result is the same:

top - 22:50:56 up 3 min, 1 user, load average: 2.36, 1.24, 0.50
Tasks: 133 total, 4 running, 129 sleeping, 0 stopped, 0 zombie
Cpu(s): 9.7%us, 78.1%sy, 0.0%ni, 6.6%id, 5.6%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2047572k total, 1850700k used, 196872k free, 74540k buffers
Swap: 0k total, 0k used, 0k free, 1654748k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 1149 root 20 0 0 0 0 S 66 0.0 0:34.38 kworker/1:3
   11 root 20 0 0 0 0 R 56 0.0 0:50.80 kworker/0:1
 2128 daniel 20 0 21088 18m 1164 D 23 0.9 0:22.66 unison
    4 root 20 0 0 0 0 R 18 0.0 0:26.84 kworker/0:0
   24 root 20 0 0 0 0 R 10 0.0 0:43.71 kworker/1:1

Btw: how do I have to use to tool powertop to get the form "Top causes for wakeups"?

dmesg does not show any more information after booting

[Per-Inge (per-inge-hallin)]

This seems to be a problem also in Ubuntu 12.04. Kworker takes 100% CPU on one CPU core every 60 sec. Sometimes there is a clash with pulseaudio and audio is distorted.
This is on a desktop with an AMD X2 260 processor at about 3.2 GHz.

[Christian Siefkes (christian-siefkes)]

I sometimes have the same problem with my Thinkpad X61s, running Ubuntu Natty (2.6.38-13-generic #54-Ubuntu SMP i386 GNU/Linux). I finally found out that the problem occurs if two conditions are met:

1. There is no LAN connection (no LAN cable is plugged in or the LAN/router has no working Internet connection).

2. The laptop runs on mains power, not just on battery.

If both conditions are met at startup, the kworkers run amok, taking all CPU power they can get. Since I discovered these conditions, I was able to avoid the problem by starting the laptop from battery power when I have no LAN. I can plug in the power cable later without problems, but it must not be plugged in at startup or the kworkers will go crazy.

When they did, I once run "sudo perf record -ag sleep 10" and "sudo perf report", which essentially showed the following::

    53.80% kworker/0:4 [kernel.kallsyms] [k] delay_tsc
                |
                — delay_tsc
                    __const_udelay
                    e1000_check_phy_82574
                    __e1000e_read_phy_reg_igp
                    e1000e_write_phy_reg_m88
                    e1000e_phy_has_link_generic
                    e1000_configure_k1_ich8lan
                    e1000e_has_link
                    e1000e_up
                    process_one_work
                    worker_thread
                    kthread
                    kernel_thread_helper

[Kyle (kyleflanagan)]

After a fresh install of Ubuntu 12.04, I'm also having this issue. (The side effects are nearly identical to a bug I reported here: https://bugs.launchpad.net/ubuntu/+source/virtualbox/+bug/924684, however the cause may be different.)

This occurred when I was trying to install virtualbox through the software center. There was an error because libpython2.7 could not be downloaded. When I closed the software center, the system hung up.

Also running were: Chrome and Rhythmbox, and Pidgin in the background.

After the system hung up, the audio from Rhytmbox skipped then stopped. Changing to another tty and issuing top revealed that pulseaudio was taking up ~50% CPU with other processes and 12 kworker processes taking up the rest.

I killed Rhythmbox, pulseaudio and Chrome, but anytime I'd kill one of these processes, some other process would begin taking up a large amount of CPU with two or three kworkers always in the top 10 most-CPU-intensive processes.

[Alexander Thole (a.t)]

I recently updated to 12.04 from 11.10 and the problem still existed even with the 3.4 kernel.

Disabling ACPI solved the problem but then the CPUs were running at full speed. As a Workaround I found this:
https://wiki.ubuntu.com/DebuggingACPI#Trouble_Booting

and booting with acpi=noirq worked for me.

$ uname -a
Linux Trifix 3.4.0-030400rc5-generic #201205011817 SMP Tue May 1 22:18:19 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

top:

top - 12:28:01 up 21 min, 1 user, load average: 0.50, 0.40, 0.26
Tasks: 178 total, 1 running, 177 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.2%sy, 0.0%ni, 98.7%id, 0.8%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 6094080k total, 952640k used, 5141440k free, 90032k buffers
Swap: 6269948k total, 0k used, 6269948k free, 403796k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 1998 alex 20 0 394m 11m 8304 S 2 0.2 0:06.51 multiload-apple
 2551 alex 20 0 742m 153m 35m S 2 2.6 0:15.86 firefox
    1 root 20 0 24432 2408 1352 S 0 0.0 0:01.00 init
    2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
    3 root 20 0 0 0 0 S 0 0.0 0:00.09 ksoftirqd/0
    6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
    7 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
    8 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
    9 root 20 0 0 0 0 S 0 0.0 0:00.10 kworker/1:0
   10 root 20 0 0 0 0 S 0 0.0 0:00.10 ksoftirqd/1
   12 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1
   13 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/2
   14 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/2:0
   15 root 20 0 0 0 0 S 0 0.0 0:00.06 ksoftirqd/2
   16 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/2

The Computer is a Lenovo Y560p.

[Harry Rostovtsev (rost0031)]

Alexander's fix (above comment) worked for me (same hardware, Lenovo Y560p) but it kills my battery/power supply detection. It only seems to detect on boot if I'm plugged in or on battery. If I unplug, it doesn't update the power settings or show that I am draining the battery.

[Carlo Capocasa (carlotheman)]

Same hardware (Lenovo 750p), same fix, same problem with the fix (battery detection). Oh well.

To test the fix, I pressed E in grub and added acpi=noirq after "quiet splash".

To make the fix permanent, I went ot the line that said

  GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

in /etc/default/grub and edit it to look like

  GRUB_CMDLINE_LINUX_DEFAULT="quiet splash acpi=noirq"

[Carlo Capocasa (carlotheman)]

Now using pci=noacpi. Fixes the problem, battery detection works.

Correction: There is no lenovo 750p, I'm using the (utterly magnificent) Lenovo y560p.

[Harry Rostovtsev (rost0031)]

Carlo:
Can you verify that you have battery detection working by unplugging your power chord and having your laptop show the battery draining? I just tried your fix and I still have no battery detection. It sees the battery but doesn't know when I unplug the power supply.

[[e-mail] (charles-fox-gmail)]

Is this the same bug as https://bugs.launchpad.net/ubuntu/+source/linux/+bug/887793 ?

[Harry Rostovtsev (rost0031)]

"Is this the same bug as https://bugs.launchpad.net/ubuntu/+source/linux/+bug/887793 ?"

I don't think so. There seem to be 3 causes of these symptoms:
1). Certain ATI/Intel video card combinations. There's a workaround for these.
2). Network driver caused, as the one mentioned in the bug you linked.
3). This one caused by (seems like) ACPI.

[Thomas Andraschko (andraschko-thomas)]

Same on Samsung Serie 7 NP700Z5C-S04DE with Mint 13 or Ubuntu 12.10.
noacpi works but nvidia and bumblebee will not work with noacpi.

[Scott Deagan (scott-deagan)]

I have a Samsung Series 7 Chronos (NP700Z7C-S02UK) and have this problem. I'm not willing to run Linux on my laptop while this issue exists because it's thrashing my fan and making my laptop run hot.

Adding acpi=noirq and pci=noacpi does not solve the issue for me. Screenshot of system monitor and top attached.

[penalvch (penalvch)]

Pieter, this bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue? If so, could you please test for this with the latest development release of Ubuntu? ISO images are available from http://cdimage.ubuntu.com/daily-live/current/ .

If it remains an issue, could you please run the following command in the development release from a Terminal (Applications->Accessories->Terminal), as it will automatically gather and attach updated debug information to this report:

apport-collect -p linux <replace-with-bug-number>

Also, could you please test the latest upstream kernel available following https://wiki.ubuntu.com/KernelMainlineBuilds ? It will allow additional upstream developers to examine the issue. Please do not test the daily folder, but the one all the way at the bottom. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested. If this bug is fixed in the mainline kernel, please add the following tags:
kernel-fixed-upstream
kernel-fixed-upstream-VERSION-NUMBER

where VERSION-NUMBER is the version number of the kernel you tested. For example:
kernel-fixed-upstream-v3.11-rc5

This can be done by clicking on the yellow circle with a black pencil icon next to the word Tags located at the bottom of the bug description. As well, please remove the tag:
needs-upstream-testing

If the mainline kernel does not fix this bug, please add the following tags:
kernel-bug-exists-upstream
kernel-bug-exists-upstream-VERSION-NUMBER

As well, please remove the tag:
needs-upstream-testing

Once testing of the upstream kernel is complete, please mark this bug's Status as Confirmed. Please let us know your results. Thank you for your understanding.

[Pieter (diepes)]

Hello,

I have not been affected by this bug for a while. ( I see it was reported 2 years ago)

Please close, as it seem's not to be a problem any more.

Regards,
Pieter

[penalvch (penalvch)]

Pieter, this bug report is being closed due to your last comment https://bugs.launchpad.net/ubuntu/+source/linux/+bug/717919/comments/45 regarding this being fixed with an update. For future reference you can manage the status of your own bugs by clicking on the current status in the yellow line and then choosing a new status in the revealed drop down box. You can learn more about bug statuses at https://wiki.ubuntu.com/Bugs/Status. Thank you again for taking the time to report this bug and helping to make Ubuntu better. Please submit any future bugs you may find.