Runaway irqbalance consuming CPU and Memory

[Francis Ginther]

Looks like a runaway irqbalance process is consuming massive amount of Memory and CPU. This is running on a calxeda arm server used as a build server. Was unable to find any mention of irqbalance in /var/logs. Hopefully apport recorded the right data.

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  861 root 20 0 1896m 1.5g 372 S 40.5 39.1 1447:25 irqbalance

$ apt-cache policy irqbalance
irqbalance:
  Installed: 1.0.5-2
  Candidate: 1.0.5-2
  Version table:
 *** 1.0.5-2 0
        500 http://ports.ubuntu.com/ubuntu-ports/ raring/main armhf Packages
        100 /var/lib/dpkg/status

ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: irqbalance 1.0.5-2
ProcVersionSignature: Ubuntu 3.8.0-31.46-generic 3.8.13.8
Uname: Linux 3.8.0-31-generic armv7l
ApportVersion: 2.9.2-0ubuntu8.5
Architecture: armhf
Date: Fri Nov 1 06:33:51 2013
MarkForUpload: True
ProcEnviron:
 LANGUAGE=en_US:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: irqbalance
UpgradeStatus: Upgraded to raring on 2013-07-24 (99 days ago)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Agel (archagel)]

Same issue Odroid-XU with Ubuntu 13.10 (GNU/Linux 3.4.70 armv7l) .

[Thorsten (kekschen)]

Also seen on OMAP4 Pandaboard with Ubuntu 13.10 (GNU/Linux 3.5.0-236-omap4 armv7l)
CPU Usage is not a problem until irqbalance has eaten almost all ram.

[E. Sununs (sununs11)]

Same on RK3066 board with Ubuntu 13.04 (GNU/Linux 3.0.36+ armv7l).

[Namless One (namlessone)]

Same on cubieboard2 with Ubuntu 13.10

[Michael Sparmann (theseven)]

Seen on several ARM devices (ODROID-X2, Cubieboard1, OLinuXino-A20-MICRO) on Ubuntu 14.04 as well.

[Leon (leonbo)]

Same issue on Rpi2

[Nathan Rennie-Waldock (nathan-renniewaldock)]

Also on Raspberry Pi 2 with irqbalanxe 1.0.6-3ubuntu1 (vivid). It was using 270MB, I've now disabled the service.

[Nick B. (futurepilot)]

I seem to have run into this on the Raspberry Pi 2 with 1.0.6-2ubuntu1 on 14.10. It may be related to https://github.com/Irqbalance/irqbalance/issues/5 which was fixed in 1.0.7. Indeed the Pi 2 does not have /sys/bus/pci. I've packaged 1.0.9 for the Pi 2 in this PPA https://launchpad.net/~futurepilot/+archive/ubuntu/raspberry-pi-2/

Is there a reason Debian and Ubuntu are using such an old version?

[ageekhere (veggieman25)]

I also have this issue with the Raspberry Pi 2 running Ubuntu 14.04.2 LTS (GNU/Linux 3.18.0-20-rpi2 armv7l).

With irqbalance 1.0.6-2ubuntu0.14.04.1

irqbalance needs to be restarted dayily.

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better, and to Nick for your investigation of the issue.

It looks like Debian is still on 1.0.6 but has https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=773843 open requesting an update.

However, we won't update the versions in existing stable releases for stability reasons. If someone can verify the fix Nick identified though, we can cherry-pick just this fix back to 14.04 and to 14.10. See https://wiki.ubuntu.com/StableReleaseUpdates for details of update policy, rationale and procedure. The memory leak seems trivial and obvious enough to fix - we just need someone to test that the cherry-pick works, prepare the updated packaging and to test the final binary proposed package before it is recommended to all users.

It's tricky for me to do it because I don't have appropriate hardware to verify the fix. But if someone who is affected can follow the procedure documented at https://wiki.ubuntu.com/StableReleaseUpdates#Procedure please, I'd be happy to sponsor an upload for you if you are prepared to complete the final verification.

[Olli Niemi (olliniem)]

Happens also with Raspberry Pi 2 and Ubuntu Mate 15.04. Irqbalance has 497 MB usage.

[Olli Niemi (olliniem)]

The leak was around 60 MB/day. Now running irqbalance 1.0.9-0ubuntu1~ppavv1 from the ppa. I will report more later.

[Olli Niemi (olliniem)]

Memory usage with irqbalance 1.0.9-0ubuntu1~ppavv1 is 5 MB after 23 hours.

[Olli Niemi (olliniem)]

After 3 days memory usage is still the same 5 MB as reported previously. Is 5 MB within normal bounds?

[Olli Niemi (olliniem)]

Raspberry updated to 15.10 Ubuntu Mate and 480 MB memory lost in irqbalance.

[Olli Niemi (olliniem)]

Seems like Ubuntu 16.04 (release date 2016-04-21) will have irqbalance 1.0.9-1. It is still the old one for 15.10.

[Nuno Sucena Almeida (slug-debian)]

Same here, running ubuntu mate 15.10 on raspberry pi 2:

Out of memory: Kill process 472 (irqbalance) score 849 or sacrifice child
Killed process 472 (irqbalance) total-vm:641256kB, anon-rss:638124kB, file-rss:244kB

apt-cache policy irqbalance
irqbalance:
  Installed: 1.0.6-3ubuntu3
  Candidate: 1.0.6-3ubuntu3
  Version table:
 *** 1.0.6-3ubuntu3 0
        500 http://mirrors.mit.edu/ubuntu-ports/ wily/main armhf Packages
        100 /var/lib/dpkg/status

[sam tygier (samtygier)]

This is fixed in 1.1.0-2 version in Xenial (16.04). It seems to work fine to install that xenial version into 15.10.

wget https://launchpad.net/ubuntu/+archive/primary/+files/irqbalance_1.1.0-2_armhf.deb
sudo dpkg -i irqbalance_1.1.0-2_armhf.deb

After 3 days it is only using 1.2 MB.

[sam tygier (samtygier)]

1.1.0-2 version looks good a week later, so i'll mark this fix committed