popularity-contest will stat() every file installed file or ~80k files on my system which grinds things to a halt

Bug #493852 reported by Ævar Arnfjörð Bjarmason
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
popularity-contest (Ubuntu)
Undecided
Unassigned

Bug Description

Binary package hint: popularity-contest

I was trying to use my puny laptop when the popularity-contest cronjob started running and killed all I/O performance on my poor ~5000 rpm disk. When popularity-contest is run it'll stat() every file installed by a package on my system or around 81,000 files on my system which has 2252 packages installed.

I'm willing to send statistics to Ubuntu but I'm not willing to have them grind my laptop to a halt. The Ubuntu installer should be more specific about what users are getting themselves into and provide the option of sending a lite statistics report equivalent to dpkg -l|grep ^ii

ProblemType: Bug
Architecture: i386
CheckboxSubmission: 0169f76fbe64c26c287457397cc58891
CheckboxSystem: bb422ca46d02494cdbc459927a98bc2f
Date: Tue Dec 8 00:56:20 2009
DistroRelease: Ubuntu 9.10
Package: popularity-contest 1.48ubuntu1
PackageArchitecture: all
ProcEnviron:
 LANGUAGE=en_US.UTF-8
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-15.50-generic
SourcePackage: popularity-contest
Uname: Linux 2.6.31-15-generic i686

Revision history for this message
Ævar Arnfjörð Bjarmason (avarab) wrote :
Revision history for this message
Mark Edgington (edgimar) wrote :

The script is simply too resource intensive. If there is a way to make it run over a longer period of time, and require less I/O resources at any given point in time, this would be much preferable to the current situation. I am considering uninstalling it (for now), since providing statistics isn't worth the performance penalty it brings, in my view.

Revision history for this message
Andrey Bondarenko (abone) wrote :

As of version 1.52ubuntu2 popularity-contest started as a regular process. I suppose changing IO and CPU scheduling priorities Idle should place less load to the system, but I cannot notice the change in my environment. Both cases works on my P4-3GHz 3GiB or RAM with two 7200 RPM disks equally well.

Anyway, I agree that system with less RAM or slower disk may suffer from popcon activity. But we have to test things to decide if nice/ionice is enough or we should limit IO with some other means. Please try to do some job while popularity contest is running to check how much it slows down your system.

The following test should simulate current situation:

sudo sh -c "echo 3 > /proc/sys/vm/drop_caches; sync"
time popularity-contest

The next test should run popularity-contest as a low-priority background task

sudo sh -c "echo 3 > /proc/sys/vm/drop_caches; sync"
time ionice -c 3 nice popularity-contest

In case, changing priority is enough, patch it really trivial:

=== modified file 'debian/cron.daily'
--- debian/cron.daily 2011-05-23 15:38:47 +0000
+++ debian/cron.daily 2011-06-10 15:35:17 +0000
@@ -50,7 +50,7 @@

 run_popcon()
 {
- su -s /bin/sh -c "/usr/sbin/popularity-contest" nobody
+ su -s /bin/sh -c "/usr/bin/ionice -c 3 /usr/bin/nice /usr/sbin/popularity-contest" nobody
 }

 do_sendmail()

Revision history for this message
Ævar Arnfjörð Bjarmason (avarab) wrote :

The problem won't go away if you ionice it. It'll just run with ionice priority, but it'll still being doing an ostensibly completely redundant stat() of files on the filesystem, instead of just reading from `dpkg -l`.

So even if that could alleviate the case where it's racing other system processes for I/O, it won't solve the case of your disk being spun up for a long time (since it would otherwise be idle) due to popularity contest.

Why does it need to stat files that packages install in the first place? Can't it just trust that dpkg(1) reports installed packages correctly?

Revision history for this message
Andrey Bondarenko (abone) wrote :

No it can't. The goal of popularity-contest not only collect what packages are installed, but also determine what packages were recently used. In order to do so, It checks what files are currently open and scans last access time for files belonging to installed packages.

Probably, you may ask an option to disable collection of usage statistics, but it probably makes popcon results less usable. Such request should be forwarded to upstream developers in debain BTS, I think.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in popularity-contest (Ubuntu):
status: New → Confirmed
Revision history for this message
Kived (kived) wrote :

I've now opted out of popularity-contest. It's not worth it to me to have my system grind to a halt every week, and I generally just get pissed off and kill the process anyway. Microsoft collects application usage information in Windows as well, and while the information isn't always 100% accurate, Programs and Features loads in only a few seconds. C'mon, Ubuntu... you're not allowed to lose to Windows! :P

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers