Ubuntu

update-apt-xapian-index uses too much CPU and memory

Reported by Denis Rut'kov on 2009-04-19
This bug affects 232 people
Affects Status Importance Assigned to Milestone
APT
Invalid
Undecided
Unassigned
apt-xapian-index (Ubuntu)
Undecided
Unassigned
Nominated for Lucid by filippo colombo

Bug Description

Binary package hint: apt-xapian-index

A background silent update process shouldn't use 98% CPU. It makes system sluggish with no visible reason.

summary: - update-apt-xapian-index usses too much CPU
+ update-apt-xapian-index uses too much CPU

On my old laptop, the CPU usage is not too bad (5%) but it uses insane amounts of memory, ca 100 MB. Most annoying thing since the dreaded scrollkeeper-update.

Gonzo (alex-satellite) wrote :

I confirm that too. It uses 98% CPU but for a relatively short period of time.

1 comments hidden view all 145 comments
Vincenzo Ciancia (vincenzo-ml) wrote :

It uses all the available cpu and interrputs the work of people for no reason. Would someone kindly assign a priority to this bug?

Changed in apt-xapian-index (Ubuntu):
status: New → Confirmed
Pelládi Gábor (pelladigabor) wrote :

Just happened to me on an Asus EEE 901. I was copying large files from a pendrive, and after few minutes my desktop was almost frozen, unusable. The system responded for a mouse action in 30-60 seconds. The hard drive led (SSD in fact) was continuously on. I managed to switch to vt1 and log in (it took a minute), and ran top. It showed that update-apt-xapi was using 100% CPU. After about 8-10 minutes of struggling, the process finished it's mysterious work, and the system was back to normal again. But rendering the box unusable for minutes can be frustrating. At least can somebody tell me why we need this background process at all? Can I uninstall it, without breaking functionality?

Il giorno gio, 28/05/2009 alle 19.06 +0000, Pelládi Gábor ha scritto:
> At least
> can somebody tell me why we need this background process at all? Can I
> uninstall it, without breaking functionality?

The program is described in apt-cache show as supporting the index of
packages that you see when you use "add/remove" from the applications
menu in the gnome panel. I uninstalled it and nothing bad happened (I
use synaptic or apt to install/uninstall applications).

"At least can somebody tell me why we need this background process at all"

I think it trawls through the titles and descriptions of your apt database of packages making it possible for you to search for packages not just for their title but also by their description. However I think in the past I have been able to search by package descriptions long before this process was crippling my PC.

Anyway my solution is a little less dramatic than uninstalling the thing (if its existence is necessary I don't know, I doubt it but have kept it in case). I just went into the directory /etc/cron.weekly and moved the file apt-xapian-index to the folder /etc/cron.monthly. It still rears its ugly head occasionally but at just six times between releases I can live with it.

Tanker Bob (tankerbob) wrote :

Confirmed problem on Ubuntu 9.04 x86_64 Jaunty. I am running in Intel Core 2 Quad Q9650 CPU, and the update-apt-xapian process takes 100% of one core. As of this writing, it has run for 468:50:30 according to top with no end in sight. This is absurd.

I tried purging the package and killing the process, but then the Quick Search window in Synaptic no longer works. I tried making the /etc/cron.weekly/apt-xapian-index script non-executable, but it runs anyway.

Other than the Quick Search in Synaptic, I see no use for this process or its huge database. It either need to be fixed so that it doesn't eat 100% of a CPU core, or eliminated as a requirement for Synaptic's Quick Search so that it can be removed from the distribution.

Tanker Bob (tankerbob) wrote :

I just killed the process I wrote about this afternoon. In checking the database, I noticed that it was last written over 2.5 hours ago. Yet, the update-apt-xapian process was still taking 100% of one CPU core until I just killed it. I took David's advice and moved the executing script from /etc/cron.weekly to /etc/cron.monthly. This needs to be fixed.

jpangamarca (jpangamarca) wrote :

It eats up a very high percentage of memory on my machine. See attachment. I removed the package, seems like it's not very useful anyway.

JeffB (n35cwii9) wrote :

I have an old 866Mhz machine with 256MB RAM.
Top only shows this process consuming around 40% of the CPU.
However, this process brings my machine to its knees.
The mouse doesn't move well. Menus take forever to pop-up.
Programs won't close without the warning "Application is not responding".
I think perhaps I am out of RAM and thus thrashing the cache.

I don't know how the kernel handles a background task wanting all available memory, thus leaving none for the application in focus.

It takes around 10 to 20 minutes to complete running on my machine.

I am running Ubuntu 8.10

Michael Vogt (mvo) on 2009-07-14
Changed in apt-xapian-index (Ubuntu):
status: Confirmed → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apt-xapian-index - 0.21ubuntu2

---------------
apt-xapian-index (0.21ubuntu2) karmic; urgency=low

  * run the weekly update with nice and ionice (LP: #363695)

 -- Michael Vogt <email address hidden> Tue, 14 Jul 2009 09:22:34 +0200

Changed in apt-xapian-index (Ubuntu):
status: Fix Committed → Fix Released
Michael Vogt (mvo) wrote :

I made it run with nice and ionice now. That should make it somewhat less problematic. I'm testing the memory behavior now in a virtual machine with little resources to see what can be done here.

Tormod Volden (tormodvolden) wrote :

> I made it run with nice and ionice now.

That sounds like a band-aid solution. This must be broken either in code or concept if it consumes so much resources. Why doesn't each package installation add itself to a database, instead of having an indexing monster run regularly even if no new packages are installed?

Tanker Bob (tankerbob) wrote :

Is the "fix" committed just in Karmic? It needs to be fixed in other releases since that's where we all encountered it.

Any changes I made to the status of this bug where accidental. Currently update-apt-xapian-index is running and was making my system extremely sluggish with lots of IOWait, and my mouse clicked the Fix Released button on accident.

Changed in apt-xapian-index (Ubuntu):
status: Fix Released → Incomplete
status: Incomplete → Fix Released

Considering the Add/Remove programs doesn't have much of a "live search" feel to it (when I search for packages it doesn't update for a few seconds and takes a lot of CPU) I don't see what this index is accomplishing.

As someone already mentioned, why can't this just be a done at package install/removal time? The prelink package does this. People expect the system to be under heavy usage while doing package management, but a daemon either needs to be unobtrusive and quick.

Also from a scalability point of view I imagine that this daemon redundantly touches a lot of packages that haven't changed, since it's execution time seems to (understandably) grow with the amount of packages I have installed. A post-install solution would cut down on the overall time wasted creating this index.

Olafur Arason (olafra) wrote :

This is still a problem with:
0.21ubuntu2

Firefox and tracker are bad enough but this is ridicules.

Jasmine Hassan (jasmine-aura) wrote :

Confirming #18 Olafur Arason's experience..

On Karmic, with apt-xapian-index-0.21ubuntu2

Agreed with #14 Tormod Volden, and I really wish it was even a band-aid solution.

Test-bed:
Pentium-III 500Mhz with 192MB RAM running a minimal install and a light-weight LXDE/openbox desktop, idle, with only 48MB of the 192MB physical RAM being used.

Result: update-apt-xapian-index consumes 92-98% CPU, and over 37.5% of physical RAM.

I'm sorry, but I do not think the "run with nice and ionice" could be considered a proper fix. So if you'll kindly allow me, I'll change the bug status back to "confirmed" for now.

Changed in apt-xapian-index (Ubuntu):
status: Fix Released → Confirmed
NoOp (glgxg) wrote :

85.4% CPU - Intel 2.4Ghz/1G Ram.
Jaunty 2.6.28-16-generic (Gnome)

$ apt-cache policy apt-xapian-index
apt-xapian-index:
  Installed: 0.16
  Candidate: 0.16
  Version table:
 *** 0.16 0
        500 http://archive.ubuntu.com jaunty/main Packages
        100 /var/lib/dpkg/status

Tormod Volden (tormodvolden) wrote :

On Ubuntu 9.10, it accumulates a whole minute of CPU (observed with top), on a machine with dual-core 3GHz and 2GB RAM.

1 comments hidden view all 145 comments
Mirza (mirza-dervisevic) wrote :

is this bug still present in 9.10 karmic ?

Isak Frants (isakfrants) wrote :

Mirza: Please read the other posts before adding comments. #21 is clearly still affected by this issue and Yes it takes waaaay too much system resources to update its index. On my AMD 3700+ it uses 100% CPU for at least 20s. (I'll monitor it the next time it runs in Karmic to get more exact time.)

Is it only Synaptic that uses this index? Does anyone know? Does Software Center use it as well? I mean, a non-geek person uses Software Center and the ultimate geek uses apt-get. This massive indexing affects all users but how many need or use this feature if it's only Synaptics Quicksearch that takes benefit from it.

The general search function in Synaptic does its job. The user types in a word, clicks enter and expects its CPU to work in order to find the target. Isn't the purpose of an indexing service to speed up the searching? It does, BUT, right now it's almost like its just MOVING the time when the CPU must work, FROM that time when the user expects it to work (when the user wants to search) TO that time when the user maybe watches a movie or playing a game? At least update-apt-xapian-index can in Karmic still bounce up while surfing the net, making the computer quite sluggish.

Graeme Glass (graemeglass) wrote :

Running fully updated karmic and am still affected by this. 1 core sits at 100% for minutes

Isak Frants (isakfrants) wrote :

Not trying to cause a "me too" storm or anything, but since I promised to mention more exactly for how long it uses my CPU I'll add yet another comment. On AMD 3700+ it uses maximum resources for 40-60s. The situation is better now than in Jaunty where it would consume resources on almost every boot while in Karmic it runs more seldom.

Still bad though when I was running Gimp, Word 2007 through Wine and Firefox that this suddenly bounces up and makes everything sluggish :S If it has to run, run when computer is not in use.

Me too! ;-) Couldn't this be launched with a lower nice value?

Rolando (rolcamus) wrote :

update-apt-xapian-index

In my Toshiba One Core notebook uses 100% cpu for several minutes and started at any time making the machine unusable,

MarcS (marc-schmitzer) wrote :

I'm experiencing the same problem on a Dell Laptop.

From the update-apt-xapian-index manpage:
> -u, --update
> incremental update, reindexing only those packages whose version has changed since the last run

Sounds reasonable, but is not used in /etc/cron.weekly/apt-xapian-index.

The attached patch adds this option.

s2 (saulius2) wrote :

MarcS, I suppose unified diff format is the standard nowadays (diff -u:)

MarcS (marc-schmitzer) wrote :

Oh, my bad, excuse the newbie.
It's kinda trivial anyway.

Evan Carroll (evancarroll) wrote :

Still present in up to date karmic Feb 3, 2010. Going to proceed with patch.

tags: added: patch
drizzle (darran-kelinske) wrote :

was experiencing this on lucid lynx alpha2

Kristian Kißling (kkissling) wrote :

affects my too on Lucid Lynx, alpha 2 with all updates, running as guest in virtual box

William Shand (williamshand14) wrote :

Affects me too, starting to get rather annoying :(

Dave Martin (dave-martin-arm) wrote :

I've been experiencing this at least since Karmic. On armel especially, it can sometimes cause sluggish performance for a few minutes after boot.

jonobe (jonolewis) wrote :

I have just switched to Kubuntu 9.10 from the Ubuntu Gnome distros, only to find it is all locked up until I kill this xapi process. It's pretty serious - completely locking up my computer for ages (until I kill it!).
I am amazed reading these comments it has been around so long and still hasn't been dealt with. I'm a bit annoyed because i finally persuaded my Dad (a fervent Windows man) to give Linux a go and gave him the Kubuntu disc I'd just written. D'oh.

Why do I think he's actually going to love this?

I too am plagued by this bug. My system locks up and my fans start spinning full-powered when this process runs. Perhaps defaulting it to a lower-priority nice level would remedy the issue?

Tamás Kiss (tkiss80) wrote :

In Jaunty, I found an interesting piece of code in /etc/cron.daily/mlocate. It checks if the laptop is on AC power, and if not, it exits without updating the locate database. I know this is not closely related to this bug, but apt-xapian-index doesn't seem to have this check. Maybe someone who has a laptop should run some tests to see if it worth putting into the script and could file a bug report if necessary. (I've never had a laptop, so I can't experiment with it).

jferguson (jferg977) wrote :

running 9.10 on a 500mHz P3 notebook, this application pretty much shuts it down for a couple of hours on saturday morning. My experience is same as others I read here. I figured out what it was, killed it and saw the disk-banging stop and got my cpu back.

If all this does is give Synaptic Package Manager quick-search a current index - at least on saturday efternoon, it has to be the silliest resident ap I've ever seen.

I'm taking it out of cron.weekly, making it available to be run manually, while I'm on vacation and seeing if that lets it do whatever it seems to be doing.

If the wise folks who watch the gates on what can get into a Linux installation would keep an eye out for stuff like this, some of use with older machines would be better served.

To think that the Linux believers think that Windows is the only system with bloatware. This is a prime example.

aexl (aexl) on 2010-05-02
Changed in apt-xapian-index (Ubuntu):
status: Confirmed → Fix Committed
Changed in apt-xapian-index (Ubuntu):
status: Fix Committed → Confirmed
aexl (aexl) on 2010-05-11
Changed in apt-xapian-index (Ubuntu):
status: Confirmed → Fix Committed
Changed in apt-xapian-index (Ubuntu):
status: Fix Committed → Fix Released
Changed in apt-xapian-index (Ubuntu):
status: Fix Released → Confirmed
Changed in apt-xapian-index (Ubuntu):
status: Confirmed → Fix Released
aexl (aexl) on 2011-04-28
Changed in apt:
status: New → Confirmed
65 comments hidden view all 145 comments
Jeroen Hensing (hensing) wrote :

Still happens on 11.10 server x86, apt-xapian-index 0.44ubuntu4
why the heck is this marked as fixed even though countless people say it isnt ?!

Jorge Gustavo (jgr) wrote :

Still happens on 11.10 on 3.0.0-12-generic x86_64.

primefalcon (primefalcon) wrote :

I guess one solution is to run

sudo rm /etc/cron.weekly/apt-xapian-index

Still happens on Kubuntu 11.10.

Same here in Oneiric 64bit. In half an hour it ran three times using nearly 100% CPU each time.

Tormod Volden (tormodvolden) wrote :

boozehead, if it was running several times, maybe you experienced bug #594820 ?

For those wondering why this bug report is marked Fixed, and who did not read the whole thread because it has become unreadable, see for instance comment 77. See also bug #655831 which is still open. As I explained there, 100% CPU usage in itself is not a problem.

okay guys. they're not going to look at the case or do anything if you keep adding to a bug that was marked as fixed.
you need to start a new bug.

> Date: Thu, 27 Oct 2011 06:59:08 +0000
> From: <email address hidden>
> To: <email address hidden>
> Subject: [Bug 363695] Re: update-apt-xapian-index uses too much CPU
>
> Same here in Oneiric 64bit. In half an hour it ran three times using
> nearly 100% CPU each time.
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (801007).
> https://bugs.launchpad.net/bugs/363695
>
> Title:
> update-apt-xapian-index uses too much CPU
>
> Status in APT:
> Confirmed
> Status in “apt-xapian-index” package in Ubuntu:
> Fix Released
>
> Bug description:
> Binary package hint: apt-xapian-index
>
> A background silent update process shouldn't use 98% CPU. It makes
> system sluggish with no visible reason.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/apt/+bug/363695/+subscriptions

eixmg (ei-xmg) wrote :

same problem!!!!!!!!!!!!! ubuntu 10.10

kit (andrea-kit) wrote :

i got the same problem, ubuntu 11.10, no way to solve it?

ticket (tickettothemoon2004) wrote :

Same problem here - using Ubuntu 11.10, kernel 3.0.0-15-generic.
Uses 100% of one core.

Sam_ (and-sam) wrote :

Newer bug 903787
Workaround here for Oneiric and Precise, removed apt-xapian-index, no issues found yet.

kit (andrea-kit) wrote :

I found something that helped my laptop to work properly ;) ... hope you find it useful too

1) http://ubuntuforums.org/showpost.php?p=9304431&postcount=8

2) http://ubuntuforums.org/showpost.php?p=10453047&postcount=15

dronus (paul-geisler) wrote :

In my opinion, this is not solved. With nice and ionice the system performance may be less affected, but the power drain by heavy CPU usage will still remain.

The question are: Does the update need to be that costly, and does the user need this updates altogether. As far as I see, the index provide two functions: The quick search mode in synaptic pagage manager, which is not installed by default, and the package recommenadtions in the terminal, rarely useful to normal users either, and expendable by power users if it comes at this large cost.
That's it, or have I forgotten something? If not, I would conclude that apt-xapian-index just should not be installed by default, and has to prevail for users that intentionally install it. It must be made sure however, that no one misses the features without knowledge how to get them back.

aexl (aexl) wrote :

hello paul,

i don't know either, but i think opening and cross-linking a new issue "apt-xapian-index just should not be installed by default" will help get this clearer...!

This bug is currently affecting my old low-end PC running Ubuntu 12.04. The system become completely unusable when apt-xapian-index is running in the background.

This issue is far from solved. My machine is blocked for about 5 minutes regularly:

$ uname -a
Linux xander-pc 3.2.0-24-generic #37-Ubuntu SMP Wed Apr 25 08:43:22 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

$ sudo lshw -C cpu
  *-cpu
       description: CPU
       product: Intel(R) Celeron(R) CPU 2.93GHz
       vendor: Intel Corp.
       physical id: 4
       bus info: cpu@0
       version: Intel(R) Celeron(R) D CPU 2.93GHz
       serial: To Be Filled By O.E.M.
       slot: CPU 1
       size: 2933MHz
       capacity: 2933MHz
       width: 64 bits
       clock: 133MHz
       capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx x86-64 constant_tsc up pebs bts nopl pni dtes64 monitor ds_cpl tm2 cid cx16 xtpr lahf_lm

I forgot to mention that I do not have any RAM shortage (2 GB), so paging should not be the problem (top shows that there is enough free memory but update-apt-xapian-index uses about 11% of it).

this is pretty amazing you should give it a look http://www.inews15fl.net/biz/?read=4902269

somejan (somejan) wrote :

I'm also still experiencing this problem, both on a high end machine (high end, but programs are using up all available memory usually), and a low end system.

atimonin (atimonin) wrote :

Also have this problem with update-apt-xapian

Linux bimer 3.2.0-26-generic #41-Ubuntu SMP Thu Jun 14 16:26:01 UTC 2012 i686 i686 i386 GNU/Linux
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04 LTS
Release: 12.04
Codename: precise

I may do some tests/checks if needed

onyxrev (entp) wrote :

Agreed - this is not fixed. Bug took one of my 12.04 servers out of commission. update-apt-xapi ate up 123MB of memory (the server only has 256MB) and then proceeded to top off the swap until the machine hung hard, forcing us to hard reboot.

vovaseagull (vovaseagull) wrote :

Hello,

Please help, I can't get updates, it says update apt xapian index unfound

1 comments hidden view all 145 comments
Hannibal (hannibal-wurst) wrote :

Affects me too, update-apt-xabian is using one core at 100% for a minute everyday.

Version:
apt-xapian-index 0.25ubuntu2

System:
Distributor ID: Ubuntu
Description: Ubuntu 10.04.4 LTS
Release: 10.04
Codename: lucid

Kernel:
2.6.32-45-generic x86_64

CPU:
Intel(R) Core(TM)2 Duo CPU U9400 @ 1.40GHz

If any more information are needed, I'd be happy to provide them.

Artem (Tim) (cordatus) wrote :

The same problem on the old PC.
On modern PCs do not encounter this problem.

Lubuntu 12.10 - x86
Intel Celeron 1.7Ghz
768 RAM

update-apt-xapian-index
loads of 100% CPU

Help please!

Still a problem on 12.10 (64 bit) - all 4 CPU cores maxed out and hard disk being totally thrashed for several minutes every time I boot up and the desktop loads.

Bug is still there, Ubuntu 12.04.1 server 32bit, running on an old laptop. Process gets started but the daily apt cron job, and after a while it gets killed for running out of memory.

Max Polk (maxpolk) wrote :

A weekly fatal unable to fork error message from /etc/cron.weekly/apt-xapian-index on a server with 512MB memory with a web server and database server running leads to this bug. That cron job calls /usr/sbin/update-apt-xapian-index, which is a Python script that imports axi and axi.indexer.

Sorting is a classic computer science problem with both time and space complexity. For time think CPU. For space think memory.

The solution is to change the sorting algorithm in the python axi and axi.indexer modules. The first priority is to switch to an algorithm that consumes a whole lot less memory (i.e., each step of the algorithm keeps less objects in memory), and it will stop crashing and stop thrashing (memory swapping to disk).

The second priority is of lesser importance (because renice can solve a lot of the effect), which is to switch to an algorithm that takes a lot less time to run (i.e., takes fewer steps to complete), and it will stop consuming so much CPU for so long.

summary: - update-apt-xapian-index uses too much CPU
+ update-apt-xapian-index uses too much CPU and memory
Max Polk (maxpolk) wrote :

On an idle system with 512M memory and 768M swap, while /etc/cron.weekly/apt-xapian-index was running, I used free to see memory and swap used. It rose to then peaked at this:

             total used free
Mem: 503528 497736 5792
Swap: 786428 231196 555232

Total used memory is 728M.

After ending, it immediately dropped back to this:

             total used free
Mem: 503528 259160 244368
Swap: 786428 224484 561944

Total used memory is 483M.

I conclude that apt-xapian-index consumes the difference, which is 245M.

Running "apt-cache stats" I see at the end "Total space accounted for: 26.0 M".

Therefore, it takes 245M to sort and index 26M of information. This seems conclusive that the algorithms, containers, and functions chosen are very inefficient. It should definitely not require 10 times the memory space of what is being indexed.

Max Polk (maxpolk) wrote :

I added my above comments to bug 655831

Julien Olivier (julo) wrote :

I just stumbled on this bug today in raring. So, I confirm that the bug is still not fixed.

Yup, this exists in 13.04 all right.

Germán Bobr (x-german) wrote :

I can confirm this bug is still affecting Ubuntu 12.04 LTS

Encountered this on a fresh instance of raring immediately after installing and updating apt-file. sudo apt-file purge immediately fixed the issue.

Karim Rekik (wkrekik) wrote :

Bug encoutered in 14.04 (dev branch) also.

Braiam Peguero (braiampe) wrote :

Please, fill new bug reports if you still find this issue. The usage is *by-design*. It was designed to not hog the CPU.

Changed in apt:
status: Confirmed → Invalid
Serhiy Zahoriya (xintx-ua) wrote :

> "The usage is *by-design*. It was designed to not hog the CPU."
> "update-apt-xapian-index uses too much CPU"
I'm lost in these arguments.

John Mullen (jmullentech) wrote :

Bralam's comment is irrelevant.

I just encountered the problem with LUbuntu 13.10 32-bit, after installing "Ubuntu Software Center". I then replicated on Ubuntu 13.10 32-bit.

System stats once "update-apt-xapian-index" begins: 99% CPU usage on FOUR CORES with 257MB of memory used by the process which renders the system completely unusable.

I'm seriously tempted to just re-write the entire program and push it all upstream for the sake of the community. This is RIDICULOUS! FIVE YEARS after the bug was noted and there is NO fix outside of simply *not using it*.

You're driving people away from Ubuntu by not fixing this.

Displaying first 40 and last 40 comments. View all 145 comments or add a comment.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers