update-apt-xapian-index bogs down system

[jhfhlkjlj]

Binary package hint: apt-xapian-index

This is related to bug 363695 but is not a dupe - that has been "fixed".

every time the update-apt-xapian-index occurs, the system crawls. Possibly related to the I/O responsiveness bug of bug 131094.

The process is extremely intrusive, taking simple things such as flash video and throwing the performance down the toilet.

While this apport bug was collected via a Lucid machine this also affects maverick.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: apt-xapian-index 0.25ubuntu2
ProcVersionSignature: Ubuntu 2.6.32-25.44-generic 2.6.32.21+drm33.7
Uname: Linux 2.6.32-25-generic i686
NonfreeKernelModules: nvidia
Architecture: i386
Date: Wed Oct 6 12:32:35 2010
InstallationMedia: Ubuntu 10.04 LTS "Lucid Lynx" - Release i386 (20100429)
PackageArchitecture: all
ProcEnviron:
 LANG=en_US.utf8
 SHELL=/bin/bash
SourcePackage: apt-xapian-index

[NoOp (glgxg)]

$ sudo update-apt-xapian-index

On 10.04.1 32bit 2.4Ghz/1GiB:
  from top:
32305 root 20 0 99.9m 83m 4636 R 84.9 8.3 1:06.20 update-apt-xapi

84.9% CPU - and I've seen it get as high as 99%. Fans kick in, system respones slow during the update.

On a 10.10 64bit (updated as of today) 2.1Ghz/3GiB:
  from top:
10620 root 20 0 272m 95m 11m R 99 3.2 0:06.18 update-apt-xapi

99% CPU. Fans kick in, system respones slow during the update.

[Tormod Volden (tormodvolden)]

Yes, I think also that bug 131094 is responsible for this.

NoOp, note that when update-apt-xapian-index is run automatically, it is launched with nice and ionice to be less intrusive, so "benchmarking" with calling it directly is not correct. And high CPU usage is perfectly normal if it runs alone.

What I find abnormal is the accumulated CPU usage, it seems unnecessary to do so much computation to update an index. But that can be more poor design than a bug. Naturally, most developers have very powerful machines, so these things often go unnoticed.

[jhfhlkjlj (fdsuufijjejejejej-deactivatedaccount)]

It is the accumulation that is destroying the performance. Try playing an intensive game when that process decides to kick in and all hell will break loose with the computer.

What is the point of this Xapian index other than for the quick search function of Synaptic? It seems EXTREMELY useless to me. Will this package see the end when synaptic is finally replaced?

[NoOp (glgxg)]

On 10/06/2010 01:24 PM, Tormod Volden wrote:
> Yes, I think also that bug 131094 is responsible for this.
>
> NoOp, note that when update-apt-xapian-index is run automatically, it is
> launched with nice and ionice to be less intrusive, so "benchmarking"
> with calling it directly is not correct. And high CPU usage is perfectly
> normal if it runs alone.

I notice the same when it runs automatically (CPU wise). I can actually
tell after booting in the morning when it is running... the fans on my
desktop kick on & stay on until it's finished. I also monitor via top &
System Monitor.

>
> What I find abnormal is the accumulated CPU usage, it seems unnecessary
> to do so much computation to update an index. But that can be more poor
> design than a bug. Naturally, most developers have very powerful
> machines, so these things often go unnoticed.

Agree. Other applications that use the xapian backend also tend to have
high cpu overhead - recoll for example.

I didn't find any cpu related bugs at xapian:
http://trac.xapian.org/report/1
or upstream:
http://bugs.debian.org/cgi-bin/pkgreport.cgi?package=apt-xapian-index
http://packages.qa.debian.org/a/apt-xapian-index.html
So I'm not sure how to debug/troubleshoot. Perhaps file an upstream bug?

[Tormod Volden (tormodvolden)]

NoOp, if nothing else is running, you would want the indexer to use all CPU available so that it finishes as soon as possible (before you want to use the machine for instance). And full CPU makes the fan spin up for a while, that is just healthy (although I admit I hate to hear the fan spin up when I am not using the machine myself). Having it running for hours at crippled CPU levels would be even worse.

IMO, apt-xapian-index might be a wonderful software architecture for indexing of meta-data and all that, but it does not belong to the Linux desktop unless it is re-engineered.

[Vidar Braut Haarr (vhaarr+launchpad)]

I do not understand the purpose of this piece of software.

It seems that when I found out about it and removed it, the only functionality I lost was being able to "quickly" search using the toolbar in Synaptic?

Surely that's not worth heating up my computer for.
I manage fairly well with Ctrl+F to search (or whatever your hotkey might be.)

Does the new "Software Center" thingy use it? If it does, I don't see why it should; the traditional search in Synaptic is just as fast when you consider that actually redrawing the list view takes just as long as the actual search.

Further, I don't understand why you need to index metadata on top of apt, surely apt-cache or whatever has all this information available somewhere already? This all seems very complicated for absolutely no gain at all.

There is probably some purpose to it that I can't see. But at least it's gone from my computer from now on.

[NoOp (glgxg)]

@Tormond: Actually the indexer (IMO) should work as a background job using CPU (limited) when idle. Instead it takes control of the system regardless of what is running. It causes issues with other applications, creates a possible hardware issue, etc. I do however agree with your last comment that it doesn't belong on the desktop unless re-engineered.

I suspect that the above is why there were changes made in the Maverick release:

apt-xapian-index (0.39ubuntu1) maverick; urgency=low

  * debian/postinst:
    - when upgrading, ensure the index is fully rebuild (in the
      background) to ensure that we get updated information in
      /var/lib/apt-xapian-index/{index.values} and that the index
      fully utilizes the new plugins (LP: #646018)

I guess the question is how to test in lucid. I've just installed that version <https://launchpad.net/ubuntu/+archive/primary/+files/apt-xapian-index_0.39ubuntu1_all.deb> & will give it a few days to determine if it helps.

[Janet (bugzilla-kerridis)]

Same problem here with Kubuntu 10.10, apt-xapian-index 0.39ubuntu1. On my notebook with Celeron M 1,3 GHz processor & RAM 512 MB shared (464 available for the system) this process takes up to 91% CPU and eats the whole memory so that the system nearly freezes (takes up a a minute to open kickoff menu after the cursor finally made it to the button). The process interupts my work for about 10 to 15 minutes.

[W3ird_N3rd (w3ird-n3rd)]

On my 2,9Ghz dualcore this generally doesn't cause problems, however, I'm typing this message on a Pentium 3 laptop with lucid. 850Mhz, 384MB RAM minus some for integrated video, which also ruins RAM performance. You don't want update-apt-xapian-index on this machine.

Maybe it would be an idea to check the system before running. Anything that either has a high load, has a clockspeed under 1Ghz or is an Atom and anything <1GB RAM is probably better off without this process and that's fairly easy to detect, isn't it?

[niknah (hankin0)]

ubuntu 10.10, apt-xapian-index 0.39ubuntu1

This happened to me this morning.
i5 CPU, 4gig ram. But my computer was under heavy load already.

Completely froze for a few minutes, hard disk light on constantly.
It's probably allocating too much memory somewhere cause "top" says I've used 1.3gig of swap when it finished. Usually, I have zero swap usage and 1gig of free ram. This thing could do with a ulimit somewhere.

Unfortunately I couldn't duplicate it by running with the -f option, it's maybe a random thing. It usually eats up 300meg of virtual, 156meg of resident ram, probably not a good thing for <=512meg computers. If you have a small/slow computer remove this one and maybe mlocate too, another indexer that runs overnight.

Note: This program is runned twice from cron.weekly/apt-xapian-index & cron.daily/apt. And it doesn't seem to complain when I run it twice at the same time. (but that wasn't my problem cause the weekly cron wasn't running today)

[Poul Wittig (e-poul)]

Same problem for my Ubuntu Server 11.10, fully updated micro instance servers running in the Amazon cloud. When it happens I can neither SSH into them, TeamSpeak server starts lagging immensely and my blog is no longer connectable to. The problem I solved by removing the package, but it is quite a severe bug I would say, for a standard installation to have.

[pfrenssen (pieter-frenssen)]

Also happens on Ubuntu 11.04.

[Mathieu Leocmach (mathieu-leocmach)]

Also happens on Ubuntu 11.10
This is especially painful when you have a single core machine and 1GB of RAM. The CPU use is somehow tamed by nice, but not the memory use.

[Anoop (dr-adshah)]

I purged this package and found no problems with Synaptic or Software Centre. However the update-apt-xapi process always used to cause my (single core Intel Celeron) computer to run so slowly as to be unusable (on Ubuntu 11.10).

I think this package should not be included in the default installation as it does not seem to provide any benefit.

[Andy Duffell (andy-duffell)]

This bug is still extant on 11.10. Checked the anacron job and nice is set to 19, but it still maxed out my dual-core E8400 and caused music playback to stutter horrifically. That's a bit ridiculous for a background maintenance job.

[ticket (tickettothemoon2004)]

Same problem here - using Ubuntu 11.10, kernel 3.0.0-15-generic.
Uses 100% of one core.

[Dan Kegel (dank)]

On Lubuntu, on a 600 MHz Celeron with 128MB of RAM (!), this process really, truly bogs the system down :-)
It uses 79MB of physical ram... which means the entire rest of the system is swapped out.
It never completes.

[Jonathan Marsden (jmarsden)]

I think the simple workaround is to uninstall it if you don't need it:

  sudo apt-get purge apt-xapian-index

Getting it to play more nicely could be difficult; it already uses both nice and ionice from what I can see.

A default install of Lubuntu 12.04 (as of the 2012-03-23 daily build) does not install apt-xapian-index, so this issue should not happen in Lubuntu 12.04, unless people explicitly *choose* to install that package.

[Otus (jan-varho)]

The amount of memory it uses seems to depend on the number of packages installed?

On my Precise netbook with 1GB ram it uses at least 150MB memory, which pretty much always causes some swapping. Audio and video are completely unresponsive for 5 minutes at a random time almost every day.

Sucks if there's something like a Skype call going on. Would be highly embarrassing if I happened to be showing a presentation just then.

Maybe it should only run if no multimedia is on* and/or if there is a lot of free memory? But really, there has to be something wrong with the design, because apt/dpkg is much less intrusive when it's actually installing the stuff and spinning the disk.

* Can the hints apps give to inhibit suspend and screensaver be used?

[Alexander Korsunsky (fat-lobyte9)]

The whole thing is getting ridiculous. The bug is two and a half years old (that makes five (!) releases), there seems to be no benefit to it, so what is the point of keeping this package?

I realise that this job is started with a very low ionice and nice value, and it's *supposed* to not disturb the user because of that. But that plainly doesn't work.
It effectively kills my system for 10-15 minutes.

At least for my system, the real problem is a combination of low ressources:
If memory is low, update-apt-xapian-index starts using swap space (for me, "kswapd0" uses the most I/O ressources when u~-index runs). That combined with actual data being read from the disk and slow hard drive speed effectively makes a system unusable.

Also, the job is run much more often than it should. I already moved the start script from cron.weekly to cron.monthly, but it *still* runs just about every other day.
Also, what is the point of building an index, if Synaptic and software-center have to rebuild the index anyway?

This bug is a serious annoyance for people with weak hardware, and effectively makes the Ubuntu default installation unsuitable for netbooks.

[Ivan Zakharyaschev (imz)]

I don't agree with the comment saying that it is not installed in Ubuntu 12.04 by default: I've installed Ubuntu 12.04 for Toshiba AC100 (ARM) from the "homepage" of this project.

And I have this disaster in my system.

It shouldn't be installed by default, at least, for such weak machines as the little Toshiba AC100.

[Marc Dietrich (marvin24)]

Make sure zram and zcache are enabled, e.g. "cat /proc/swaps" and "dmesg|grep zcache". This may help in low memory situations.

[Ivan Zakharyaschev (imz)]

@marvin24: Thanks for the hint!

zcache seems not to be enabled in my system (but the head of dmesg can be cut off).

Is there another way to tell whether zcache is enabled?

Still, even if it were enabled, I doubt that I wouldn't notice this extra-heavyweight program.

[Oliver Grawert (ogra)]

dmesg cant be cut off in /var/log/dmesg, thats the file where your dmesg output gets dumped to right after boot, just grep in there instead of piping teh dmesg output to grep ...

zram is enabled by default in all ubuntu ac100 images "cat /proc/swaps" should confirm this ...

[Ivan Zakharyaschev (imz)]

Yes, I'm sure zram is enabled. I'm glad to know this.

But zcache is not mentioned anywhere in the logs. Has something gone wrong? Must it have been enabled in the Ubuntu 12.04 for Toshiba ARM that I've installed (from the wiki page)?

Where to learn how to enable it in a sensible way for this system?

[dino99 (9d9)]

Thats what i get on RR i386, logged as gnome-classic:

Traceback (most recent call last):
  File "/usr/sbin/update-apt-xapian-index", line 97, in <module>
    indexer.slave()
  File "/usr/lib/python2.7/dist-packages/axi/indexer.py", line 454, in slave
    childProgress = ClientProgress(self.progress)
  File "/usr/lib/python2.7/dist-packages/axi/indexer.py", line 211, in __init__
    self.sock.connect(axi.XAPIANDBUPDATESOCK)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 2] No such file or directory

That report is beeing old and is still unassigned; Does that mean no one care ?

[Max Polk (maxpolk)]

A weekly fatal unable to fork error message from /etc/cron.weekly/apt-xapian-index on a server with 512MB memory with a web server and database server running leads to this bug. That cron job calls /usr/sbin/update-apt-xapian-index, which is a Python script that imports axi and axi.indexer. Those Python packages are the heart of the problem.

On an idle system with 512M memory and 768M swap, while /etc/cron.weekly/apt-xapian-index was running, I used free to see memory and swap used. It rose to then peaked at this:

       total used free
Mem: 503528 497736 5792
Swap: 786428 231196 555232

Total used memory is 728M.

After ending, it immediately dropped back to this:

       total used free
Mem: 503528 259160 244368
Swap: 786428 224484 561944

Total used memory is 483M.

I conclude that apt-xapian-index consumes the difference, which is 245M.

Running "apt-cache stats" I see at the end "Total space accounted for: 26.0 M".

Therefore, it takes 245M to sort and index 26M of information. This seems conclusive that the algorithms, containers, and/or functions chosen are very inefficient. It should definitely not require 10 times the memory space of what is being indexed.

The solution is to change the sorting algorithm in the python axi and axi.indexer modules. The first priority is to switch to an algorithm that consumes a whole lot less memory (i.e., each step of the algorithm keeps less objects in memory), and it will stop crashing and stop thrashing (memory swapping to disk).

The second priority is of lesser importance (because renice can solve a lot of the effect), which is to switch to an algorithm that takes a lot less time to run (i.e., takes fewer steps to complete), and it will stop consuming so much CPU for so long.

[Hontvári József Levente (hontvari)]

Max Polk has a good analysis. Partly to help google, I would add that on a Quantal 64 bit server installed with "Install a minimal virtual machine" option and 256 MB RAM, the command (simulating the cron job)
  sudo /usr/sbin/update-apt-xapian-index -v -f
results in an additional 300 MB RAM usage (100 MB from swap).

I noticed this issue because almost every day I receive a mail from CRON saying:
  /etc/cron.daily/apt:
  Killed

Looking at syslog, the kernel frequently kills it:
Apr 13 10:37:43 regmail1 kernel: [325133.105582] select 1 (init), adj 0, size 167, to kill
Apr 13 10:37:43 regmail1 kernel: [325133.105592] select 625 (java), adj 0, size 15923, to kill
Apr 13 10:37:43 regmail1 kernel: [325133.105596] select 3915 (update-apt-xapi), adj 0, size 36048, to kill
Apr 13 10:37:43 regmail1 kernel: [325133.105598] send sigkill to 3915 (update-apt-xapi), adj 0, size 36048

Obviously the indexing algorithm which was chosen is either extremely RAM hungry, or the implementation has a bug.

[Gergely Csépány (cheoppy)]

Just an idea: the case and environment Hontvári József Levente (hontvari) mentioned on 2013-04-13 should be easy to recreate, and could be used to track down the problem with this indexer.

[Michael Murray (reasgt)]

This bug affects Ubuntu 12.04 (Precise Pangolin). I noticed the CPU hogging for several minutes. I did not notice any disk I/O slow down.

[Adam Porter (alphapapa)]

Still affecting Raring.

We've yet to receive an explanation of the rationale for this package being installed by default (or even for its very existence). It's been causing major usability problems for almost 3 years now. It's unfathomable that it's installed on Ubuntu server VM images!

This is exactly the kind of problem that puts people off using Ubuntu and drives them, bewildered, back to Windows, knowing nothing except that "Linux is slow as molasses compared to Windows; randomly it will get so slow I can't even watch a video on YouTube!"

Here are the packages that depend on or recommend apt-xapian-index:

$ apt-cache rdepends apt-xapian-index
apt-xapian-index
Reverse Depends:
  software-center
  muon-installer
  muon-discover
  muon
  synaptic
  packagesearch
  packagekit-backend-aptcc
  muon-installer
  muon-discover
  muon
  goplay
  fuss-launcher
  ept-cache
  software-center
  python-apt
  aptitude

What does it take to get something done about this? Should I upload patches to these packages that removes it from their dependencies?

[memartin (memartin)]

For me, the apt-xapian-index cronjob used to crash a complete Xen domU (Raring minimal install w/ linux-vm, 512 MiB RAM, 1 vcpu, 1 GiB swap) on a weekly basis until I disabled the cronjob. Took me a while to find out why the domU crashed in such a timely manner.

I do not have this problem on other hosts running 12.04 Precise, but this one I upgraded to quantal/raring and the problem appeared. I have yet to try an upgrade to saucy with the cronjob reenabled.

[memartin (memartin)]

Just had another look into my notes. The problem appeared for me after upgrading from quantal to raring. I have also found bug 1152736 and bug 1185172 dealing with a potential kswapd regression in Kernel 3.8 and might be related.

[rogerdpack (rogerpack2005)]

Just ran into "this issue" (FWIW) today, on 13.10. Basically, if you install a new distro, the first time you go to use it it drags to a crawl thanks to something named "update-apt-xapi" in top using all of one core. Very frustrating for users of fresh installs (i.e. new users included). Could it be niced/ionice'd please? Thanks.

[Roman (m01brv)]

In KUbuntu 14.04 this bug is getting terrible. This process, update-apt-xapi, tries to take about 400 M of memory, and given the minimum cumulative KDE memory consumption of 1.5 G at idle, all my 2 G get consumed, so I can run nothing more! This indexing process starts almost each time when I start an upgrade through muon or a muon-related software. I cannot just remove the apt-xapian-index, because important things depend on it.

[Gergely Máté (sportember)]

update-apt-xapi is so powerful that it can also block 14.04 users from doing their job.

[quadra (info06)]

On Xubuntu 14.04 (supposed to be light-weight), same problem, update-apt-xapian-index is knocking down the system

[R. Diez (rdiezmail-ubuntu)]

I hit this problem while upgrading from Kubuntu 13.10 to 14.04. I have an old laptop with 512 MiB RAM and the updating process crawled to a halt.

I just created a fake package to get rid of apt-xapian-index without having to edit system configuration files, see here:

https://github.com/rdiez/Tools/tree/master/FakeReplacementForAptXapianIndex

[dino99 (9d9)]

That issue is still disturbing when updating vivid archive; maybe it's time to 'assign' a Debian/Ubuntu maintainer

[Mateusz Stachowski (stachowski-mateusz)]

This is still a problem in 16.04 LTS.

This process uses 100% CPU for a long time and up to 500MB of RAM on my machine.

[dino99 (9d9)]

With actual commonly used cpu having multi cores, the index should be rebuilt on at least 2 cores (or half the total cores number, to let the others available for the other processes)

[Singtoh (singtoh)]

Hello All,

Just thought I would add a comment to this.
The apt-xapian-index has been driving me mad since installing Ubuntu 16.04, being very slow rebuilding and it was rebuilding every time I opened Synaptic taking a few minutes. I had a script that I found on ubuntu forum that would rebuild the index on a cronjob but that didn't solve the slow rebuilding of the index. I just found this build of apt-xapian-index and now the index rebuilds in only a couple of seconds and seems to have solved the problem on my machine. Here is the link if anyone is interested: https://packages.debian.org/sid/all/apt-xapian-index/download. This file(apt-xapian-index_0.47+nmu2_all.deb) seems to have cured things for me. All I did was purged Synaptic and also apt-xapin-index and re-installed Synaptic, then the new apt-xapian-index from the link and it works super on this machine. I hope this helps someone as I have tried everything to get this index thing working properly. What a PIA,

Cheers,

Singtoh

[Mateusz Stachowski (stachowski-mateusz)]

Installing a pure Debian package over Ubuntu one isn't a good advice.

Anyway I tried it and all it did was a crash fest for update-apt-xapian-index process. It crashes for every single operation.

[Mateusz Stachowski (stachowski-mateusz)]

Correction I didn't purge Synaptic and apt-xapian-index before. Now I did and the crashes stopped.

The update-apt-xapian-index process is still heavy on resources but it does work faster. It needs about one minute on my machine after refreshing sources in Synaptic.

[Tynach (tynach2)]

Singtoh, I give many thanks for this fix. I have 4 cores (2 hyperthreads each, so '8 cores'), so the high single-thread usage didn't bother me too much... But the amount of time it took really frustrated me.

This goes from a 'leave Synaptic open for a while and do something else' sort of thing, to a 'wait a couple seconds more' sort of thing. HUGE difference!

[dino99 (9d9)]

Problem has been fixed long ago

[Tiedemate (bjoerntiedemann)]

Nope, the problem has not been fixed. Still there with Kubuntu 18.10.

[Jānis Kangarooo (kangarooo)]

Still in 2022 on fresh 18.04.5 updated to 18.04.6 Kubuntu 32-bit