tracker causing very high disk useage/thrashing

[Tom Badran]

Binary package hint: tracker

I've been tracking gutsy for a few months now, recently i noticed my disk just seems to be constantly in use (its not very quiet in this machine). The only things that seem to be running according to top are trackerd processes. They are reniced to 19 which seems sensible, but the disk useage is just crazy.

Doing a full dd dump of the entire disk takes roughly 15minutes. However the tracker processes just seem to run for ages (>1hour) and cause the disk to be constantly in use for this whole time. Every now and then the system just seems to stall and things that take < 1 second usually (opening a terminal window is a good test of this) end up taking approximately 10 seconds. During these stalls the whole interface seems to be unresponsive too.

I dont know exactly whats going on that causes this, but the only consistent element when investigating this has been 3 trackerd process, niced at 19, using roughly 9% cpu each (although this cpu use percentage does jump around alot, varying from 1% to 10%).

[Jamie McCracken (jamiemcc-blueyonder)]

We have improved this in svn. It should not slow down other processes accessing to disk (such as compilation or check outs)

indexing can take a long time but you can configure it to exclude certain directories if you have tons of indexable content (preferences->indexing preferences)

feel free to reopen this bug if problems persist (IE if it is highly detrimental to other running apps )

[Sebastien Bacher (seb128)]

tracker (0.6.2-0ubuntu1) gutsy; urgency=low

  * New upstream version
    - New Sqlite based indexer which utlises the new incremental blob I/O
      in sqlite 3.4
    - Highly optimised email indexing (up to 5x faster)
    - Dramatically reduced disk access and disk contention (LP: #131983,#135115)
    - Indexer now pauses for a grace period when non-tracker processes write
      to disk (providing changed files are being watched by tracker) -
      minimises slowdowns when compiling or checking out source code
    - Makes use of idle class disk IO scheduling if available
    - Makes preliminary use of NO_ATIME (some disk access still uses fopen)
    - Fixed restore of user metadata on re-index (keywords are auto-restored)
    - Added increased number of (junk) files to automatically ignore
    - Improved stopwords
    - New deskbar handler that uses the new deskbar api (2.19+)
    - Fixed old deskbar handler to remove race condition causing crashes
    - Fixed a number of annoying bugs in email indexing and tracker
      preferences (LP: #132921)
  * debian/libdeskbar-tracker.install:
    - updated
  * debian/rules:
    - use auto mode for the tracker configure option

 -- Sebastien Bacher <email address hidden> Wed, 05 Sep 2007 12:58:14 +0200

[Tom Badran (tom-badran)]

Recent update makes no noticieable difference to this problem for me, the cpu useage is definately down, but my disk is still grinding away and its noticeably affecting interactivity of other programs.

There doesnt seem to be a reopened status, so set back to confirmed, not sure if thats correct.

[Miguel Rodríguez (migrax)]

Can you try to run
sudo ionice -c 3 -p $PID_OF_TRACKERD
(you have to manually replace $PID_OF_TRACKERD with the right value) to see if this is related to #138249?

[Tom Badran (tom-badran)]

Hey Migeul, afraid that makes absolutely no noticeable difference. I guess this bug is due to something else. Any thing else you can think of that can help diagnose the problem? I can collect any system info you need here.

[Jamie McCracken (jamiemcc-blueyonder)]

Tom,

0.6.2 has a bug that leaks memory so I guess check rss of trackerd with top.

If free memory is in short supply the kernel will have a hard time caching updates to disk.

To debug iowait state run vmstat 1

the last column wa shows the percent of cpu waiting for IO. If its consistently above 90% then thats very bad and will account for severe loss of desktop responsiveness

If memory is not an issue then its likely a kernel issue or the index files (file-index.db and email-index.db) in $HOME/.cache/tracker are > 1gb in size (sqlite is only scalable up to about a GB or so)

[Tom Badran (tom-badran)]

I've done a fresh gutsy install (tribe 5 + immediately updating) with a new home directory too (i wanted to split my / /home and /usr/local partitions anyway, so it seemed like a good time) and i dont seem to be having any problems now.

In response to Jamie, the files in .cache/tracker were around 100-200megs each, as opposed to gigs.

The only difference i know for certain between a fresh install and what i had however, was that i did have a n extensive wine directory before (although it was marked as not to be indexed)

I'll leave the bug open for the next few days, and post some more info based on jamie's hints for debugging. If it doesn't reappear ill close the bug off then.

[Jamie McCracken (jamiemcc-blueyonder)]

Tom, thanks for the info

This was exactly my experiences when upgrading - https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/131094

[Tom Badran (tom-badran)]

Jamie, that bug pretty much describes my entire experience with gutsy up until now, and oddly like metioned in that bug disabling esd actually helps too.

It does seem that there is a serious problem with 2.6.22 with regards to io/cpu scheduling, i've never been very happy with gutsy in that regard. I do have a very powerful machine, and notice regular problems (which were extremely exacerbated by tracker hence filing the bug here).

[Paul Sladen (sladen)]

New install => empty home directory => no content to index(!)

Upgrade/real world situation => 10GB of files in ~/ => sudden urge to index 10GB of files.

So *of course* the symptoms will never show up with a fresh install.

[Jamie McCracken (jamiemcc-blueyonder)]

Paul,

After fresh install to a new partition , I added partition of old home to tracker-preferences to index and fresh install makes a huge difference.

So same number of files were being indexed in both cases.

A clean install of tribe 4 or higher resolves the high iowait issues if they were present beforehand in mine and others cases which means the possibility of a bug during the dist-upgrade of kernel (perhaps an old driver is not being updated?)

If you have a free partition to do a fresh install, can you try and see if it makes a difference in your case?

[penalvch (penalvch)]

Tom Badran, this bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue? If so, could you please test for this with the latest development release of Ubuntu? ISO images are available from http://cdimage.ubuntu.com/daily-live/current/ .

If it remains an issue, could you please run the following command in the development release from a Terminal (Applications->Accessories->Terminal), as it will automatically gather and attach updated debug information to this report:

apport-collect -p linux <replace-with-bug-number>

Also, could you please test the latest upstream kernel available (not the daily folder, but the one at the top) following https://wiki.ubuntu.com/KernelMainlineBuilds ? It will allow additional upstream developers to examine the issue. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested. If this bug is fixed in the mainline kernel, please add the following tags:
kernel-fixed-upstream
kernel-fixed-upstream-VERSION-NUMBER

where VERSION-NUMBER is the version number of the kernel you tested. For example:
kernel-fixed-upstream-v3.13-rc4

This can be done by clicking on the yellow circle with a black pencil icon next to the word Tags located at the bottom of the bug description. As well, please remove the tag:
needs-upstream-testing

If the mainline kernel does not fix this bug, please add the following tags:
kernel-bug-exists-upstream
kernel-bug-exists-upstream-VERSION-NUMBER

As well, please remove the tag:
needs-upstream-testing

Once testing of the upstream kernel is complete, please mark this bug's Status as Confirmed. Please let us know your results. Thank you for your understanding.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]