[gutsy] trackerd kills disk io

[Oleksij Rempel]

Binary package hint: tracker

after migration from fiesty to gutsy new application trackerd kill all disk io with no respekt to user aktions. Only way is to killall trackerd.

[Jamie McCracken (jamiemcc-blueyonder)]

when I migrated to gutsy same issue occurred. A clean install of gutsy tribe 4 seemed to fix this (upgrade did not and I indexed the same files as feisty)

can you confirm if a clean install solves your problem?

[Jamie Lokier (jamie-shareable)]

(I'm not the original poster. I'm not going to do a clean install; I use my laptop for work and don't have time to reinstall from scratch).

I'm running Gutsy, and generally keeping up with updates. As soon as trackerd was installed, I noticed my system became unusable.

My laptop is a Core Duo 2GHz with 1GB RAM. Kernels 2.6.22-{7,8,9}.

The only way I can do any work is killall -STOP trackerd. I've got it set in "Indexing Preferences" to the slowest setting. I have free RAM, and the CPU is not particularly busy. The disk activity monitoring applet shows occasional but not constant activity, but the disk LED is constantly on.

Yet, the desktop is sometimes unusable, and always affected, while trackerd is running. Dragging a window may take up to 10 seconds before it moves. (I'm _not_ running the 3d window effects, by the way). Menus can take a second or two before appearing. Apps take 5-20 seconds to start, when before they'd be a second or two.

It feels exactly as though the system was swapping heavily, but it isn't swapping. Only 42MB of swap space is used, and that's probably due to entirely idle pages.

It's weird that this happens with free RAM, little CPU used, and little disk activity according to the applet - and not a lot of seeking sound from the disk, either, although the disk LED is always on.

I am guessing a kernel bug or scalability problem is involved. Something to do with VM or I/O scheduling. Perhaps made worse by the way trackerd spawns lots and lots of large processes, and the way they don't use O_NOATIME.

I have set "noatime" in the mount flags now. This improves desktop responsiveness enormously, but it is still noticably laggy, so I still do "killall -STOP trackerd" when I want to get some work done or start a new app.

It's taken about 4 days now, and still hasn't finished clobbering my laptop. Shouldn't it have stopped indexing by now?

[Jamie McCracken (jamiemcc-blueyonder)]

Please make use of the No Watch folder setting in tracker-preferences to omit path roots (roots include sub dirs)

If you have a massive home folder then it makes sense to omit stuff that does not need indexing

If you are a developer then pls make use of this for source code. If you want source code indexed then I suggest using the crawl folders option

I have noticed slow downs when tracker is not running when upgrading (try any heavy disk activity like rhythmbox looking for music files) and it slows desktop. A clean install has removed this problem and I dont know why?

We are changing our index format to be smaller and more compact which should lessen disk seek activity on it (current qdbm hashtable fragments badly during initial indexing and can consume way too much disk space - we optimise this only after initial indexing is done but that is not sufficient for your case and others and we will remedy this)

O_DIRECT and O_NOATIME will be used in the next version.

We also plan on using index merging which should lessen the effect of updating a huge index (which will be very slow to update if its bigger than available free RAM)

[Oleksij Rempel (olerem)]

i don't know if there is othe way to chack package, so i made this way:

for i in $( dpkg -L tracker ); do
          if [ -f $i ]; then md5sum $i >> tracker_check;
       fi;
done

I don't think cleen install is a solution. Did you mean system clean install or only home/.clean

[Jamie McCracken (jamiemcc-blueyonder)]

I mean I had tribe 3 installed and problem was present. It was still present when upgrading to present.

A clean install of tribe 4 seemed to fix the issue where any signiifcant disk io slowed the desktop

I tried both package and compiled from source and they were the same in both cases

[Jürgen Kreileder (jk)]

Maybe tracker should use io scheduling class 'Idle' instead of 'Best effort (prio 7)'. See src/trackerd/tracker-ioprio.c.
At least "ionice -c 3 -p <trackerd pid>" seems to help quite a bit.

Beagle worked much nicer on my systems, it uses 'Idle'!

[Jamie McCracken (jamiemcc-blueyonder)]

We now do that

on older kernels you needed root permission to set idle class - that seems to have been relaxed on newer ones like in gutsy

we now attempt idle class first then set best effort 7 if that fails

[Gonzhauser (gonzhauser)]

Git (http://git.or.cz/) is slowed down by a factor of ten to hundred when trackerd is running.
I removed tracker completely from my system because working was not
possible anymore.

[Jamie McCracken (jamiemcc-blueyonder)]

please use the no watch directories settings in tracker-preferences to tell it to ignore your checked out source directories

[Oleksij Rempel (olerem)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> please use the no watch directories settings in tracker-preferences to
> tell it to ignore your checked out source directories

I didn't heard so match stupid thing like this: "We will install you
tracker but it make some problems for you, so it will be best if just
disable watching of files you would have to be indexed and there well be
no bug any more." :-0 .... It is not responsible to put this software to
the Gutsy... It will be the same mistake like with bug #85488. Now i see
do not interest in fixing and testing of this bug so the best thing i
can do is remove tracker. Suddenly beagle do not working on gutsy now
and tracker seems like not gutsy ready at all.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG0YPwVVCoNUmKuAcRAlWQAKCb/FpBM70ivnJKOPf39fwtTv7zdwCgm5xv
llMCMDxX3PcT3f9DMqmg2vk=
=Cg9J
-----END PGP SIGNATURE-----

[Jamie McCracken (jamiemcc-blueyonder)]

We are working to improve it substantially but...

*any* indexer which does real time watching will slow down devs with source code when compiling or checking out- thats why we created the options for them to omit them from being indexed or crawled only at start up. We can not eliminate the overhead entirely unless they use the options we provide

note that this issue is for devs with loads of source code - most ordinary users will not experience them.

Fortunately devs are smart enough to configure tracker to suit their needs...

[Oleksij Rempel (olerem)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

*Any* user friendly indexing software should be smart enough to start
indexing if there is *no* user activities. What do you think normal user
do if it jast start the laptop to check emails bat can't just because
tracker kill all *IO* not a CPU.

I'm not using source in my /home/ at all, but i have a lot of email and
other 50G of *.odf *.ogg *.jpg ... what should be indexed, so you try to
explain me i'm not "smart enough" "dev" to just black list may /home/path.

OK. May be i am. So i'm just stupid user who can't adapt own life to one
 rely smart tracker. You mean i need to wait about 15 min after ubuntu
start to check emails or start firefox...

> note that this issue is for devs with loads of source code - most
> ordinary users will not experience them.

how many "ordinary users" use gutsy now?! all user i have in my network
use feisty and just little sad if ubuntu.com trying new *beta* software
on them... not really better what microsoft do. So i trying my best and
 trying test this software as normal user or users i can and i can say,
*all* this normal user will have the same issue i have.

There is no sense to continue this discussion... i'm happy to hear you
working to improve it. So i thank you for your work but don't ever try
me to explain how "smart" am i, i can explain you about you the same.

Jamie McCracken schrieb:
> We are working to improve it substantially but...
>
> *any* indexer which does real time watching will slow down devs with
> source code when compiling or checking out- thats why we created the
> options for them to omit them from being indexed or crawled only at
> start up. We can not eliminate the overhead entirely unless they use the
> options we provide
>
> note that this issue is for devs with loads of source code - most
> ordinary users will not experience them.
>
> Fortunately devs are smart enough to configure tracker to suit their
> needs...
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG0ZouVVCoNUmKuAcRAt9TAKCbfTCHRy4uXM+O+sj1EbaEWvqcpgCgu2Yt
nxLB4mOt1jBaBEDp0JHuQjk=
=EUlT
-----END PGP SIGNATURE-----

[Martijn vdS (martijn)]

I have to agree with fishor.. either improve trackerd so it doesn't suck the load up to 5 in ANY use case, or disable it by default.

I didn't know tracker was installed before I saw trackerd running, and my first action was "What is this? KILL".. Yes, I'm a developer, and I know how to configure my software, but I don't expect software I didn't install myself (afaik) to break my system when I leave it unconfigured.

Maybe it's an option to disable indexing/watching of all known "developer files" (.py, .c, .pl, .o, to name a few.)?

[Jamie McCracken (jamiemcc-blueyonder)]

We have substantially improved iowait times and prevent slowing down other apps with disk i/o

there may still be cases where tracker is harming disk i/o so please let us know by reopening this bug if that is the case

[Sebastien Bacher (seb128)]

tracker (0.6.2-0ubuntu1) gutsy; urgency=low

  * New upstream version
    - New Sqlite based indexer which utlises the new incremental blob I/O
      in sqlite 3.4
    - Highly optimised email indexing (up to 5x faster)
    - Dramatically reduced disk access and disk contention (LP: #131983,#135115)
    - Indexer now pauses for a grace period when non-tracker processes write
      to disk (providing changed files are being watched by tracker) -
      minimises slowdowns when compiling or checking out source code
    - Makes use of idle class disk IO scheduling if available
    - Makes preliminary use of NO_ATIME (some disk access still uses fopen)
    - Fixed restore of user metadata on re-index (keywords are auto-restored)
    - Added increased number of (junk) files to automatically ignore
    - Improved stopwords
    - New deskbar handler that uses the new deskbar api (2.19+)
    - Fixed old deskbar handler to remove race condition causing crashes
    - Fixed a number of annoying bugs in email indexing and tracker
      preferences (LP: #132921)
  * debian/libdeskbar-tracker.install:
    - updated
  * debian/rules:
    - use auto mode for the tracker configure option

 -- Sebastien Bacher <email address hidden> Wed, 05 Sep 2007 12:58:14 +0200

[Jamie Lokier (jamie-shareable)]

This version is improved, and clearly a lot of work has gone into it. But I still see problems with it affecting the rest of the system.

I deselected "Enable Indexing" and "Enable Watching" in the Indexing Preferences (first tab, General), but it still indexed, even after rebooting. Because of this, I had to ununstall tracker completely, to get my system usable with the old version.

I decided to give this new version a try, as it's clearly had a lot of work done. So I installed it, and this is what I found. I'm using tracker-0.6.2-0ubuntu3:

1. I see it still insists on indexing even though the "Enable Indexing" preference is deselected. This is either a bug or a misleading UI. It means if I don't want indexing, even temporarily, I either have to "killall -9 trackerd" or uninstall the package. It would be much better to be able to disable indexing, and re-enable it when I don't mind the impact.

2. The first time it ran (the new version), my laptop slowed to a crawl after about 5-10 minutes. I thought this might be the old disk I/O problems, but it turned out to be excessive VM usage. trackerd was using 900MB virtual memory, of which 704MB was RSS. I have 1GB RAM total, and a Gnome desktop plus Firefox needs about half of that, and so everything ran very slowly. I think needing 704MB RSS is excessive for an indexer, and probably indicates a bug. I got out of this by killing trackerd (-9).

3. The next it ran was after a power cycle. This time, for a couple of hours it stayed quite small (10M RSS), nice. But it was using 100% CPU, and hardly any I/O according to the Disk Usage monitor. A quick strace shows it doing repeated SQLite commits (and creating and unlinking a temporary log file with each commit). But crucially: it's doing this and no other system calls. This means it's doing a lot of commits, but not indexing anything. It's not reading my filesystem at all. Also, presumably it should never use 100% CPU for a sustained long time, if it's on the maximum throttle setting (which it is).

4. Despite the low I/O activity according to Disk Usage monitor, it's actually I/O bound. What's happening is that every small SQLite write create a log file on /tmp, writes to that, calls fsync, writes the main file, calls fsync on that, then unlinks the log file. Perhaps it isn't obvious: those sequential fsyncs on the main database will be causing tracker to run a lot more slowly than usual, and they also force the disk head to remain close to the filesystem logging area (fsync only has to commit the log, nothing else). I noticed with the earlier version that _this_ is sometimes the cause of "kills disk I/O", not the reading for indexing, not the inotify watching, but the continuous rapid rate of fsync calls on the database. The solution to this is to aggregate many db writes into single transactions, to reduce the fsync rate safely. For an indexing application like this, you can use a timer to decide when enough writes have been gathered and a commit should be done, so that fsyncs are rate limited by time.

So, I still find that I cannot use tracker on my laptop for now. But I have these suggestions, which might make it pos...

[Jamie McCracken (jamiemcc-blueyonder)]

There is still more work going on to improve things (notably index merging which means creating a smaller index when the main index is large and merging them together later on which should reduce iowait time during non-merge periods substantially as a small index can be updated much quicker)

1) is a known bug and will be fixed shortly. A workaround you can use in the meantime is to put a dummy path in the watch directories setting (uncheck index my home directory as well) and disable email indexing in preferences.

see https://bugs.launchpad.net/ubuntu/+source/tracker/+bug/132320

Also we will be adding a pause indexing option to a tracker applet to make it easy to pause when the user desires

2) This is already fixed in svn. RSS stays at around 12-14MB during initial indexing with svn version (with heap around 6-8MB and about 5-6MB shared).

3) I would need more info on this. When does it happen? During watching? (we crawl all the sub dirs in your home dir before indexing to place watches on them). Does this only happen when watching/indexing is disabled in the prefs?

The more info you can give us here will help us solve this problem.

4) The point you raised is how sqlite works. Sqlite has its own journal (on top of the ext3 journal) so whenever a transaction is committed it writes to /tmp journal first. We cant prevent that - its part of sqlites' ACID compliance and general resistance to db corruption. As the index is expendable to us, it is not needed by tracker but the best I can do is request an option from sqlite team to disable journalling in the future. Sqlite recommends stacking up many writes into a single transaction for performance reasons which we do in most instances. We will check for any instances where this is not the case

Note earlier versions used qdbm not sqlite which did not do this. QDBM suffers more from fragmentation causing the index on disk to bloat up and prevent effective caching by the kernel when it gets too large (this would kill disk IO in itself)

5) already fixed in svn

6) http://www.sqlite.org/lockingv3.html

read section Rollback Journal. Sqlite defaults to creating a journal in /tmp but we can specify another path if there is a compelling reason?

anyway thanks for the feedback - the more constructive criticism you can give us the better we can make tracker for you

[Jamie McCracken (jamiemcc-blueyonder)]

has anyone tried to see if booting with elevator=deadline helps the problem?

as suggested by kernel team in https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/131094

[Roland Ronquist (roland-ronquist)]

regardless if you run out of batteries or not, it is, at least in most cases,
stupid to let a background daemon take most of the computers internal
bandwidth. I am running this on my desktop machine and it is almost
useless for several hours after logging into the system. As I sure am
a geek and I know how to kick my way around my system with a few
well deserved kill commands this is certainly not a big issue for me.
But for any of those ordinary people that Ubuntu is supposed to serve,
this is a big put of.

hence this should not be considered a laptop only issue but a
general usability issue for ubuntu regardless if you are running
it in a server room, on a desktop or on the go.

[Pausanias (pausanias)]

I can confirm the behavior reported below by Jamie Lokier. Basically, Tracker seems to be done indexing (none of the permanent files grow by any amount), but it continues to write a temporary file to .cache/tracker and eat up 100% CPU.

3. The next it ran was after a power cycle. This time, for a couple of hours it stayed quite small (10M RSS), nice. But it was using 100% CPU, and hardly any I/O according to the Disk Usage monitor. A quick strace shows it doing repeated SQLite commits (and creating and unlinking a temporary log file with each commit). But crucially: it's doing this and no other system calls. This means it's doing a lot of commits, but not indexing anything. It's not reading my filesystem at all. Also, presumably it should never use 100% CPU for a sustained long time, if it's on the maximum throttle setting (which it is).

[Pausanias (pausanias)]

Unmarking duplicate as this seems to represent real issues with tracker, not just with the kernel i/o.

[Emilio Pozuelo Monfort (pochu)]

This should be hopefully fixed by the end of this week

[Jamie Lokier (jamie-shareable)]

Responding to Jamie McCracken:

Point 4: yes, it's correct that SQLite writes a journal file and two calls to fsync() to maintain ACID properties. In fact, it's necessary just to make sure the database doesn't get corrupted. It serves the same purpose as ext3's journal does for the filesystem itself.

Each fsync() to the disk is a huge latency overhead, and a huge effect on other processes doing any I/O (reads or writes) when there's a constant trickly of database writes to the disk.

To avoid those, the application (i.e. Tracker), not the database, must be changed to aggregate many writes into a single transaction. That's what I mean by not doing as many commits; I mean batching writes into larger transactions. The fsync()s are only done per transaction, not per write.

For some applications you can't batch writes like that, but for Tracker you obviously could.

Point 6: The rollback journal must be in a place where it survives crashes, otherwise it is completely ineffective and the database file can become corrupt. It should be on the same filesystem as the database file. When it's in /tmp, it doesn't protect against crashes and power failures.

To be precise: If trackerd is killed and restarted, the rollback journal in /tmp will be effective because it's still there. After a reboot or power failure, if /tmp is in RAM (as it is nowadays), then the rollback journal will be gone, so the database can be corrupted by rebooting or power failures. Even "clean" reboots, if they kill a running trackerd and it's not able to commit it's last transaction in time, or if trackerd has been manually killed earlier, and then a clean reboot. If /tmp is on disk, a reboot might still clean it, so the journal might as well be in RAM.

In other words, the journal needs to be in the same directory as the database in practice, otherwise it doesn't effectively prevent the database from low level corruption. That's similar to filesystem corruption: garbage structures, not just wrong data in the database.

Of course the journal isn't perfect for the reasons given on that SQLite page: Linux doesn't always do the right thing with fsync(), and neither do some IDE drives. But on most systems with 2.6 kernels it should work.

[Jamie McCracken (jamiemcc-blueyonder)]

Point 6:

rollback journal is in the same directory for all the databases in HOME/.cache/tracker

the journal in tmp is for the cache database which is wiped when tracker starts (its for storing session properties only like what directories are being watched) The cache db is stored in /tmp too as it does not store any permanent data.

we have reoptimised and found a few cases where transactions were not being used for the cache db - this has been rectified. All the other databases now commit at every 100 files indexed or 500 emails indexed intervals (thats meta data and contents).

Please await next version before testing again...

[Avery Fay (averycfay)]

I think this needs to be disabled by default in gutsy unless it really improves over the next month (or whenever gutsy is supposed to be released). I just upgraded from Feisty and my machine is almost completely unusable. ":w" in vim on a small text file took around 45 seconds. This is a really unpleasant first impression of gutsy. I knew enough to find out what was causing the slowdown and remove it, but I think a lot of users will just decide that gutsy is really, really slow.

[Jake Anderson (jake-vapourforge)]

would it be possible for it to have some form of "cooldown" period on file system changes. IE if something unzips a whole bunch of files leave them there for 10 minutes before scanning. I just installed google earth with the shiny new package and my disk went nuts for 10 minutes. The computer still seemed responsive however and you average user probably wouldn't notice thrashing, it also pulled about 90% cpu in this time. have a 15krpm scsi and a quad core though so my experience with responsiveness probably isn't representative.

[Jamie McCracken (jamiemcc-blueyonder)]

We no do index merging which both speeds up indexing massively (250mb of linux source indexed in 8 minutes) while at the same time drastically reducing prolonged disk IO time.

Please wait for 0.6.3 to hit gusty in a few days before testing and reopen bug if problem persists

[Eric S. Raymond (esr-thyrsus)]

I installed Gutsy Beta on a Thinkpad X61 today and noticed slow booting and fairly constant disk activity. Yes, top(1) revealed that it was trackerd both eating the CPU *and* hammering my disk -- I was relieved and not too surprised to find this bug report.

trackerd is a pig. If it can't be made an order of magnitude faster it should be disabled by default or (better) outright removed.

[Martin Pitt (pitti)]

tracker (0.6.3-0ubuntu1) gutsy; urgency=low

  [ Emilio Pozuelo Monfort ]
  * New upstream release (LP: #130794, #131983, #132320, #137352, #138331,
    #139173, #132505, #131559, #131735, #132710, #133246, #137873, #138778.
  * debian/patches/01-version_fix.patch,
    debian/patches/02-getenv.patch,
    - Removed, fixed upstream.
  * debian/patches/03-system_ioprio.patch: not needed anymore, as tracker
    now tries ioprio system syscalls if available.
  * debian/patches/01_from_upstream_fix_stemming.patch:
    - Added, fixes language selection.

  [ Martin Pitt ]
  * debian/control: Promote o3read to a dependency. That way, updates will get
    it, too, and we avoid making it a dependency of ubuntu-desktop. With the
    external dependency we can avoid installing the internal code copy.

 -- Martin Pitt <email address hidden> Fri, 28 Sep 2007 17:45:16 +0200

[Tim Wright (timw)]

Another "me too" post. I just did the beta upgrade from Feisty to Gutsy (beta). This system comprises an AMD Athlon Barton 2500+, 1GB RAM, and "fakeraid" dual 160GB drives. The system is quite unusable with trackerd chewing over 50% of the cpu and obviously hammering the disks to death. Load average is 4-5. I have been trying to install apt-file via synaptic (a pretty simple operation) for the last *ten* minutes. If this isn't made at least two orders of magnitude more efficient, then it must not ship in gutsy.

[Jürgen Kreileder (jk)]

btrace(8) (from package blktrace) shows a storm of trackerd i/o traffic after every login.
I assume it's checking known files and directories for modifications while trackerd wasn't running. This process probably should be slowed down a bit, there's no need to scan everything right away.

[Jamie McCracken (jamiemcc-blueyonder)]

Please note 0.6.3 is now in gutsy - it was not in beta so comments related to beta or versions prior to 0.6.3 are no longer relevant to the issue

0.6.3 of tracker will automatically reindex your stuff (and in a fraction of the time and with considerably less IO)

Pls upgrade to latest tracker, retest and confirm if this is so or if you encounter situations where it still performs badly and kills IO. Feedback from those who have done so far has been very positive on this issue so am assuming its fixed unless I hear otherwise

[Jamie McCracken (jamiemcc-blueyonder)]

Juergen,

we have optimised that part in 0.6.3. We need to crawl to get all directories to place watches on them as well as checking they are up to date (we do not check every file to see if its indexed as the parent directory's mtime changes if a file in it changes)

[Avery Fay (averycfay)]

I just tried the new version and it's definitely much improved although I have to say I'm still not entirely satisfied. The initial scan still takes a long time and still slows down my machine incredibly (apt-get install sysutils took around one minute). This time though the period of slowness only lasted about 1:15:00 instead of more than a day.

Here's a question though: why can't the initial scan just be scheduled with cron for the middle of the night or something? (or look at when updatedb is scheduled for and use that in case a user has changed it)

Basically, I'm concerned because performance still sucks during the initial scan and while it's faster overall, this is going to be the user's first impression of gutsy. I think simply delaying the scan will go a long way.

[Jamie McCracken (jamiemcc-blueyonder)]

avery,

the scan is to get all folders to put inotify watches on them

if you have a huge number of directories (10,000+) then it will take several minutes to get all subdirs

the index delay setting in seconds in the preferences determines when this scan happens so you could set this to a very high value

for the general case I suppose if the number of directories is large then we could sleep a bit whilst retrieving them to give disk IO to other apps.

trackerd runs with disk IO scheduled (best effort 7) so it should give priority to other apps but perhaps that does not work so well in this case?

[Björn Lindqvist (bjourne)]

The first time I booted gutsy it indexed for almost a day which I guess is to be expected since I have almost 100 gb data. But it is still a very unpleasant experience. A daemon just shouldn't take over your computer like that unless explicitly asked to. Now it "only" spends about 20 mins each time I login to update its index. The symptoms are exactly the same as others have described in this thread.

This is on a P4 3.4 ghz with 1 gb ram. I also have an older laptop and upgrading to gutsy just is totally out of the question unless this bug is fixed. Tracker looks really useful but please disable it by default.

[ubuntu_demon (ubuntu-demon)]

This might be one of the contributors to the famous Load_Cycle_Count bug :
https://bugs.launchpad.net/ubuntu/+source/acpi-support/+bug/17216
https://bugs.launchpad.net/ubuntu/+source/acpi-support/+bug/59695

A related tracker bug for laptop users:
https://bugs.launchpad.net/ubuntu/+source/tracker/+bug/160468

[timduk (timduk)]

As a first time Ubuntu user/installer/developer (7.10 64-bit gusty goatee or suchlike), here's my 2-cents worth.
(trackerd version 0.6.3 on Pentium 3.4 dual with 2Gb ram)

Almost everything has been impressive, apart from some very irritating periodic system freezes and an unresponsive gui. These almost made me give up on Ubuntu.

I have spent the best part of 2 days trying to get dual-head *something" working other than looking at cloned versions of a desktop. Whenever logging back in to see what the latest changes have done, something has kicked in and caused everything and anything to do with windowing to almost freeze, giving the impression that there's a problem in the graphics changes that have just been made.

I discovered trackerd and it's cpu hogging by accident, having given up on the dual head thing and moved on to installing netbeans, in order to get some proper work done. When netbeans started to freeze, I had a look at cpu usage and discovered the culprit. A google for "Ubuntu trackerd" revealed this and another thread.

This utility, in it's present state, blows big-time.
What on earth are you guys doing releasing this on an unsuspecting public? (ie. me)

I do not want, or expect, any utility to hog my cpu to such an extent that gui work is almost impossible, or the cooling fan is screaming non-stop when the system is otherwise idle, no matter how good it is supposed to be. Period.

Needless to say, the best advice received from this thread has been "uninstall" - done.

[gcc (chris+ubuntu-qwirx)]

I agree, trackerd should NOT be enabled by default. However nice it might be in theory, right now it's much too buggy and hogs too much memory and disk.

Frankly I'm amazed that it got into a default install of Gutsy in this state. Jamie McCracken, do you not care about all the users who complain about it? Would you not rather disable it by default until it's stable and people are happy with it, than breaking the Ubuntu experience for millions of innocent users and giving trackerd a VERY bad name? (search Google for trackerd if you don't believe me)

I don't think Ubuntu releases are a good place to beta-test new software on anyone, especially not if it's been tested in testing and people have asked for it to be removed from the final release and those requests have been ignored.

On our first Gutsy system I discovered trackerd eating 100% cpu... by sitting in a loop of continuous SIGSEGV. Catching SIGSEGV is usually a bad idea, and then reinstalling the signal handler is even worse. Please fix that.

[pid 13171] --- SIGSEGV (Segmentation fault) @ 0 (0) ---
[pid 13171] sigreturn() = ? (mask now [])
[pid 13171] --- SIGSEGV (Segmentation fault) @ 0 (0) ---
[pid 13171] sigreturn() = ? (mask now [])
[pid 13171] --- SIGSEGV (Segmentation fault) @ 0 (0) ---
[pid 13171] sigreturn() = ? (mask now [])
[pid 13171] --- SIGSEGV (Segmentation fault) @ 0 (0) ---
[pid 13171] sigreturn() = ? (mask now [])
[pid 13171] --- SIGSEGV (Segmentation fault) @ 0 (0) ---
[pid 13171] sigreturn( <unfinished ...>

[Jamie McCracken (jamiemcc-blueyonder)]

gcc,

it would not hurt to be more constructive with your criticism

tracker works well enough for most users

In some corner cases it does not - that is to be expected. We wont know what these corner cases are unless tracker is put into the wild and is used by lots of people

We have a new release coming that fixes most of the serious problems so far

Can you please run trackerd in gdb and get us a backtrace of the problem - that would go a long way to help us fix your problem.

[Pausanias (pausanias)]

Well, first of all there are two separate issues---the 100% CPU issue and the disk hogging issue. For more on the disk hogging issue see https://bugs.launchpad.net/bugs/155244 and its duplicates.

Since 0.6.3, the 100% CPU issue has been gone for me, though I have a relatively fast system, so if throttle is set to 0 it might still be an issue for some. However, the disk hogging issue is a major one, and since it silently fills up disk space at a slow rate, it might quietly be affecting a lot of users.

Jamie, for every one of us who reports here, there are likely hundreds that experience the same problems but don't know it or don't know how to report it. Frankly, before tracker incorporated sqlite, it seemed fairly stable... most of the problems seem to be related to the new sqlite backend.

I believe that whenever the fixes you are discussing are ready, they should be pushed into the critical systems update queue in gutsy (rather than waiting for hardy).

And I agree that catching a SIGSEGV is a bad idea.

[Pausanias (pausanias)]

I also wanted to add that I think tracker is a great program and regularly returns better and more relevent search results than its competitors, Spotlight included. This is why I support keeping it in gutsy and future releases.

[Eric S. Raymond (esr-thyrsus)]

gcc <email address hidden>:
> On our first Gutsy system I discovered trackerd eating 100% cpu... by
> sitting in a loop of continuous SIGSEGV. Catching SIGSEGV is usually a
> bad idea, and then reinstalling the signal handler is even worse. Please
> fix that.

I had the same experience. trackerd is a nasty hog, and it's a pain
having to kill it every time I boot. Even turning off every coverage
option I can find doesn't stop it from overwhelming my CPU -- and the
fact that I can't find whatever way there may be to disable it is
itself a bug.

Yes, this is under 7.10 kept current.
--
  <a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

[Jamie McCracken (jamiemcc-blueyonder)]

Eric,

You can turn it off permanently by either:

1) removing it from gnome-session (which starts it on boot)

2) via tracker-preferences you can disable watching and disable indexing

[gcc (chris+ubuntu-qwirx)]

Jamie McCracken,

I think my criticism was constructive. Please read it again if you doubt that I offered positive suggestions for each criticism that I had. Please bear in mind that I'm seriously annoyed that this software is enabled by default in Gutsy and is causing problems for many users including myself. Apologies if I caused any offence.

I develop and maintain OSS as well, and if any of my projects were lucky enough to get into Ubuntu and to be enabled by default, and users reported problems with them that made their systems almost unusable, I'd be the first to pull it or disable it.

I don't even want desktop search. I'm happy with grep, thank you. I didn't ask for tracker and I sure wasn't expecting it to (a) be installed and running, (b) sit in a loop eating cpu or (c) try to fill up my hard disk without even asking me.

Since according to comments above it is not even possible to disable it user by user in preferences, and anyway we run a multi-user system where we don't want to do this for each user individually, and I don't even want it, I just removed the package entirely. I'm afraid I don't have time or interest to help you to debug it right now. If I'd chosen to install it and then found it to be buggy it would be quite a different matter.

[Jamie McCracken (jamiemcc-blueyonder)]

gcc,

thanks but for your info, none of the issues which are currently affecting tracker could be replicated by myself, those testing tracker on our ML nor the ubuntu devs who put it through its paces and would have pulled it if it had serious problems.

so 0.6.3 was tested and approved for gutsy final by ubuntu and I had no info that it caused any of the problems you describe at the time. I honestly felt tracker would deliver the goods for everyone and I would not recklessly release something that would have knowingly caused disastrous results.

Alas, as I stated before, there are corner cases in all software which might cause problems but these can only be discovered by having it installed on thousands of machines and being in routine use. So its a chicken and egg situation.

rest assured, we are working very hard to fix these issues and even if you dont want desktop search tracker will still be useful for its metadata capability in hardy heron. Tracker will also be a good replacement for updatedb and locate and you can configure it to for example not index the contents of files to make it more lightweight

jamie

[Tim Wright (timw)]

As somebody who commented earlier about the issues with trackerd Gutsy Beta, I feel it only fair that I should update and point out that, for me at least, the final shipped version of trackerd is quite usable on both my desktop system (not ancient but hardly current), and my laptop. So, whilst I cannot speak for others, or claim that tracker is perfect, it is entirely fair to say that a lot of issues were addressed, and that for a lot of people, if not the vast majority, it is quite usable at the moment.

I will emphasize again, please don't take this as me saying "oh trackerd is perfect and people are just whining". I am not. But I felt it only fair to point out that it has improved to the point where it is "mostly harmless" :-)

[timduk (timduk)]

Re: chicken and egg..

It would appear that the underlying problem could be revealed quite easily. I would hazard a guess that we few "corner cases" have perhaps something in common - we are installing Ubuntu on already densely populated systems?

In my case, I suspect the combination of auto detection and mounting of my MS partitions (with 3.5 million files) and trackerd's attempt to index them.

Is that a fair assessment? Would trackerd attempt to do that?

If so, then it is easily replicated... create a Windows base system (XP, Vista, WIN2000 or whatever) with a few thousand dummy directories, and fill each directory with a few hundred randomly generated text files, THEN install Ubuntu and see what happens.

Is it reasonable to suggest that your testing is done on clean installations?

[Stefan Wasilewski (swasilewski)]

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23010 smw 34 19 1672m 1.6g 436 S 1.0 81.4 35:12.77 trackerd

This seems... unreasonable. I don't care how many files or directories I have on my system.

[Sam Chodur Jr. (samuelchodur)]

timduk ...

I just upgraded and I have been experiencing slow downs with the only app I am running. Will a clean install fix things? I saw the 'trackerd' process running.

[gcc (chris+ubuntu-qwirx)]

Sam: unlikely, this is not Windows. You need to find out what's causing it. Run "top" and see what the top process is and kill it.

[Emilio Pozuelo Monfort (pochu)]

Sam, did you upgraded to what release?

[Sam Chodur Jr. (samuelchodur)]

I just upgraded to 7.10 ..

I may try a fresh install.. and if that doesn't help things, i am going to downgrade to 7.03

[Emilio Pozuelo Monfort (pochu)]

Sam, could you try with 0.6.4 from https://launchpad.net/~pochu/+archive ? And report feedback in bug 175676, so we know whether this fixes your issue or not, in order to backport it.

Thanks in advance!

[Kevin Ross (kevinro)]

I just installed ubuntu 7.10 a couple days ago. trackerd is indeed consuming a tremendous amount of cpu time:

$ uptime
 09:35:28 up 2 days, 18:42, 5 users, load average: 3.13, 3.15, 3.14

$ ps -C trackerd
  PID TTY TIME CMD
23057 ? 1-01:34:56 trackerd

Any explicit advice on how to remove this would be appreciated. Assume I know very little about how to disable this thing. What detailed steps are needed?

Thanks
Kevin

[Kevin Ross (kevinro)]

In case that wasn't obvious, in 2 days 18 hours trackerd has used 25:35:00 of CPU time, which is bad!

[Emilio Pozuelo Monfort (pochu)]

Kevin Ross wrote:
> I just installed ubuntu 7.10 a couple days ago. trackerd is indeed
> consuming a tremendous amount of cpu time:
>
> $ uptime
> 09:35:28 up 2 days, 18:42, 5 users, load average: 3.13, 3.15, 3.14
>
> $ ps -C trackerd
> PID TTY TIME CMD
> 23057 ? 1-01:34:56 trackerd

That could be caused by the initial index. Have you rebooted since trackerd did
its initial index?

>
> Any explicit advice on how to remove this would be appreciated. Assume I
> know very little about how to disable this thing. What detailed steps
> are needed?

I'd appreciate if you could test tracker 0.6.4 from
https://launchpad.net/~pochu/+archive first. If you still want to remove it, you
can do it from System>Preferences>Sessions.

Cheers

>
> Thanks
> Kevin
>

[Mika Fischer (zoop)]

What's the status of this? I also have major problems with trackerd bringing the system to its knees.

I also get:
Could not set idle IO priority...attempting best effort 7 priority

Anybody know why this is? Doing "sudo ionice -c 3 -p <trackerd-pid>" really seems to help at first sight...

[Martin Emrich (emme)]

Actually, idle priority is not allowed for non-root processes. So your sudo... line works, but tracker (running as non-root) cannot ionice itself to idle. There's some good explaination on the web why idle is only allowed for root (IIRC an idle I/O-process could starve a depending realtime process to death or something like that).

For now, I for my part have uninstalled tracker, as it reindexed my whole $HOME (with both some big and many small files) again and again. But i'll stay tuned and retry it when the next minor release hits the repositories.

Ciao

Martin

[Brian Takita (brian-takita)]

Its still a critical issue on Hardy Heron. I uninstalled trackerd because its pretty much renders my development environment unusable.

[Peter Ballinger (peter-j-ballinger)]

Absolutely still an issue in my system upgraded from Gutsy to Hardy (and tracker 0.6.6). I have no large directory of source that I'm aware of it indexing, but system IOwait spikes and renders the accessing program unusable and unresponsive on almost any disk access, for 10 seconds to upwards of minutes, unless trackerd simply is not running. (and the only way i've found to prevent it running is uninstalling it. removing it from the session does nothing.)

[Peter Ballinger (peter-j-ballinger)]

I have to recant my previous post; my problem may not be directly related to tracker after all. The problem disappeared for several days after shutting down and removing tracker, but seems to have come back, so I can't say it's tracker's fault. I suspect it was contributing to the issue, but it doesn't appear to be the cause.

[Slight Slightly (slight--deactivatedaccount)]

Still a problem on Hardy for me. I've just uninstalled tracker as the IO it was generating was causing FF to become unresponsive regularly and slowing everything else down too.

[craigfisk (friskyfrog)]

Just used Synaptic to uninstall tracker from 1 of 4 systems that were upgraded from Feisty to Hardy. The system monitor showed it taking up to 100% of CPU, overheating the processor and making a bios temperature alarm go off from time to time. It's a home system. One of the other non-problem systems is, I think, completely identical hardware, is at the office, and should have largely the same software and data on it. I found tracker on it only using negligible CPU.