Tracker should not be enabled by default until it doesn't clobber everything

Bug #132741 reported by Jamie Lokier
28
Affects Status Importance Assigned to Milestone
tracker (Ubuntu)
Fix Released
High
Unassigned

Bug Description

Binary package hint: tracker

Related to bug #130935 (indexing takes a very long time, no idea of progress, disk needed or time left), bug #131983 (kills desktop performance while running), and bug #131094 (more on kills desktop performance while running), and bug #130794 (kills battery on laptops),

I think Tracker indexing should be disabled by default, even when running on mains power, until these problems are solved.

I'm running Gutsy and generally keeping up to date. As soon as Tracker was installed automatically by Update Manager, I noticed my desktop was completely unusable. Dragging windows, menus, starting apps, even sometimes the shell prompt, all were unusuably slow. We are taking 10-20 seconds delay for things to respond to the mouse.

This is on a quite capable laptop: Core Duo 2GHz with 1GB RAM, and curiously, not much CPU used, free RAM, and little disk activity shown. I set Indexing Preferences to the slowest setting.

After a while I realised that setting "noatime" in /etc/fstab made a huge difference, but still noticably harming desktop usability. So I do "killall -STOP trackerd" when I need to do some work, and start it again to continue giving it a chance to index.

After a few days, I find it's going to take longer than I hoped. This wouldn't be a problem if it didn't affect performance, but currently it has a big effect so it's a problem.

I also find, after a few days, that my disk is full with Tracker's index, currently 3.1GB for a 72GB home directory index, and that's not a finished index: the disk is full. When I free up some space, Tracker fills it up by doing some more indexing. I have no idea how much space in total will be used.

After every boot, it scans the home directory again of course. With 662,000 files in mine, this is quite a long time and even if it's just stat() to validate its index. (The Tracker folks know that a better kernel mechanism would fix this, and I have discussed it on linux-kernel years ago, alas back when the filesystem maintainers didn't see any point...)

With all these things together, even though it might be a useful feature when it works, currently Tracker is actively harmful on my system and probably many others. Therefore its default settings should be to disable it until explicitly activated by the user, or some other way to ensure the default settings won't break an otherwise working desktop system. Perhaps it could bring up a dialog warning that it's experimental and may harm system performance, and how to disable it, when asking whether to enable it the first time. Or just not install it until requested by the user.

It has been suggested that GNOME apps may come to depend on Tracker. I suggest that for now, the technology isn't ready.

It makes my laptop (and reportedly a few others) nearly unusable, and usable only because I know workarounds. No other update in Dapper, Edgy, Feisty or Gutsy has had such a major effect on usability for me.

Thanks

Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote :

A few corrections:

At startup we scan folders only not files - we need folders in order to place inotify watches on them.

The parent folders mtime alwyas changes whenever a file underneath it changes so we dont need to scan files unless folder has changed, When we have a proper kernel notifcation system we can dispense with the above

Having a massive home folder is not typical of ordinary users and suggests a power user or dev instead who will know how to configure tracker to index only a subset

Revision history for this message
Jamie Lokier (jamie-shareable) wrote : Re: [Bug 132741] Re: Tracker should not be enabled by default until it doesn't clobber everything

Jamie McCracken wrote:
> A few corrections:
>
> At startup we scan folders only not files - we need folders in order to
> place inotify watches on them.
>
> The parent folders mtime alwyas changes whenever a file underneath it
> changes so we dont need to scan files unless folder has changed, When we
> have a proper kernel notifcation system we can dispense with the above

I'm suggesting when you have a proper kernel notification system, then
Tracker will be usable :-)

> Having a massive home folder is not typical of ordinary users and
> suggests a power user or dev instead who will know how to configure
> tracker to index only a subset

I don't buy that argument. Even 'ordinary users' often have disks
which are nearly full, even if they are smaller. So the problem of an
index using up all the disk space after an upgrade will still occur.

And if it's a much smaller home dir, say 20GB, it would still be an
unusable desktop due to the noted performance interactions for the
first day or two of indexing wouldn't it?

Imho, breaking a 'power user' desktop _just_ because someone has
accumulated a lot of files over the years, even though they have a
_bog standard_ Ubuntu install, with no customisations, no dev, no
unusual packages, and possible no knowledge of such things, seems a
bit harsh to me. I know Gutsy isn't bog standard but obviously it
will be when it's released, and it needs to work smoothly by then.

It's common to buy a laptop with 100GB disk now. Are you saying it's
unusual for someone to half-fill such a disk? If so, surely that will
change?

However, if Tracker works perfectly well on smaller home dirs, perhaps
it would be better if the default install checked the size, and
enabled it only if there's a certain percentage of disk space free,
warning with a dialog if not.

-- Jamie

Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote :

Ok 100Gb is no problem if majority is not text files, docs or source code - size matters not in those cases

a non-dev will typically have loads of music and video files which will have negligible footprint on any indexer.

A dev will likely have tons of source code and that will tax any indexer.

With the optimisations I hope to get in in the coming days, I believe even extreme cases like yours can be handled much better.

Im optimistic by using smaller non-fragmenting database for the indexer will greatly lessen the impact on system and disk space but time will tell...

Revision history for this message
Andrew Frank (frank-geoinfo) wrote :

i observe the same - tracker is running with nice 19, but it is still hugging the system such that it is unusable when tracker starts. (my indexing is seeming completely done )- but any start up the system is unusable for a couple of minutes (this is especially when i sit there and want to use it).

hints - can you start tracker only after there was no user activiity for some time?
and stop it immediately if the user does something.
(example: searching for a mail in a mailbox takes 2 minutes (mailbox is not extremely large) - and few seconds after tracker has completed.

IMHO it is not ready for primetime and should not be enabled in the final gutsy version (i vote for delivering, but not for enabling by default: it is a show stopper! as was beagle.... and the same goes for similar systems on windows, fortunately).

good luck with optimization. i do not see a lot of cpu or disk activity, but nevertheless, it hugs the system and does nothing else to be done. why?

(and it does not index mbox mailboxes which are not in evolution - despite showing an interface. i wait for this fix)

Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote :

There is an index delay (default 45 secs) in tracker-preferences - feel free to increase this as well as the cpu throttle

tracker 0.6.2 was a bit buggy (due to our new indexing backend) and tends to eat all memory - we have fixed this and done some more tuning and should have a new release soon

I cant say whether it will be perfect or good enough for you but if it can be fixed in time we will do our best.

Changed in tracker:
importance: Undecided → High
status: New → In Progress
Revision history for this message
Andrew Frank (frank-geoinfo) wrote :

i did not want to be negative in my comment - indexing is very important, but also very difficult.

i tell you my experience yesterday (with version 0.6.2.0)
i was searching a mailbox in balsa - and it did not make progress

i was installing software (apg-get probably) and it seemed to stall such that i restarted (and got into troubles with the lock...)

but i observe that neither memory usage (i have 1 gb) nor disk usage is high. what are the other resources tracker competes with others (or rather hugs).

i look forward to the updates!

(ps. it should be relatively easy to analyse mailboxes (mbox) wherever they are encountered and not specialize for evolution - you leave out users of firefox and of other mail programs)

good luck - i know that this is difficult

andrew

Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote :

balsa and other mboxes are not supported yet - they are planned

iowait times can stall your desktop - use vmstat 1 and look at last column wa. This is the percentage of cpu thats idling waiting for IO - consistently high values 90+ will stall other apps

high iowai could be due to the size of indexable content (can you let me know size of files in $Home/cache/tracker) you have or bugs in tracker/kernel/hardware issues

Revision history for this message
Andrew Frank (frank-geoinfo) wrote :

i have 75 mb in .cache/tracker --

--
suggestion:
for beagle an icon was added in the right down corner of the screen to switch it off and on.
could you do something similar for tracker - and allow user to switch it off immediately when they have the impression it is interfering. and then switch it on again. - the difficulty for the user is due to perception. if the response is slow and you do not know why then tracker is the first suspicion (but need not be). if you give user control then the complains will (perhaps) end, because the issue is one of uncertainty and perception.

Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote :

Andrew,

Its highly unlikely that its tracker's fault with such a small db (if you have > 75mb free RAM then entire db would be cached in memory by the kernel)

see https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/131094

hopefully the kernel devs can fix that as I dont think asking everyone to do a fresh install to cure the problem is reasonable

Revision history for this message
Emilio Pozuelo Monfort (pochu) wrote :

I'm closing this bug report, since this is a development version of Ubuntu, and it's likely to eat your dog or fire your house at some point ;)

Seriously, if something has a bug, the solution isn't to remove it from the distro, but to fix it. And that's what we are going to do, instead of disabling it. Those critical bugs are been working on, even are fixed in svn trunk, and will be fixed in Gutsy later this week, with 0.6.3 release.

Tracker is, and will be enabled by default in Gutsy, hence I'm closing this report.

Changed in tracker:
status: In Progress → Won't Fix
Revision history for this message
Darik Horn (dajhorn) wrote :

> I'm closing this bug report, since this is a development version of Ubuntu, and it's likely to eat your dog or fire your house at some point ;)

Gutsy beta freeze is happening in two days and general release is less than a month away. The system should be stable and working properly for most people.

> Tracker is, and will be enabled by default in Gutsy, hence I'm closing this report.

This is bad logic and it reads like "we are forcing the tracker package into release regardless of whether it is ready".

Revision history for this message
Emilio Pozuelo Monfort (pochu) wrote : Re: [Bug 132741] Re: Tracker should not be enabled by default until it doesn't clobber everything

Darik Horn wrote:
>> I'm closing this bug report, since this is a development version of
> Ubuntu, and it's likely to eat your dog or fire your house at some point
> ;)
>
> Gutsy beta freeze is happening in two days and general release is less
> than a month away. The system should be stable and working properly for
> most people.

And it will be. Tracker critical bugs will be fixed for the beta
release, with the new 0.6.3 release. I haven't said it won't, just that
we will fix them instead of removing tracker, as long as we can.

>
>> Tracker is, and will be enabled by default in Gutsy, hence I'm closing this report.
>
> This is bad logic and it reads like "we are forcing the tracker package
> into release regardless of whether it is ready".

That's not what I wanted to note, but rather "we are shipping tracker
because we can and we are fixing these bugs. If we couldn't, we wouldn't
ship it".

Sorry if I didn't expressed well.

Cheers,
Emilio

Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote :

The tracker team is committed to solving this issue and it is making good progress...

If, at the end of the day, tracker does not provide a good enough experience for the majority of users then it will have to be pulled (and that is true for just about any package in ubuntu)

Revision history for this message
Jamie Lokier (jamie-shareable) wrote :

My objection to the "won't fix" status is that fixing these problems may be difficult. Until we know they are fixed, we don't know if it's possible to fix them in time for the Gutsy general release. It's just wishful thinking to say "this will be in Gutsy final (soon) and the performance issues will definitely be fixed by then (very soon)".

That's the point of bug reports like this: to keep an issue open until it's confirmed fixed. It's not confirmed (in reality) yet... So why is it now "Won't Fix"?

You say it's fixed in SVN, but a lot of work was done last time around which was supposed to fix them, that didn't fix the performance problems on some systems, and it didn't affect developer's systems until others tried it. These performance issues need wider feedback.

Fixing it might require changes to the kernel. Until now, we don't know if there's a problem with 2.6.2x kernel I/O scheduling interacting badly with Tracker.

The recent "if you have lots of non-media files you must be a developer so you can turn it off yourself" does not give me confidence that the problems are taken all that seriously. Especially when turning it off in the GUI doesn't work.

But, on those systems where it's a problem, it is quite severe and in a general release would make Ubuntu look bad.

"We will not enable by default in the distribution until we have made sure it works well for 99% of users" would be much more confidence inspiring.

As would keeping the bug open until it's fixed and there is feedback confirming it, rather than declaring it "won't fix".

Revision history for this message
Jamie Lokier (jamie-shareable) wrote :

Emilio wrote:

<i>"we are shipping tracker because we can and we are fixing these bugs. If we couldn't, we wouldn't ship it"</i>

That's great. If you could post a comment when you are happy with a version in the Gutsy repository, that would be helpful. Those of us who have a problem with the older version have uninstalled it (because it's the only way to turn it off), so we won't notice when the new version is out.

If you post a comment here (or to another bug I'm subscribed to), I'll install it soon after and let you know how it went. I can give quite specific feedback on whether the specific technical points I've reported in other bugs are better.

Thanks.

Revision history for this message
Emilio Pozuelo Monfort (pochu) wrote :

I see your point, and now I think I made a mistake closing it. We can close this bug report as soon as those issues are fixed, and if they aren't, then we should disable tracker, as this bug says.

Sorry for my error, it won't happen again.

Changed in tracker:
status: Won't Fix → Confirmed
Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote :

We will notify you when its ready to confirm if this bug is fixed

To clarify a few issues:

1) we are reverting back to qdbm over sqlite for the index as its much faster at inserts and causes less IO
2) The issue with fsync/sqlite no longer applies
3) we are increasing cache size such that each flush of the cache is done to its own mini-index
4) Once files have all been indexed we merge all these mini indexes into one big index

with the exception of (4) which will only take 10-20 secs or so, no slow down of the system (cpu and or disk io) occurs
+ even large 500mb source directories are indexed in less than half an hour

the kernel sucks at updating a relatively small index of 30mb with both sqlite and qdbm as it involves lots of seeks, reads and writes (it should cache all the disk pages in ram and flush them out in an optimal fashion but it does not appear to do that) hence we are working around the problem by merging small indexes together and avoiding the seeks completley

The above version will be in svn soon and once its got some testing we will submit it to ubuntu

Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote :

We have done lots of optimisations and index merging which should cure this problem

Once tracker 0.6.3 is in gutsy pls retest and reopen bug if problems persist

Changed in tracker:
status: Confirmed → Fix Committed
Revision history for this message
to be removed (liw) wrote :

I've just installed the gutsy beta on a laptop (Fujitsu-Siemens P7120, 1.2 GHz, 1 GiB of RAM), and until I removed tracker, it was unusably slow: dragging a mail from one folder to another in Evolution would routinely take 10-15 seconds, for example. After removing tracker, it's happens in an instant. I'd say the version in gutsy is, therefore, not fixed yet.

It is true that I am a developer, but most of the data on my laptop's disk is images and videos from digital cameras.

Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote :

Lars,

tracker 0.6.3 has just been added to gutsy - pls upgrade and retest and confirm if ok

0.6.3 will automatically reindex for you

It is not in the *beta* version so you must upgrade to get it

Revision history for this message
Tobias Heinemann (theine) wrote :

My system remain painfully sluggish with tracker 0.6.3

Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote :

tobias

did you killall -9 trackerd before running tracker 0.6.3? (old version will still be running)

you would be the first to report sluggishness with 0.6.3 as a significant number have confirmed it does not produce any significant slowdowns

can you get me log file in ~/.local/share/tracker for starters?

Revision history for this message
to be removed (liw) wrote :

I installed the 0.6.3 version and indeed things seem to work better now, as far as resource usage is concerned at least. It spent a couple of hours indexing things, and has now stopped. It doesn't index my e-mails (the setting to do so is turned on), and the stuff that it does index isn't returning enough useful hits for me to be interested in this, and the software is generally uncomfortable to use (search window is not on the window list making navigation hard, enabling plugins makes CPU usage suddenly go up for 5-10 seconds and preventing anything else from working, etc), so I'm going to uninstall tracker again, and unsubscribe from this bug (meaning: if you need further input for me, please contact me directly, outside Launchpad).

Good luck with the software, though, it's on the path of becoming good. I look forward to trying it again for the next Ubuntu release.

Revision history for this message
Xvani (fredrile+launchpad) wrote :

It is still really annoying that I have no way of knowing how long it will hog 50% of my CPU and use a lot of I/O.....

using 0.6.3.......

Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote :

Xvani,

We are just about to release a new version with a notification icon that shows progress, status, stats and allows manual pausing too.

hope that helps as soon as 0.6.4 hits repos...

Revision history for this message
Morpheus (hemingwayernest) wrote :

Hi,

I have a similar problem.

First, I'm new to Ubuntu and I have no technical knowledge whatsoever.

I upgraded to Gutsy from Feisty as soon as it was released and everything was fine but yesterday I got a message at startup that " the disk volume at / is 100% full" . After some research I realized all the 14G of my home directory was full, mainly taken by tracker (10.5G) in the folder cache.

I have no idea when tracker was installed and whether it was enabled in Gutsy release or was installed with updates.

Apart from taking all the space on my drive, it slows down the system and my CPU usage goes to 50 to 60 % ( normally it's between 1 and 5%)

System Configuration:

Dual boot ( Windows XP and Gutsy)
CPU Core Duo 2140 ,1.6 GHz
RAM: 2 G
Hard Disk Size: 160 G
Available disk space: 0 bytes

I tried unmounting my NTFS drives with no effect.

Thanks

Revision history for this message
ubuntu_demon (ubuntu-demon) wrote :

This might be one of the contributors to the infamous Load_Cycle_Count bug (for some people) :
https://bugs.launchpad.net/ubuntu/+source/acpi-support/+bug/17216
https://bugs.launchpad.net/ubuntu/+source/acpi-support/+bug/59695

Revision history for this message
jeremy-list (quick-dudley) wrote :

I've had the same problem with both tracker and beagle and have removed them both. Unfortunately the latest version of Nautilus requires tracker to do any kind of searching. This is really unacceptable: It is totally inadequate to limit searches to a directory small enough for tracker to handle.

Revision history for this message
kdvolder (kdvolder) wrote :

I'm having similar issues, they started after I began having files in my home directory. (Before it was as good as empty and no big problems).

However, besides the performance bug, there's also a "usability" bug in tracker.
I can't use it because what I want most of the time, is simply search for a file with some string IN ITS NAME.

When I run this search through tracker, or the search feature in Nautilus, it gives me too much irrelevant results (presumably the irrelevant ones are files where the string appears somewhere in its contents).

Anyway, I turned off content indexing. I'm hoping this will do the trick.

If I ever wanted to search in the content of files (quite possibly useful I admit) at least it ought to still be possible (and probably) the default to search only based on name.

UPDATE: This didn't do the trick, apparantly, now search doesn't work anymore at all (finds 0 results now for any search)

Even better, give me search tool that lets me specify more precise queries (E.g search files, changed in the last hour within which have a *.log name, or something like that).

Revision history for this message
mojotoad (sisk) wrote :

kdvolder: this is a little off topic, but you might like ack:

http://petdance.com/ack/

Lots of options for whittling down the search list, including regexps on the filenames.

Matt

Revision history for this message
Simone Tolotti (simontol) wrote :

UP!
I'm testing Tracker again on Hardy alpha.
I have about 100GB of data in my home... mixed videos,images,music and text files.
I'm not a developer and I have disabled indexing of ~/Downloads/SRC that is the folder where I keep sofware sources.
I've also removed .cache/tracker before starting indexing.
On first run tracker cpu usage is about 25% and sometimes reaches 50% (core2duo 1.8Ghz) and obviously slowed down my system.
I wonder how an app working this way coul be called "ready for everyday usage"...

It's since tracker was added to feisty repo that many people have this kind of problems...
Do you still think they could be fixed or it's time to look for an alternative?
I think it's hard for non-geek users to disable or remove tracker so don't enable it by default (until these issues are fixed) or use Beagle instead!

Revision history for this message
Jamie Lokier (jamie-shareable) wrote : Re: [Bug 132741] Re: Tracker should not be enabled by default until it doesn't clobber everything

Jamie McCracken wrote:
> trackerd is niced +19 so cpu will nose dive to 1% if another app uses
> cpu. Indexing needs lots of cpu which no indexer can escape from.
> Switching to another will not change this

> (unless you consider slowing indexing to such a crawl that it takes
> days to index which you can do already in tracker-prefs by setting
> the throttle slider)

Does this work now? It had no noticable effect when I tried it, and
certainly didn't stop the I/O overloading problems. At all.

> Only slow downs Im aware of is during index merging and these are 100%
> due to I/O issues

Don't forget scanning files after reboot, just to check they haven't
changed. That takes a few minutes of heavy disk seeking, and is very
annoying after booting a laptop to do something quickly.

> during index merging I/O can become overloaded which can starve other
> processes of disk due to bugs in kernel (lots of small writes kills
> performance on ext3 but hoepfully ext4 will have pre allocation to
> alleviate this)

You shouldn't be doing lots of small writes with fsyncs - that's
application silliness.

I've explained how to avoid them at the db level but as far as I could
tell you weren't interested. To be fair, index merging is a smart (if
suboptimal) alternative - but you _really_ shouldn't have small writes
with fsync when merging indexes. That's algorithms from the 70s, long
ago solved. (See Knuth).

Imho, ext4 preallocation and delayed allocation will not make small
single-file writes with fsyncing much faster, if at all. But I would
like to be pleasantly surprised.

> In any event, next version of tracker (due this week) now pauses
> automatically during merges whenever the user moves the mouse or
> presses a key/button so there should not be any perceptible slow downs.

Oh boy. Talk about krufty hacks! 'Tis a good idea. What about
watching long TV/animations in Totem and other video players. Is that
affected? It would be a good idea to index during that, provided I/O
+ CPU isn't affected.

-- Jamie

Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote :

trackerd is niced +19 so cpu will nose dive to 1% if another app uses cpu. Indexing needs lots of cpu which no indexer can escape from. Switching to another will not change this (unless you consider slowing indexing to such a crawl that it takes days to index which you can do already in tracker-prefs by setting the throttle slider)

Only slow downs Im aware of is during index merging and these are 100% due to I/O issues

during index merging I/O can become overloaded which can starve other processes of disk due to bugs in kernel (lots of small writes kills performance on ext3 but hoepfully ext4 will have pre allocation to alleviate this)

In any event, next version of tracker (due this week) now pauses automatically during merges whenever the user moves the mouse or presses a key/button so there should not be any perceptible slow downs. tracker will be the only indexer AFAIK that has this feature and in addition to other tweaks will have very low impact on system (except when its idle of course).

Revision history for this message
Simone Tolotti (simontol) wrote :

"...In any event, next version of tracker (due this week) now pauses automatically during merges..."
I'm pleased to read about this new feature, I've talked about this here: https://bugs.launchpad.net/ubuntu/+source/tracker/+bug/160646/comments/1
I really hope to see this working in Hardy.

Thanks for you hard work.
Cheers, Simone.

Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote : Re: [Bug 132741] Re: Tracker should not be enabled by default until it doesn't clobber everything
Download full text (3.9 KiB)

On Tue, 2008-02-19 at 17:40 +0000, Jamie Lokier wrote:
> Jamie McCracken wrote:
> > trackerd is niced +19 so cpu will nose dive to 1% if another app uses
> > cpu. Indexing needs lots of cpu which no indexer can escape from.
> > Switching to another will not change this
>
> > (unless you consider slowing indexing to such a crawl that it takes
> > days to index which you can do already in tracker-prefs by setting
> > the throttle slider)
>
> Does this work now? It had no noticable effect when I tried it, and
> certainly didn't stop the I/O overloading problems. At all.

As I said that cuts cpu usage and not I/O ones. Its mainly intended to
stop laptops getting too hot and annoying fans coming on due to
constantly high cpu usage.

>
> > Only slow downs Im aware of is during index merging and these are 100%
> > due to I/O issues
>
> Don't forget scanning files after reboot, just to check they haven't
> changed. That takes a few minutes of heavy disk seeking, and is very
> annoying after booting a laptop to do something quickly.

the scan is done only on directories in order to place watches on them.
until the kernel supplies us with recursive folder watching all indexers
have to do this. We also check to see if the directory mtime has changed
but that adds no overhead to above. I have a cunning plan to solve this
but it must wait for another day...

>
> > during index merging I/O can become overloaded which can starve other
> > processes of disk due to bugs in kernel (lots of small writes kills
> > performance on ext3 but hoepfully ext4 will have pre allocation to
> > alleviate this)
>
> You shouldn't be doing lots of small writes with fsyncs - that's
> application silliness.

we dont

we batch 5000 small writes then fsync. if perform fast merges is enable
din tracker-prefs then we dont do any fsync

>
> I've explained how to avoid them at the db level but as far as I could
> tell you weren't interested. To be fair, index merging is a smart (if
> suboptimal) alternative - but you _really_ shouldn't have small writes
> with fsync when merging indexes. That's algorithms from the 70s, long
> ago solved. (See Knuth).

sure but that logic must be applied to sqlite/qdbm or other indexer
backend. Theres nothing we can do in tracker source. If you or someone
else writes us a smart indexer backend we would be happy to use it.
(indexer backend independence will be in the next major release of
tracker 0.7)

>
> Imho, ext4 preallocation and delayed allocation will not make small
> single-file writes with fsyncing much faster, if at all. But I would
> like to be pleasantly surprised.
>

wee you can see today - mount ~/.cache/tracker on XFS (which has said
pre allocation) and turn fast merges on in tracker-prefs. Index merges
are about 100x faster and cause no perceptible I/O overload AFAICT.

pre allocation is a highly desirable feature for us.Also note that ext3
is the slowest to write to so some of above overhead might be due to
other reasons specific to ext3.

> > In any event, next version of tracker (due this week) now pauses
> > automatically during merges whenever the user moves the mouse or
> > presses a key/button so there sho...

Read more...

Revision history for this message
Jamie McCracken (jamiemcc-blueyonder) wrote :

Correction:

I meant "Allocate on flush" instead of pre-allocation

http://en.wikipedia.org/wiki/Allocate-on-flush

This stuff really rocks with indexers!

Revision history for this message
Nicola Gallo (nicola-ga) wrote :

i can confirm this bug with version 0.6.6 and ubuntu 8.04 updated. When I disconnect the cable the cpu keep indexing, even if the settings for index-stop on battery state are marked.

Revision history for this message
Emilio Pozuelo Monfort (pochu) wrote :

Tracker isn't enabled by default in Hardy. This decision was made by the Technical Board and will be reconsidered in the future.

Changed in tracker:
status: Fix Committed → Fix Released
Revision history for this message
zac (zac-wheeler) wrote :

Uh, it's *not* enabled by default? Is it carried over from Gutsy then? I never had an issue with it until I dist-upgraded from Gutsy to Hardy, but I sure didn't intentionally enable it. I have removed it after finding this.

My problem: not a huge amount of code/text files in my home directory (at least not from my perspective), but symlinks to code outside $HOME because I'm lazy and like to use the tilda when cd-ing around. Does tracker follow symlinks that lead below $HOME?

There is no way it should default to the user experience it does. I would rather it not finish for six months than ever being forced to notice its IO (block/us)age.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.