Transmission should use a temp-dir to exclude from indexing

[Jo-Erlend Schinstad]

Binary package hint: transmission

When you download large video files with Transmission, the speeds might be low, causing the video-files to be changed continually over a long period of time. This causes indexing issues. If Transmission had a temporary and incoming directory, like the mules do, then you could exclude the temporary directory from indexing, and still having downloaded files tracked.

[Jo-Erlend Schinstad (joerlend.schinstad-deactivatedaccount)]

I can see why you'd consider this a wishlist bug, but I'm not sure it should be. Trackerd runs as default on all new Ubuntu installations and other GNU/Linux distros. Transmission is now a default application in Hardy Heron as well and Transmissions features will cause the system to slow down considerably. Load average went from 0.05 to 2.0-3.0 on my system when downloading a 5GB torrent, and would have stayed that way for a week if I hadn't excluded that directory.

When viewing Transmission in isolation, this isn't a problem at all, but when seen in real life on the Ubuntu desktop, it's a big problem. It's easily fixed though, at least as a temporary solution; make Transmission use a standard download folder and configure Tracker to exclude that directory when indexing.

[Murat Gunes (mgunes)]

Doesn't Tracker stop indexing a file once it starts to change too often?

[Jo-Erlend Schinstad (joerlend.schinstad-deactivatedaccount)]

I don't know about that. Sounds like a good idea. However, a torrent often is a set of many files, so the problem might persist in any case. There is also totem-video-thumbnailer. I don't know if that's run seperately or by tracker-extract though, but they both work very hard when downloading videos.

[Wouter Stomp (wouterstomp-deactivatedaccount)]

The next (not yet in ubuntu) version of tracker will stop trying to index a file if it has changed >10 or so times in a certain period of time, so that should solve this bug.

[John Dong (jdong)]

I assume that Transmission opens files it's editing in write (or read-write) mode. Why is tracker nosy enough to be indexing things currently open for writing?

I believe this is not a feature to implement in Transmission because it's nothing but a hackish workaround for a deeper problem: Tracker needs a way to tell it to leave a directory or file alone! This probably should be some sort of standardized dbus inhibit call that beagle and friends can use, too.

To the bug submitter, we're not trying to say this interaction is not serious (in fact it is quite a pain) or that we are not gonna fix it, just that I'm personally not sure if making a separate "downloading" vs "downloads" dir is a good thing for Tranmission overall or just a way to work around a specific bug on a specific distribution's indexer choice.

[Jo-Erlend Schinstad (joerlend.schinstad-deactivatedaccount)]

ok, so let's assume that the user downloads a torrent containing 30 lectures in xvid, each 700MB in size. With the improvements in Tracker, he'd only have to index the 700MB files >300 times for nothing? I think it's a little early to talk about that >10 times in a "certain amount of time" feature as a complete fix for this.

I don't know if Transmission keeps all files for all torrents open all the time while downloading. Tracker has a way to leave your files and directories alone. That's not a problem at all. But then you're saying, "I don't want you to index or pay any attention to thiese files and folders". You do want it to index those files and folders, only, you want them to be indexed after they've been downloaded completely. I don't think Tracker should have it's own configuration option for that.

[John Dong (jdong)]

Is there a way Transmission can tell tracker via an API call or command to add or remove files from indexing exclusion on the fly? It already does this for mdutil on OS X, and probably can be done on Ubuntu as an analog if Tracker supports doing this.

[Charles Kerr (charlesk)]

In #ubuntu-devel, jamiemcc said that the trackerd people are working on a new release that detects files that change too often. They're aiming for getting this release in the Feb 14 freeze, so IMO this ticket will be moot in a week or so.

07:25 < charles_> jamiemcc: ping
07:25 < jamiemcc> hi charles_
07:25 < charles_> hi
07:25 < charles_> jdong asked me to talk to you about
                  https://bugs.launchpad.net/ubuntu/+source/transmission/+bug/189439
07:26 < ubotu> Launchpad bug 189439 in transmission "Transmission should use a temp-dir to exclude from indexing"
               [Wishlist,Incomplete]
07:26 < charles_> the basic problem is that downloading huge binaries somewhere under the users' home directory
                  seems to mix poorly with indexing :)
07:27 < jamiemcc> charles_: yeah we are working on auto blacklisting files that change frequently
07:27 < charles_> the recommendation in that ticket is to download to a tmp dir instead, but I'm not sure storing
                  downloads in a tmp dir is a good idea :)
07:28 < jamiemcc> charles_: we should be able to detect stuff thats chaniging too often in the next version
07:28 < charles_> cool
07:28 < charles_> how far away is the next version?
07:28 < jamiemcc> charles_: we are aiming for feb 14th - the ubuntu feature freeze
07:28 < charles_> oh, that's great news
07:29 < charles_> I'll annotate the ticket with that information
07:29 < jamiemcc> charles_: ok thx
07:29 < charles_> thanks

[Chris Coulson (chrisccoulson)]

This was fixed in Tracker 0.6.6 in Hardy:

tracker (0.6.6-0ubuntu1) hardy; urgency=low

  * New upstream release.
  * Merge with Debian, remaining Ubuntu changes:
    - debian/control:
      + Addhere to DebianMaintainerField spec.
      + Do not build-depend on universe dependencies:
        libunac1-dev, libqdbm-dev.
      + tracker depends on o3read instead of recommend it, so we have
        OOo indexing by default.
    - debian/rules:
      + Enable sqlite external db instead of qdbm.
    - debian/tracker.py:
      + Added an apport hook to ignore tracker-extract crashes, since they
        are caused by setrlimit terminating tracker-extract when it raises
        it's cpu/memory limit.
    - debian/tracker.install:
      + Install tracker.py in /usr/share/apport/package-hooks/
    - debian/patches/02_no_kde_autostart.patch:
      + Do not autostart trackerd in Kde, as they have strigi.

From the upstream changelog:
temporary blacklisting of frequently changing files (they are indexed after one hour or on next restart of tracker daemon). prevents constant indexing from downloads and torrents

[Jisakiel (jisakiel)]

Nevertheless the suggestion was good. I, for instance, would like to download to a temporary directory in ram (which I have plenty of, free) while my disks are spun down, which would only get online again once the film is complete. That way my server (using laptop-mode) would be most of the time idling on the disks, and would use less power. I know it's sort of a corner case, but I'll end up resorting to scripting and bittorrent just because of transmission not supporting it (as it already has a non-gtk mode).

[Hew (hew)]

The transmission part of this bug is essentially a dupe of bug 183501, and since tracker has already been fixed, I'm closing the transmission task.