Track “Unwanted Search Results”

[Charles M.]

I’d like to propose a feature for DC++ that would make working with search results easier. Basically the idea is to keep track of files which have already been downloaded, or that have been marked by the user as “unwanted”, and when searches return such unwanted files, display them in a visually distinctive manner, or optionally not at all. It would be similar to the way email clients display emails as unread or read. It would be easy to pick out the search results you’ve never seen before from the ones you’ve seen again and again.

This would help when you end up repeatedly search for the same or similarly named items, such as:
- searching for files which are parts of a series, especially ones being released over time, where you rerun the same search periodically.
- searching for general categories of files, such as “churchill speech” or “quantum mechanics pdf”.
- searching for files in topic areas where wildly different file names are used for the same content, making it difficult to realize you’ve already downloaded or excluded a TTH based on another filename you’ve seen it shared under. (In this case, if you are now sharing the file its search result can already be excluded using the “don’t show items in my share option”. However, you may not share some files you’ve downloaded because they were lousy or you’ve run out of space and moved them off to optical discs, say.)

I would generally refer to this as unwanted search results and it would basically be a stored list of unique TTHs. A TTH would become an unwanted search result in these situations:
- A file with that TTH is successfully downloaded.
- In the search dialog the user right click on a search result for that TTH and selected the “mark as unwanted” option.
- A file with that TTH is added to the user’s shared files list. (This would get the files you didn’t download through DC++.)
- An additional “Add File(s) to Unwanted Search Results” would allow the user to select a set of local files or folders to hash and add to the unwanted search results. This would let a user scan items they didn’t think want to share.

The TTH of each unwanted search result would be stored in a file, presumable an XML file similar to the Queue.xml file. When a search was performed, results that match any of the TTHs in the saved unwanted search results list would be displayed along with the other search matches, but in light gray text, like a disabled menu item. An option to not display unwanted results at all would also be available. If the user attempts to download search results including unwanted search results, a dialog box would inform the user that some/all of the results selected for download were marked as unwanted. The user would be asked to choose to either not download the unwanted results or to download them anyway.

The stored list of unwanted search results would need to have some management facilities. I think the ideal would be to allow the user to open their unwanted list in a similar way that the “Own Files” list or the Download Queue can be opened. The unwanted results could be organized in a hierarchical folder system (obviously not corresponding to any particular filesystem folders, just displayed within DC++.) The user should be able to create folders within the unwanted search result list, move folders and unwanted results into folders, delete unwanted results or folders, rename unwanted results or folders.

Along with the TTH and the filename, the size of the unwanted result and date/time it was marked unwanted, and reason marked unwanted (Downloaded, Shared, or Marked by User) should be saved. Users should be able to sort the unwanted results in a folder by any of this metadata. I would use such management features to organize the unwanted results based on the different domains the search results were coming from and purge entire domains when they were no longer of interest. Also it would help users examine the unwanted results to determine if they may have erroneously marked them as unwanted. The ADL search could also be applied to let users search for items.

A couple of years ago, I implemented a primitive version of a similar scheme for saving unwanted results for a different file-sharing client on another network. It was just a set of perl script that built a hash table indexed by the unwanted file hashes, then brute force scanned the client’s binary temp files looking for hash matches, and deleted the temp file if it hit a match. Over about 3 years, it built up to over 22,000 files, but despite that volume performance and resource use was no problem. I found it was a real help in navigating the chaotic filenaming that was typical of that network. DC++ shares are usually better organized than that particular network, but there’s still a lot of scope for this kind of feature to help people avoid downloading duplicate files and just making dealing with searches easier.

[Neil Harding (8-launchpad-nharding-co-uk)]

I am adding the concept of offline storage to DC++ (so that you can store which files you have on backup storage), and an additional virtual directory called Ignored, where you would right click on the search result and select "Ignore this file in future searches". I am also adding the ability to search your own files since offline files will not appear in your filelist. I want it so that if you do a search for a file, and do a TTH search it would say file is on DVD 037, DVD 182. And I also want a search own files only page as well.

[Charles M. (charlesmalias-launchpad)]

Neil,

Sounds excellent and a clever amplification of the idea. (Though I guess in my scheme you could load up your DVD resident files into the unwanted search results and organize them in virtual folders such as /dvd/037/, etc.) Usually I make my grandiose suggestions like this and am rewarded only with thunderous silence, so I'm glad I'm not alone in thinking along these line. Even better if you're the kind of person who actually does something about their ideas instead of just banging on about them like me. Heh.

Do you think you might incorporate my suggestions of having the option to display the unwanted/ignored search results in a visibly distinctive way, say light gray text, rather than hiding them entirely necessarily? I didn't think of it at the time I wrote my magnum opus here, but that might be a good way to deal with the ignored/unwanted files when you are looking through someone else's file list. That way you could still see, in total, what they were sharing which can be helpful in understanding the way they've organized their files, but at the same time discriminate between the stuff you already had or knew you didn't want, versus new/unknown files.

Do you intend that offlined files would be treated as ignored when those same files came up in a search? I think that would be the way to go.

Another thought that occurs to me is that this sort of facility of managing offline files and other "files I have the hash of, but not the file itself" really deserves to be it's own application which applications like DC++ and similar could interface so they don't all have to build it in and users wouldn't have to maintain duplicate data. Maybe someday.

[Neil Harding (8-launchpad-nharding-co-uk)]

I have an additional checkbox, that says Hide files already available offline (the same as Ignore files already in share). I want to make it so that if you are viewing a downloaded filelist it will distinguish files you have already in your share, or offline. When you set a directory to be offline, it appears in your share (to you only) and it grabs the volume name to act as the virtual directory name, so if you insert a DVD Backup001 it would offer Backup001 as the default name for the virtual share.