File list slow to update

[Ben]

Very often files that are reported in the system logged to have finished hashing do not show up in my file list right away. In particular, any file which finishes hashing after "File list refresh finished" is reported tends not to show up in the list until a second file list refresh is completed, even though the files were detected during the first refresh and already hashed. Thus I must always refresh twice when I wish to share new files, which can take quite a while due to the large number of files and folders that I am sharing.

Operating System: Windows XP Home SP2
DC++ Version: 0.705, (and previously in 0.699)

[MikeJJ (mrmikejj)]

I can confirm this. This was definitely an issue in 0.699, but i've not noticed it recently (0.705). But then again i've not hashed much recently.

Just tested now, refreshed, hashed, and "opened own filelist" about 5 mins after some had finished hashing, and they were in it.
But some files further down the hash list ain't in it yet. (and still aren't 10 mins after they finished hashing).

I got someone to get my filelist and they confirmed that the files i can see in it are the same and they don't see the "missing" ones either. (I believe dc++ generates filelist on request now, hence me asking them to get it, to check,

[MikeJJ (mrmikejj)]

After 30 mins without another refresh these files have now turned up in the filelist. :)

[Pseudonym (404emailnotfound)]

I could be wrong, but I think that the filelist is only updated every 15 minutes unless a list refresh has happened in the intervening time. This is intentional.

Also, I would think that the second refresh would go quite quickly compared to the first, as most of the lag is I/O and the inodes needed will already be in the cache.

[Ben (mimenarrator)]

I am speaking of a delay from the time the system log reports that a particular file has "Finished hashing," and when that file actually appears in the file list as downloaded by others or viewed myself, not any delay for DC++ to actually look for new files, which is not an issue since I refresh manually. What generally happens is as follows. I refresh the file list (Ctrl+E) when the new files are in a shared folder. Soon after DC++ finds the new files and adds them to the queue of files to be hashed. Some of he files finish hashing, and then DC++ reports "File list refresh finished," which means it is done scanning for new files. The remaining new files not yet hashed then finish hashing. If I open my file list, or have someone else download it at this point, they will usually find only the files that finished hashing before the "File list refresh finished" message was displayed when DC++ finished scanning for files. The files which were reported to have been hashed after that point are absent. If I then refresh the file list a second time (Ctrl+E), which is indeed generally faster (except when new files are being downloaded to a shared folder, as is sometimes the case), wait for DC++ to report "File list refresh finished" in the system log, and then open my file list or haver someone else download it, all o fthe files appear, including the previously missing that were files hashed after the first refresh finished. So you see it isn't so much about the exact amount extra time it takes the files to appear in the list, but rather the fact that it can take two refreshes where it should only take one.

[eMTee (realprogger)]

I can also confirm this its a behavior. Its like that for ages I mean I am sure I experienced many times this with 0.674 as well. An addition : sems the larger the share the more time one can experience this...

[poy (poy)]

as Pseudonym wrote above, this is intentional. i don't see anything wrong with it...

[Ben (mimenarrator)]

It is intentional that in many cases one must refresh the file list twice for new files to be added? Why would that be desirable?

[poy (poy)]

one file list refresh to find the file, and put it in the hashing queue; another one once hashing is done. unless hashing the file could take less time than scanning for files (which it sometimes does, which explains why small files are added instantly to the file list), this seems logical to me.
note that to avoid manual refreshes, you can set a lower refresh time in Settings > "Experts only".

[Razzloss (razzloss)]

Actually what I've experienced (with 0.705 LinuxDCPP, with 0.698 you had to refresh twice no matter what) it's exactly the opposite. If refresh finishes before the hashing is done, files aren't found in the list. But if the hashing is done after the refresh files appear "instantly" (when the file list is opened).

--RZ

[Jacek Sieka (arnetheduck)]

marking the list dirty when the last file is hashed would be one possibility (maybe with some check that it's not done too often for small files)

[poy (poy)]

is it not what it already does in ShareManager::on(HashManagerListener::TTHDone, ...)?

[Jacek Sieka (arnetheduck)]

well, maybe it needs some other tweaking then - I agree it's kind of annoying to refresh twice when you add a single file, so something is wrong...

[Ben (mimenarrator)]

I've noticed that in my build of revision 1456 (and also in .707), the behavior now seems to be slightly different. If a file is hashed after the "File list refresh finished" message, it appears in the file list right away, but if it is hashed before the "File list refresh finished" message, it does not appear in the file list until one additional file list refresh is initiated and completed, and yet another file list refresh is initiated. To make sure I'm being clear, here is an example:

[12:38] File list refresh initiated
[12:39] File list refresh finished
[12:40] Finished hashing: ...\Some_File.rar (10.90 MiB/s)
(file appears in list)

[12:38] File list refresh initiated
[12:39] Finished hashing: ...\Some_File.rar (10.90 MiB/s)
[12:40] File list refresh finished
[12:41] File list refresh initiated
[12:42] File list refresh finished
[12:43] File list refresh initiated
(file appears in list)
[12:44] File list refresh finished

[Ben (mimenarrator)]

Oh, and also, if a new directory is added with files inside it, the new directory structure appears in the second example above appears after 12:42, before the files in the directory appear after 12:43, if that helps.

[Ben (mimenarrator)]

I've just discovered that it is actually the time that a particular file begins hashing, rather than when it finishes, that controls whether it appears in the file list immediately or not. I had not noticed this before because most of the files I hash are on the order of 15MB and so there is little difference, but when a larger file begins hashing before "File list refresh finished" and finishes after, it does not appear in the list in the same way that files that finish hashing before "File list refresh finished" do not.

[eMTee (realprogger)]

The following conversation may add some more info for this...

[23:39:08] <eMTee> like the filelist refresh problem which Ive mentioned IIRC
[23:39:38] <eMTee> seems BM managed to solve it along with its side effect at the start of the program
[23:40:17] <poy> never had a problem with hashing on starting the program...
[23:41:19] <eMTee> maybe you never shared many 100's of gigabytes of 5-10Mb sized files in lots of separate directories
[23:43:11] <eMTee> the problem is with the large directory structure. if its not completly cached (by OS) at the start of DC++
[23:44:27] <poy> i do, but i use a DC++ mod based on DC++ 0.401, so hashing is smooth... :)
[23:44:41] <eMTee> checking the shared directory structure seems to run in the main thread at start, so until all of them is read the gui doesn't start
[23:45:14] <eMTee> its not a big problem if you have 300Gb of share, it takes about 10 sec for me
[23:45:22] <poy> DC++ displays a message like "loading file list", when doing that, no?
[23:45:37] <poy> (in the splash screen, before it is loaded.)
[23:45:58] <eMTee> yes but it disappears soon and the gui start is delayed and there is a huge disk activity
[23:46:21] <eMTee> if somebody shares 2TB it takes 1 minute at least for sure
[23:46:40] <eMTee> and I talk about well configured fixed SATA disks, not eg. external usb ones
[23:46:50] <poy> i think it does run into another thread, but the main GUI thread waits for all core thread to have finished loading before showing the main window.
[23:47:07] <eMTee> the effect is the same then
[23:47:34] <poy> it seems to be on purpose, that the main window waits for the core stuff to load up...
[23:47:36] <eMTee> the real problem is when you have this share, stop DC++ and change let's say 20 files
[23:48:08] <eMTee> restart PC or wait some time so the shared directories flushed out of OS cache
[23:48:47] <eMTee> now start DC++ again and the nightmare comes, since while a thread checks the shared directory structure
[23:49:08] <eMTee> another thread begins to hash the newly added files
[23:49:25] <eMTee> if the two operation is in the same drive
[23:50:02] <eMTee> it'll multiply the start by 10x not to say the damage in your precious hdd :)
[23:49:51] <poy> makes sense... so the hashing thread should wait for initialization to finish.
[23:50:23] <eMTee> this is a problem with DC++ SINCE hashing introduced
[23:51:59] <eMTee> and what's the connection between this and the filelist refresh problem?
[23:52:13] <eMTee> you may found it out yourself by now
[23:55:31] <poy> i see, same, it should start hashing only after all directories have been browsed.

[James Collier (james-collier412)]

This is not a bug.
eMTee, your issue seems to be to do with startup times? Please file a separate bug as that issue has nothing to do with this one.
NMDC Extension: XmlBZList requires clients to store file lists in a BZip2 compressed XML file called files.xml.bz2
DC++ caches changes to your filelist then updates files.xml.bz2 when the filelist is requested across the network.
However, you are probably wondering why they don't show up when you request your own filelist: because you are not requesting it across the network.
Rest assured though, your new files will be available to people downloading your filelist.

Because this bug has had no activity in a while I am marking it as invalid, if you still consider it an issue feel free to reopen it.

[James Collier (james-collier412)]

Marking as invalid due to lack of activity

[poy (poy)]

reverting vandalizing.

[James Collier (james-collier412)]

Is this really a valid bug? The current behaviour seems fine.

[Ben (mimenarrator)]

You are incorrect. The behavior I described occurs both when I open my own filelist or someone else downloads it across the network.

[James Collier (james-collier412)]

Ok, just to clarify reproducibility of the bug:
STEP 1. add a new share directory, DC++ hashes the directory
STEP 2. add *file* to the directory / change *file* in the directory
STEP 3. [Refresh the file list]

This fails to update the file list until the user:
1. refreshes again? Does 2 refreshes always work?
or
2. waits 15 minutes?

#Note: Restarting DC++ doesn't work either?

The expected behaviour is:
1. refresh should actually work! (My apologies for ignoring this first time around)
2. File list should be updated automatically at any time a file is added/changed/removed
3. Filelist should be updated when DC++ starts

Just to comment on 2:
I, personally, don't think it's a good idea because that would mean constantly monitoring all shares.
Currently the behaviour is to recheck every 15 minutes?
And on 3: There's a minor conflict here between startup time and refresh (it'll add to startup time).
Should this behaviour be kept and allow the 15 min refresh to occur? Should the 15 minutes be shortened for the startup case, then returned to 15 minutes after the first refresh?

[James Collier (james-collier412)]

Ben: What version of DC++ are you using?
I cannot reproduce your problem with the file list not updating when another user requests it across the network

[Ben (mimenarrator)]

Sorry it took so long for me to respond. First of all, to be clear, I'm not suggesting constant monitoring of all shared directories. I believe the default update interval is currently actually 60 minutes, and I see no reason why that should change. The bug I described is in the behavior that occurs when the file list is refreshed, however the refresh was initiated. I am currently using DC++ 0.7091, but a similar bug has existed in all previous 0.70x releases, and I believe also in 0.699. In older versions, the behavior was actually slightly different, as I will explain below. The change occurred in one of the 0.70x versions, but I don't remember which one.

The steps to reproduce the bug are as follows:
1. Add a new file or subdirectory to a directory that is already shared.
2. Initiate a file list refresh by pressing ctrl+e, restarting DC++, or simply waiting for one to be initiated automatically. Wait for the refresh to finish, and for all new files to hash.
At this point, a user who downloads my file list will not see the new file. If a new subdirectory was added, the user will see it, but not any new files it contains that were hashed before the file list refresh was completed. Files that were hashed after the file list refresh was completed generally seem to appear in the file list at this point, but occasionally some are missing, and when this occurs the missing files seem to be either the first or last files hashed after the file list refresh was completed.
In older versions this behavior is actually inverted. The files that finished hashing before the file list refresh finished would show up, while the files that finished after would not.
3. Initiate another file list refresh and wait for it to finish.
At this point, a user who downloads my file list will see the same thing they did after step 2.
4. Initiate another file list refresh (but don't necessary wait until it finishes).
At this point, a user who downloads my file list will see all the files and subdirectories that were added to a shared folder in step 1.

I should point out that when I first noticed this bug, all of the files would show up after second file list refresh was initiated, but at some point they stopped showing up until after the initiation of a third refresh. At the time I assumed this change occurred between versions, but later it occurred to me that it might have happened as a result of the increasing size and number of files and directories I was sharing. Also, I believe this bug might only occur when the number of files and directories shared is very large. I share about 3.2TiB of data, and hundreds of thousands of files, though the bug occurred at least since I was sharing 1-2TiB.

I hope this helps. I tried to include as much information as I could think of, but please feel free to ask for any more information that you think would be helpful, and I'll try to respond much more quickly than I did this time.

[Ben (mimenarrator)]

Just verified that this bug is also present in DC++ 0.75.

[James Collier (james-collier412)]

Thanks for the info Ben.
As you might've guessed I'm new to this code base and there are no function comments, but hopefully I can work this out.
I'm looking at MainWindow::handleRefreshFileList() which first sets the xmlDirty flag then calls the ShareManager::refresh() function.
ShareManager::refresh() asynchronously updates the filelist:
If refreshing is in progress it will log (and print to the statusbar?) the message:
"File list refresh in progress, please wait for it to finish before trying to refresh again".
It then runs the ShareManager::run() function in a thread.
If this fails it logs (and prints to the statusbar?) the message:
"File list refresh failed: xxxxxxxxxxx"

So what would be handy is a log, specifically the system.log, as follows:
(i) DC++ is not running
1. Rename your old system.log file if you want to keep it. (ensure "system.log" doesn't exist so that we start a new one)
2. Start DC++
3. Add some test files, one empty (zero size), one with some text, a directory with and empty and non empty file - to your share.
4. refresh your file list once
5. wait 'till it's done
6. quit DC++
7. post the "system.log" file here.

[James Collier (james-collier412)]

Note that in the current bzr trunk refresh() is called with
ShareManager::refresh(true, true, false);
In the version 0.698 code base that linuxdcpp uses it's called with the same arguments, the refresh function itself has not changed, however the run() function is a lot different.
First the run() thread gets a mapping between virtual and real filesystem paths.
Checks if the refreshDirs is set, if so, then starts "refreshing the filelist", the message:
"File list refresh initiated" is logged.
run() then calls buildTree() on all real shared paths and saves them in a std::list of Directory::Ptr's
run() then calls merge on each of the list items. And rebuildIndices(), then tells the Hub that this clients info is updated with ClientManager::getInstance()->infoUpdated(); ???

[Ben (mimenarrator)]

Another piece of information that I forgot to add is that I believe sometimes files that do not yet show up in my file list will be returned as search results. I'll create the log file you suggested when I have access to my computer with the large share in a few days.

[James Collier (james-collier412)]

I think we're getting mixed up in terminology here. Can you explain what you mean by filelist in a detailed way?

[Ben (mimenarrator)]

The files.xml.bz2 file downloaded by other users.

[James Collier (james-collier412)]

Okey doke, this is very interesting.
DC++ caches new files, in a std::list<Directory::Ptr> I think. But to save on disk I/O they aren't written out straight away.
This cache might be used to return search results...
However, if the filelist is requested at any stage, it should reflect internal data structures.
The log would be great!

[Ben (mimenarrator)]

I think I've finally figured out at least a workaround by preventing files from hashing when a file list refresh has been initiated but not finished. My implementation is probably far from ideal but I'll be happy to post a patch if you want. It still doesn't fix the underlying problem that is preventing files that are hashed before the refresh finishes from being added to the list, but I think I've made some progress in discovering the cause of that as well.

Near the end of ShareManager::buildTree(const string& aName, const Directory::Ptr& aParent), there are the lines:
if(HashManager::getInstance()->checkTTH(fileName, size, i->getLastWriteTime()))
lastFileIter = dir->files.insert(lastFileIter, Directory::File(name, size, dir, HashManager::getInstance()->getTTH(fileName, size)));

Looking at chechTTH, it appears to me at least that it will return FALSE for all new files as they are found, because it will return FALSE for any file not found in the hash store, and as best I can tell the new files are not added to the store until they have been hashed, which will not have occurred at that point. So none of the new files are actually inserted into dir on the first refresh. After the files have been hashed and the first refresh has finished, hashDone will have been called for each of the files, adding them to the hash store, and allowing checkTTH to return TRUE during the second refresh so that the new files can be added to dir.

This seems to explain why it would be necessary to refresh twice to get new files hashed between "file list refresh initiated" and "file list refresh finished" to show up in the file list, but not, as far as I can tell, why a third refresh must be initiated.

All of this is assuming I really have some understanding of what's going on in the code. I have looked for this bug off and on for almost a year now and feel like I have managed to get a grasp of things, but it is without a doubt the most complicated code I have ever worked with so I may just be fooling myself.

[Ben (mimenarrator)]

I forgot to add, the reason files hashed after the refresh has finished are added to the list immediately is that files of the first sort are added to the indices by ShareManager::on(HashManagerListener::TTHDone, const string& fname, const TTHValue& root), while files of the second sort are not. This is because a file is only added if getDirectory(fname) is not null, which will only be the case if the directory containing the file has already been added to directories. If the directory is new, it will not be added to directories until the end of the file list update, causing files hashed before that point not to be added, while files hashed after that point are. I also wonder whether the new files would be wiped out at the end of the file list update anyway even if they were initially added to directories by TTHDone, because they still will not be in newDirs, and newDirs replaces directories at the end of the file list update.

[Ben (mimenarrator)]

Er, that second sentence is horribly unclear. By files of the first sort I mean files hashed after the refresh finishes, and by files of the second sort I mean files hashed before the refresh finishes.

[poy (poy)]

just to sum up; hashing should be started only after file list refresh is finished.

[Ben (mimenarrator)]

Upon further testing, it would appear that in addition to preventing hashing from starting until after the file list refresh has finished, it is also necessary to either force a new XML list to be generated, or at least reset lastFullUpdate, so that the next time generateXmlList() is called, a new XML list will actually be generated. Otherwise, generateXmlList() will not actually generate a new XML list the next time the file list is requested for 15 minutes after the file list refresh finishes, or until a new file list refresh is initiated, whichever comes first.

[Ben (mimenarrator)]

Here is my patch. It prevents any files from being hashed until the file list refresh is complete, and forces a new XML list to be generated after the last file is hashed. I'm sure it uses very bad style and can be implemented in a better way, but it works, and I'd really like to see this bug finally get fixed.

I hereby transfer copyright to Jacek Sieka on the off chance he decides to use my implementation for some reason.

[James Collier (james-collier412)]

Your patch looks good to me.
Out of curiosity can I ask why you moved the declarations:
void generateXmlList(); & bool refreshDirs;
in ShareManager.h?

[Ben (mimenarrator)]

I made them public so they could be accessed from HashManager. I'm sure there's a better way but it works.

[Ben (mimenarrator)]

It looks like my patch is causing a memory leak somehow, I think while I think while in the loop that waits for the refresh to finish, based on when memory usage seems to increase. I can't figure out why though, as all the loop does is evaluate ShareManager::getInstance()->refreshDirs and call Thread::sleep(1000). Something not glaringly obvious must be going on that I don't understand.

[Ben (mimenarrator)]

Never mind, the memory leak appears to exist whether my patch is applied or not.

[poy (poy)]

attached is Ben's patch with a few changes:

- visibility: instead of making members public, use "friend class" and pass the bool as an argument instead of making it a class member.

- regarding the 1-second sleep call to avoid hashing while refreshing, which seems too hack-ish, empty the list of files in the hashing queue before file list refresh. i haven't tested, so please try and confirm it works.

[Ben (mimenarrator)]

There is a small flaw in Poy's patch that can be seen here:

[04:05] File list refresh initiated
[04:05] Finished hashing: <H:\TV-XviD\Sonny.With.A.Chance.S01E10.HDTV.XviD-ETACH\sonny.with.a.chance.s01e10.hdtv.xvid-etach.nfo> (64.29 KiB/s)
[04:05] XML file list generated
[04:05] File list refresh finished
[04:05] Finished hashing: <H:\TV-XviD\Sonny.With.A.Chance.S01E10.HDTV.XviD-ETACH\sonny.with.a.chance.s01e10.hdtv.xvid-etach.part01.rar> (65.62 MiB/s)
[04:05] Finished hashing: <H:\TV-XviD\Sonny.With.A.Chance.S01E10.HDTV.XviD-ETACH\sonny.with.a.chance.s01e10.hdtv.xvid-etach.part02.rar> (61.13 MiB/s)
[04:05] Finished hashing: <H:\TV-XviD\Sonny.With.A.Chance.S01E10.HDTV.XviD-ETACH\sonny.with.a.chance.s01e10.hdtv.xvid-etach.part03.rar> (41.71 MiB/s)
[04:05] Finished hashing: <H:\TV-XviD\Sonny.With.A.Chance.S01E10.HDTV.XviD-ETACH\sonny.with.a.chance.s01e10.hdtv.xvid-etach.part04.rar> (48.33 MiB/s)
[04:05] Finished hashing: <H:\TV-XviD\Sonny.With.A.Chance.S01E10.HDTV.XviD-ETACH\sonny.with.a.chance.s01e10.hdtv.xvid-etach.part05.rar> (70.47 MiB/s)
[04:05] Finished hashing: <H:\TV-XviD\Sonny.With.A.Chance.S01E10.HDTV.XviD-ETACH\sonny.with.a.chance.s01e10.hdtv.xvid-etach.part06.rar> (65.62 MiB/s)
[04:05] Finished hashing: <H:\TV-XviD\Sonny.With.A.Chance.S01E10.HDTV.XviD-ETACH\sonny.with.a.chance.s01e10.hdtv.xvid-etach.part07.rar> (61.13 MiB/s)
[04:05] Finished hashing: <H:\TV-XviD\Sonny.With.A.Chance.S01E10.HDTV.XviD-ETACH\sonny.with.a.chance.s01e10.hdtv.xvid-etach.part08.rar> (53.78 MiB/s)
[04:05] Finished hashing: <H:\TV-XviD\Sonny.With.A.Chance.S01E10.HDTV.XviD-ETACH\sonny.with.a.chance.s01e10.hdtv.xvid-etach.part09.rar> (76.50 MiB/s)
[04:05] Finished hashing: <H:\TV-XviD\Sonny.With.A.Chance.S01E10.HDTV.XviD-ETACH\sonny.with.a.chance.s01e10.hdtv.xvid-etach.part10.rar> (70.47 MiB/s)
[04:05] Finished hashing: <H:\TV-XviD\Sonny.With.A.Chance.S01E10.HDTV.XviD-ETACH\sonny.with.a.chance.s01e10.hdtv.xvid-etach.part11.rar> (53.98 MiB/s)
[04:05] Finished hashing: <H:\TV-XviD\Sonny.With.A.Chance.S01E10.HDTV.XviD-ETACH\sonny.with.a.chance.s01e10.hdtv.xvid-etach.part12.rar> (76.50 MiB/s)
[04:05] Finished hashing: <H:\TV-XviD\Sonny.With.A.Chance.S01E10.HDTV.XviD-ETACH\sonny.with.a.chance.s01e10.hdtv.xvid-etach.part13.rar> (43.68 MiB/s)
[04:05] Finished hashing: <H:\TV-XviD\Sonny.With.A.Chance.S01E10.HDTV.XviD-ETACH\sonny.with.a.chance.s01e10.hdtv.xvid-etach.sfv>
[04:06] XML file list generated

[Ben (mimenarrator)]

Also it seems to be updating the XML list too often as can be seen here:

[04:20] File list refresh initiated
[04:20] Finished hashing: <F:\mp3\Podcasts\Politico - Congress Full Story - Podcast\Commencement speech conundrum.mp3> (11.42 MiB/s)
[04:21] XML file list generated
[04:21] Finished hashing: <F:\mp3\Podcasts\POLITICO - Politics Full Story - Podcast\F-22 backers question Gates' cuts.mp3> (14.56 MiB/s)
[04:21] Finished hashing: <F:\mp3\Podcasts\Politico - Life Full Story - Podcasts po\Commencement speech conundrum.mp3> (12.86 MiB/s)
[04:21] Finished hashing: <F:\mp3\Podcasts\Politico - Lobbying Full Story - Podcast\Suite Talk April 30, 2009.mp3> (12.27 MiB/s)
[04:21] XML file list generated
[04:21] Finished hashing: <H:\TV-XviD\MythBusters.S07E04.YouTube.Special.HDTV.XviD-FQM\mythbusters.s07e04.hdtv.xvid-fqm.nfo> (580.79 KiB/s)
[04:21] XML file list generated
[04:21] Error hashing <H:\TV-XviD\MythBusters.S07E04.YouTube.Special.HDTV.XviD-FQM\Sample\sample-mythbusters.s07e04.hdtv.xvid-fqm.avi>: The process cannot access the file because it is being used by another process.
[04:21] Error hashing <H:\TV-XviD\MythBusters.S07E04.YouTube.Special.HDTV.XviD-FQM\mythbusters.s07e04.hdtv.xvid-fqm.r00>: The process cannot access the file because it is being used by another process.
[04:21] Error hashing <H:\TV-XviD\MythBusters.S07E04.YouTube.Special.HDTV.XviD-FQM\mythbusters.s07e04.hdtv.xvid-fqm.r01>: The process cannot access the file because it is being used by another process.
[04:21] Error hashing <H:\TV-XviD\MythBusters.S07E04.YouTube.Special.HDTV.XviD-FQM\mythbusters.s07e04.hdtv.xvid-fqm.r02>: The process cannot access the file because it is being used by another process.
[04:21] Error hashing <H:\TV-XviD\MythBusters.S07E04.YouTube.Special.HDTV.XviD-FQM\mythbusters.s07e04.hdtv.xvid-fqm.r03>: The process cannot access the file because it is being used by another process.
[04:21] Error hashing <H:\TV-XviD\MythBusters.S07E04.YouTube.Special.HDTV.XviD-FQM\mythbusters.s07e04.hdtv.xvid-fqm.r04>: The process cannot access the file because it is being used by another process.
[04:21] Error hashing <H:\TV-XviD\MythBusters.S07E04.YouTube.Special.HDTV.XviD-FQM\mythbusters.s07e04.hdtv.xvid-fqm.r05>: The process cannot access the file because it is being used by another process.
[04:21] Error hashing <H:\TV-XviD\MythBusters.S07E04.YouTube.Special.HDTV.XviD-FQM\mythbusters.s07e04.hdtv.xvid-fqm.r06>: The process cannot access the file because it is being used by another process.
[04:21] Error hashing <H:\TV-XviD\MythBusters.S07E04.YouTube.Special.HDTV.XviD-FQM\mythbusters.s07e04.hdtv.xvid-fqm.r07>: The process cannot access the file because it is being used by another process.
[04:21] Error hashing <H:\TV-XviD\MythBusters.S07E04.YouTube.Special.HDTV.XviD-FQM\mythbusters.s07e04.hdtv.xvid-fqm.r08>: The process cannot access the file because it is being used by another process.
[04:21] Error hashing <H:\TV-XviD\MythBusters.S07E04.YouTube.Special.HDTV.XviD-FQM\mythbusters.s07e04.hdtv.xvid-fqm.r09>: The process cannot access the file because it is being used by another process.
[04:21] Error hashing <H:\TV-XviD\MythBusters.S07E04.YouTube.Special.HDTV.XviD-FQM\myt...

[poy (poy)]

this should fix the problem of the file list being generated too often; also i've made its log message more explicit. what's with these "The process cannot access the file because it is being used by another process." messages, are they related to the patch?

[Ben (mimenarrator)]

No, that's unrelated to the patch and is a result of those files being written to while DC++ tried to hash them.

[Ben (mimenarrator)]

The first problem I posted seems to happen because as soon as a new file is found, s.signal() is called, allowing the first file in the queue to be hashed. This is what I could find no other way to avoid than to use a very ugly sleep loop, but I'm sure you can think of a better way.

[Twilight2013 (twilight2013)]

Go to File, Settings, Advanced & disable Use fast hashing method (disable if you have problems with hashing). In order for this feature to take effect, close DC++ application & reopen again. Even though I have my files refreshed every 1 hour, the file keeps on rehashing when that feature is enabled. When I have that feature disabled, the files hash completely & orderly.

[poy (poy)]

patch 4, i'm more confident in this one. :)

[Ben (mimenarrator)]

It still seems like It might be generating an xml list slightly too often, the one time at the end being sufficient, but this is much better:

[18:40] File list refresh initiated
[18:40] XML file list <C:\Program Files (x86)\DC++\files.xml.bz2> generated
[18:40] XML file list generated
[18:40] File list refresh finished
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.nfo> (14.32 KiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r00> (43.61 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r01> (65.62 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r02> (45.85 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r03> (65.32 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r04> (76.50 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r05> (41.58 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r06> (21.35 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r07> (50.91 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r08> (28.67 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r09> (10.53 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r10> (20.38 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r11> (19.92 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r12> (19.10 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r13> (21.84 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r14> (16.09 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r15> (18.34 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r16> (32.73 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r17> (61.13 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s06e03.hdtv.xvid-gnarly.r18> (39.85 MiB/s)
[18:40] Finished hashing: <H:\TV-XviD\American.Chopper.S06E03.HDTV.XviD-GNARLY\american.chopper.s...

[poy (poy)]

it doesn't do that here - it only generates a file list when requested.

[poy (poy)]

Fixed in version 0.760.