Improve library rescan speed

[Ben Clark]

Library rescan seems very slow compared to other applications that perform a similar function, even when there have been no modifications at all.

I have attached a small fix that (on my machine) improves rescan times by about 35-40% by surrounding the calls:

m_libraryHashDao.updateDirectoryStatus(dirPath, false, true);
m_trackDao.markTracksInDirectoryAsVerified(dirPath);

in a transaction. These calls are made when nothing has changed in a directory and mark the directory as verified and mark the tracks inside that directory as verified.

This was originally found in Bug #905669 but this strayed quite far from the suggestion in that bug so I'm opening a separate issue to track this and any other potential improvements.

[Daniel Schürmann (daschuer)]

Hi Ben,

I have tried you patch on the top of lp:mixxx with:
Ubuntu lucid, 2GB Ram, Intel Atom N270@1.6GHz.
My 3000 Tracks are partly on an 16 GB SD Card in 761 Subfolders.

There was no significant speed improvement on this system.
I have tested a complete new scan after deleting the mixxxdb.sqlite and a rescan with an unchanged library.
The only thing I have noticed is an improvement with the second scan. I think there must be a cache somewhere.

What are your test results?

[Ben Clark (bencoder)]

Hi Daniel,

thanks for trying out the patch - I have retested and noticed that the improvements weren't as large as I initially thought - probably due to caching as you mentioned, but there was still a fairly significant increase:

I tested on lp:mixxx (optimize=9) with 32bit Ubuntu Lucid, 2GB ram, AMD Athlon X2

4742 tracks on external drive across about 500 subdirectories.

                        Initial Scan Rescan 1 Rescan 2

Test 1 Without patch 220s 94s 96s
Test 2 Without patch 94s 88s 89s

Test 3 With patch 65s 59s 58s
Test 4 With patch 64s 70s* 55s

Test 5 Without patch 60s 78s 77s

* = I may have slowed this scan down inadvertently by performing other actions on the computer.

I have the log file of each of these 5 runs. I am quite surprised by the initial scan speed of test 5 (when I went back to without the patch, to ensure the improvement was not just in caching).

[Sean M. Pappalardo (pegasus-renegadetech)]

Have you taken into account disk caching performed by the kernel and the controller? You should actually power cycle (or at least restart) your system between each test to be sure it's cleared.

[Ben Clark (bencoder)]

All my testing has indicated that the slowness in rescanning is due to the writes to the database when verifying directories and tracks, and doing these inside a transaction improves this by reducing the number of writes required.

Taking this further, I can't actually determine any reason why we need to mark the directories and tracks as verified while performing the recursive scan. I'm attaching a patch that speeds my rescan time up to less than 4 seconds(!), by holding a QStringList of verified directories, and only verifying them (and the tracks inside) inside the transaction that occurs after the recursive scan, before we delete anything that is unverified. This makes our rescanning speed competitive with similar applications.

Would appreciate if someone could review this patch.

[Owen Williams (ywwg)]

Uh, *wow*, that works as advertised. Rescan speed is about 3-4 seconds (used to be a few minutes).

The only thing I notice is that before, even if the scan was canceled we would mark the directories as verified (because that marking was occurring as the scan proceeds). Now, we only mark things as verified if the scan finishes cleanly. Is this an ok change?

[Ben Clark (bencoder)]

Hi Owen,

I can't see that being an issue. I don't think we use the verified flag for anything except determining which files need to be deleted, which we don't do except on a fully completed scan anyway, so I don't think there's a difference between leaving them fully unverified vs. partially verified on an incomplete scan.

[RJ Skerry-Ryan (rryan)]

I agree I don't think any harm can come from not confirming the files/directories. The logic that actually marks them missing/moved/deleted only runs if the scan wasn't cancelled.

Thanks for the patch Ben! I'll review it and get it in this weekend. It should be in Mixxx 1.10.1 (if we end up doing a 1.10.x point release) and Mixxx 1.11.0.

Is it ok to credit you as "Ben Clark"?

Also, please sign our contributor agreement here: https://docs.google.com/a/mixxx.org/spreadsheet/viewform?formkey=dEpYN2NkVEFnWWQzbkFfM0ZYYUZ5X2c6MQ#gid=0

This gives us permission to distribute your change with MIxxx.

[RJ Skerry-Ryan (rryan)]

The patch looks good to me -- here are my comments:

1) It's fine to keep the emit() for the hash progress where it was previously. I think even though it's a little bit of a fib it's better to have the progress dialog constantly updating so you don't think it's slow / hung.

2) The two methods updateDirectoryStatus and markTracksInDirectoryAsVerified were both created specifically for this part of the scanner and aren't used anywhere else. It would be even faster if you updated these two methods so that they take the whole list of directories and just issue one big update query instead of doing 1 query for each directory.

[Daniel Schürmann (daschuer)]

Here are my test results on system from #2

Initial Scan
unpatched patched
142 s 144 s

Rescan
unpatched patched
24 s 8 s

Congratulation!
Initial Scan is slightly longer due to the patch, but its only a one time issue.

[RJ Skerry-Ryan (rryan)]

Thanks Ben -- committed to lp:mixxx/1.10

[Swiftb0y (swiftb0y)]

Mixxx now uses GitHub for bug tracking. This bug has been migrated to:
https://github.com/mixxxdj/mixxx/issues/6334