libraryscanner blocks the library for other transactions

[Max Linke]

The libraryScanner runs in a separated thread with an open transaction which blocks the sqlite database. So if you've added a large chunk of new music and scan it during that time it is not possible to create/change/delete playlists/crates/etc.

I'm not sure of the right solution for this. We could write some threadsafe transaction class or move the library and scanner into a thread that is seperated from the GUI and save a queue of transactions in that thread.

[Daniel Schürmann (daschuer)]

See: http://www.sqlite.org/faq.html#q6

This is an other argument for Bug #991717, a dedicated library thread.

[Max Linke (max-linke)]

To solve the our library backends (DAOs) should be able to talk to the LibraryScanner(LS). I imagine this to look something like this.

DAO -> request LS to pause if running
LS -> pauses any running scan and signal DAO that the scanner has paused and closed all open transactions
DAO -> receives ok signal from LS and then commits transaction
DAO -> signals LS that it's finished it's transactions and tells it to unpause
LS -> reassumes scan if paused before

This should be necessary even if the LS and the library run in the same thread. The implementation of this scheme should be relatively straight forward but I won't have time to do this in the next couple weeks

[Daniel Schürmann (daschuer)]

Is bug might be related to bug #870128

[RJ Skerry-Ryan (rryan)]

This is much reduced in severity now that the locking issues caused by the setlog feature are fixed. A better fix should come in a future release.

[Max Linke (max-linke)]

Which locks are you referring to?

[RJ Skerry-Ryan (rryan)]

Many people (generally developers) were experiencing database-is-locked
error messages and crashes because of the way we were listing Playlists on
the sidebar. We were using a QSqlTableModel which if there are more rows
than it fetches by default in the result set then it keeps a lock on the
database. Since we were never fetching more results from it the lock was
always on once the playlist table reached a certain size. The setlog
feature triggered this because it created a new playlist every time you
start Mixxx. Fixing this issue got rid of the database-is-locked issues
that were plaguing us over the summer.

On Wed, Nov 21, 2012 at 4:19 AM, Max Linke <email address hidden>wrote:

> Which locks are you referring to?
>
> --
> You received this bug notification because you are a member of Mixxx
> Development Team, which is subscribed to Mixxx.
> https://bugs.launchpad.net/bugs/1051106
>
> Title:
> libraryscanner blocks the library for other transactions
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mixxx/+bug/1051106/+subscriptions
>

[Max Linke (max-linke)]

Ah now I remember but they don't affect this bug at all. It is still not possible to change playlists/crates while a library scan is running

[Steven Boswell (ulatekh)]

Yes, it IS possible to interact with the Mixxx window while a library scan is running.
The problem is that the library-scan dialog is not modal.
A call to "setWindowModality (Qt::WindowModal)" in the LibraryScannerDlg constructor is part of the solution.
Making the dialog visible immediately, instead of after 2 seconds (done in LibraryScannerDlg::slotUpdate()) is the other part of the solution.

[RJ Skerry-Ryan (rryan)]

Hm, some people have gigantic libraries and making mixxx unusable during
the scan via modal dialog would be bad. I like the idea of periodic
commits of the scanner if only so that people who have large libraries can
start to use the library before the scan is finished. I would prefer we fix
locking issues by switching all library access to a single thread.

On Fri, Apr 12, 2013 at 11:59 AM, Steven Boswell <email address hidden> wrote:

> Yes, it IS possible to interact with the Mixxx window while a library scan
> is running.
> The problem is that the library-scan dialog is not modal.
> A call to "setWindowModality (Qt::WindowModal)" in the LibraryScannerDlg
> constructor is part of the solution.
> Making the dialog visible immediately, instead of after 2 seconds (done in
> LibraryScannerDlg::slotUpdate()) is the other part of the solution.
>
> --
> You received this bug notification because you are a member of Mixxx
> Development Team, which is subscribed to Mixxx.
> https://bugs.launchpad.net/bugs/1051106
>
> Title:
> libraryscanner blocks the library for other transactions
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mixxx/+bug/1051106/+subscriptions
>

[Steven Boswell (ulatekh)]

The enclosed patch implements incremental/pausable rescan.
Unpack this to your mixxx checkout directory (i.e. the one with .bzr, .bzrignore, and mixxx in it).
It'll unpack a patch file at the top level, and add mixxx/src/dlglibraryscanner.ui .
Let me know what you think!

[Daniel Schürmann (daschuer)]

http://www.sqlite.org/cvstrac/wiki?p=MultiThreading

[Steven Boswell (ulatekh)]

As far as I know, my patch doesn't change the multithreading behavior of library-rescan -- it remains as before, just with the possibility of incrementally committing changes. No?

[Daniel Schürmann (daschuer)]

My comment #11 was only a related link to this bug. No issue with patch #10.
Sorry for the confusion.

[quequotion (quequotion)]

I have this problem compounded by another issue: mixxx apparently never saves the database or it's configuration, and so rescans the library and asks me to configure outputs on every startup.

The result is often that I will get but 870128 and mixxx will crash on open or that it will not crash but never open....

[quequotion (quequotion)]

*bug 870128

Also, by "not crash but never open" i mean the process is running, no errors come up, nothing output to terminal either and it uses 96% cpu (about 1 hyperthread of 8 or 1/2 of a core actually). Although my desktop doesn't slow down at all and everything looks fine, mixxx seems to be working very hard on something it never finishes and then no interface ever appears on the screen.

This happens inconsistently. Sometimes it opens with no problem (but still rescans the library and requires configuration).

[Daniel Schürmann (daschuer)]

Hi quequotion

Thank you for reporting tis bug. It sound that it is a different issue with not saving the DB. There might be the same root cause, but as long it is not verified, I would prefer to track your issue separate.
So can you please file a new bug and put all infos in it that might be helpful, including the Mixxx version, your OS + Version and your mixxx.log file from faulty starts.

Thank you.

[quequotion (quequotion)]

>>Daniel, by "compounded by another issue" I mean to say that I do in fact have /this/ problem, as well as an additional problem which makes this one worse.

The other, separate issue, is that Mixxx usually needs to be completely reconfigured at each run. I haven't reported a bug for this because I am uncertain how to debug the issue. The secondary issue is inconsistent, with no particular relationship to any set of circumstances as far as I can tell.

Because of the secondary problem, wherein the library has to be completely rescanned at almost every run, I experience /this/ problem at almost every run (database locked).

Actually, I haven't looked into this for a while and I am not certain of the current status of either issue.

[RJ Skerry-Ryan (rryan)]

Bumping out of 1.12.0 -- a more sound fix is on the way for 1.13 (e.g. Nazar's library thread).

[kNottien (kentarveolsen)]

Wishlist: ...

Configurable DB Backend, so there is the possibility of Team sharing the data with the option of DropBox type of file sharing and so on..

[Daniel Schürmann (daschuer)]

Dropbox sharing deserves a separate Bug report.

[RJ Skerry-Ryan (rryan)]

The scanner has 3 phases:

1) Opening: legacy library import, mark all tracks/dirs as needs_verification

2) recursive parallel scan -- updates LibraryHases and inserts new tracks into library table

3) Final cleanup -- verify tracks outside library directories, mark non-verified tracks/dirs as deleted, detect moved files

Now that we have switched to not rolling back new track insertions upon cancel I think it makes sense to not put phase 2 in a transaction.

The scanner is the only DB connection that touches the library hashes and needs_verification bits so there is no race condition caused by writing/reading those outside of a transaction since there is only one scanner.

This will allow other parts of Mixxx to work for most of the time the scanner is running. Thoughts?

[Max Linke (max-linke)]

But keeping step 2 as one transaction is the reason why the library scan is so fast now. If we use serveral transaction then the libraryscanner will be bound by the time it takes sqlite to complete a single transaction, which is really slow due to a lot of error/consistency checking of sqlite.

I'm also not sure about the race-conditions. AFAIK SQLite supports only one transaction at a time and will fail others during an active transaction. So what happens if we start a transaction in the UI thread the same time as a transaction happens in the libraryscanner thread. One of them will fail. It will be possible to recover from that and to retry the transaction in the scanner but there is no straigtforward way to do this in the UI.

I'm still in favor of keeping Phase 2 in a transaction.

[Uwe Klotz (uklotzde-deactivatedaccount)]

At least rescanning the library on startup should be disabled:

https://mixxx.discourse.group/t/auto-dj-randomly-stopping/23919/9

[Swiftb0y (swiftb0y)]

Mixxx now uses GitHub for bug tracking. This bug has been migrated to:
https://github.com/mixxxdj/mixxx/issues/6631