Run AutoDJ processor in its own thread

[infarmer]

Mixxx animations freeze, Mixxx animations jump and AutoDj stops when certain options are used.
Options in test:
(1) Clear BPM and Beat Grid
(2) Reload Track Metadata
(3) Analize

In Normal Operation:
.- Mixxx animations freeze temporarily for long time when certain options (1), (2) are used, when those options have finished their job, Mixxx go back into normal operation. The time Mixxx will remain freezed depends of the amount of songs involved. When mixing, if the song currently played in a deck is almost at the end, you can't start the other deck to mix. When Mixxx is freezed you can do anything with the program till those features have finished.
.- Mixxx animations jump when you use option (3). It occurs exactly when you start using the Analize feature.

In AutoDj Operation:
.- Mixxx animations freeze temporarily for long time and AutoDj feature stop working when certain options (1), (2) are used, when those options have finished their job, Mixxx animations go back into normal operation but the AutoDj feature is stopped. The first song, the one who is currently playing when you use those options is played till the end, althought the animation is stopped, the second song loaded into the other deck will never be mixed, the "Enable Auto Dj" button is still activated. We start this test when the song have 25 - 30 secconds to reach the end.

Note: To reproduce the bugs you have to work with a large amount of files. We are using to test Mixxx 4 GB of songs: 600 files in 30 folders.

Mixxx 1.11.0 beta2
OS: Windows 7 64 bits
CPU: Intel Core Duo CPU E4700 @ 2.60 GHz
Ram: 4 Gb DDR 2.
Video-Card: NVIDIA GeForce 9600 GT

[Daniel Schürmann (daschuer)]

Thank you for the report.

Just tested with Ubuntu 12.4 and the latest lp:mixxx/1.11 #3683.

I can confirm that all auto DJ Comments are delayed like all other GUI commands when reloading Meta data (2).
I cannot confirm that a auto DJ command is lost.

[RJ Skerry-Ryan (rryan)]

We're aware that these operations can freeze the GUI thread, but I never realized that that means that AutoDJ logic will also freeze. That is bad news.

[infarmer (galisalsa)]

I don't know nothing about programing, so just talking here giving ideas, perhaps some can be usefull or generate another good idea.
I think you have to implement a priority system to handdle this issue, a good priority system can be used to handdle more issues/ features in the future if it is well implemented.
I'm thinking now in some programs like some antivirus or compressors (zip, rar, etc.), lets say they have two modes of operation: Normal or foreground mode: In this mode the program use a big amount of machine resources, and Background mode: In This mode the program only take a few resources to made the tasks so the rest of resources in the machine are going to other programs.
.- Outside Mixxxx (General Machine Resources)
First we assume that if you are using Mixxx you will be mixing (if you wanna listen music i pressume you will use other program) with two variants: Mixxx in foreground, normal DJ mode: Mixxx must take a maximum of 75% and a minimun of 50% from the machine resources for this operation. Mixxx in background mode, (Cafe or Restorants perhaps and probably with autodj enabled), Mixxx must take 50 % not less! of machine resources. Why no less than 50 %, perhaps you are really in foreground mode and just switch to background to make a manual search of a file inside your hdd for instance, a lesser value than this could cause troubles in the future. An adjusting bar, select an option or enter % box in settings could be nice to let the user some tweak between those presetted values.
.- Inside Mixxx (working with the resources taken by Mixxx in the previous parragraph, if you have taken 75%, this value will be now your 100% of resources (Mixxx resources)).
Here you have to set a priority system again. The maximum priority (foreground) must go the song who is currently played, the next song and program animations, etc. Operations like scan tags, analize, etc., must be in the foreground. I think for foreground you have to take from 75% (better value) to 50% (for powerfull machines) of MIXXX Resources. An adjusting bar, select an option or enter % box in settings could be nice to let the user some tweak between those presetted values.
Deeper into this, i think Mixxx should scan the machine every time you start it for values like memory amount, processor speed and processors cores amount, why everytime?, user can add/remove memory, change processor etc. Once you have retrieve those values, Mixxx should calculate the resources he needs and start working with those values. Why cores amount?, in dual core machines or better, you can use one core to foreground, and other core for the background operations, althought the memory will be the same for both.
More deeper into this, if you work with priorities, you can set more than 2 priorities, althougth this means more code to be maintained.
well it is all, excuse my English and have a nice day...

[RJ Skerry-Ryan (rryan)]

Renaming to make this bug about making AutoDJ processor independent of freezes in the main thread. Since AutoDJ is performance critical it should have its own thread.

[Daniel Schürmann (daschuer)]

I am not sure if we can afford an other thread between GUI and Engine priority.
This will just introduce probably unnecessary tread switches and overhead for syncing.

Actually there is no difference between the Human DJ using the GUI and the AutoDJ.
This mean if AutoDJ is looked out, a human DJ will be looked out as well, which Is at least as worse.
AutoDj makes also use of the database, which is still running in the GUI thread.

It is best to solve the root bug and remove the SQlite access from the GUI

Of cuase we should make the transitions independent from deck transport.
Maybe we can move some of the Audto DJ transaction process to the engine thread.
Remember that we have planed to support different transaction effects.
Not sure what will be the right architecture for it, but maybe we can utilize the Quick Effect rack and have the transaction effects also available without AutoDJ.

[RJ Skerry-Ryan (rryan)]

That's a fair point -- the music could still stop if AutoDJ gets blocked though (it could fail to load the next track or fail to set the deck playing). We should aim for never blocking the GUI thread (e.g. by moving the library access to its own thread) but we certainly have a history of doing it.

> I am not sure if we can afford an other thread between GUI and Engine priority.

I don't think we have a metric for what we can and can't afford -- any suggestions?

[Daniel Schürmann (daschuer)]

> I don't think we have a metric for what we can and can't afford -- any suggestions?

I do not have a metric for it. The whole topic is quite difficult because we have to share the CPU with virus scanners among other things.
On on hand, more threads are good, because the OS is able to make use of all cores. On the other hand, they are bad, because once we have 100 % load, we cannot gain CPU by an other thread, it will make things worse.

For Auto DJ needs the library, so we are bound with the library thread. We may only fix the issue that fading is not smooth by moving this to the engine thread.

[Owen Williams (ywwg)]

What is autodj doing that causes the gui thread to block?

[Owen Williams (ywwg)]

er maybe a better way to phrase it is, why is autodj so performance sensitive? It shouldn't need callback-level latency to push some faders around slowly.

[Owen Williams (ywwg)]

"autodj is performance critical" is the WTF

[RJ Skerry-Ryan (rryan)]

It's the other way around actually. When the GUI thread is blocked by some other (bad) code, AutoDJ is unable to do anything. It's performance critical in the sense that if it doesn't run its processing logic due to another bad actor the party can stop. It is trivial to fix this by sticking it in its own thread.

Modern OSes hardly break a sweat with hundreds to thousands of threads -- even on one core.

[Owen Williams (ywwg)]

I'm not opposed to moving it to its own thread, but then the boundaries of the control objects and signals get trickier. It seems like an autodj thread should be as minimal as possible -- just performing the co manipulation needed in response to commands from the outside: StartFade, Stop, Start Play, etc.

We have to be careful that the track load commands aren't coming from the autodj thread, for instance.

[RJ Skerry-Ryan (rryan)]

+1 to ADJ processing staying simple. And I think at some point moving the crossfader automation to the engine probably makes sense.

Hm, COs and load command are both fine since they rely on signals and slots -- which handle queueing across thread boundaries for you (The ADJ processor thread would have its own event loop).

But that reminds me that the ADJ processor uses a table model to access the ADJ playlist -- which isn't currently ready for use outside of the main thread.

[Swiftb0y (swiftb0y)]

Mixxx now uses GitHub for bug tracking. This bug has been migrated to:
https://github.com/mixxxdj/mixxx/issues/6860