Loading info takes too long (waveforms and BPM)

[toomuch]

I have a 12MB file (saved locally) loaded and playing in Deck A.
It takes about a minute to load the waveforms and the BPM info. For other tracks it is slightly faster.

Is it possible to improve the coding or is my hardware to weak? The logs show nothing suspicious.
Win7 x64, Intel Pentium Dual Core T2390, 3GB RAM (Asus X51L laptop)
Mixxx 1.10.0beta1 x64

[Daniel Schürmann (daschuer)]

Does it happen only the first time you load the track or also if you reload it?
What kind of file is it?
Do you have a proportional analysis duration with other tracks?

Btw.:
We should consider to merge Jonas Ådahl's "Track analysis progress visualisation" patch (Bug #803740) . I use it successful on my netbook.

[toomuch (toomuch)]

It's MP3 (256 kBit/s quality).
When I reload the file (via drag & drag from Windows Explorer) the upper waveform is there after a few seconds, but the lower (compacted) waveform still takes up to 45 seconds to load. When loading the track, the CPU load is about 80-90%
Proportional: Yes, the longer and larger the files are, the longer the waveforms takes to load. But I also have a file with 1.6MB (duration: 5:29 min) which also takes 15 seconds to load the lower waveform.

After loading CPU load is 50-70%, is this normal?

I like Jonas Ådahl's patch and would love to see it implemented. But still, this does not resolve the issue.

[Daniel Schürmann (daschuer)]

Ok, I have made a little benchmark with a 6:36 min track (256 kBit/s 12.1 MB) lp:mixxx #2966

Results:
1.) Ubuntu Oneiric 32 bit, Intel Dual E2180 @ 2.00 GHz, 2 GiB RAM
** 4.8 s
2.) Ubuntu Lucid 32 bit, Intel Atom N270 @ 1.6 Ghz, 2GiB RAM
** 18 s

Passmark CPU Mark:
T2390 @ 1.86GHz : 948
E2180 @ 2.00GHz : 1192
N270 @ 1.60GHz : 304
results
On both Systems I Have a CPU load during analysis of nearly 100%
Mixxx with no track loaded 45 % CPU load,
one track is loaded 55 % CPU load,
two tracks loaded 65 % CPU Load.

If I load the same track as second track, analysis takes 1 second more, on system 1.).

I have nearly the same duration with the same track encoded with 128 kBit/s. So the bit rate does not effect the analysis time on my systems.

[Daniel Schürmann (daschuer)]

Please try to load your test track from library, it should be loaded much faster the second time.
Do you have the same results with a second test track with the same length and bit rate?

[toomuch (toomuch)]

Hi Daniel,

yes, loading it from library the second time is much faster, almost instantly.

The second test file (6.5 mins, 256 KB/s, 10MB) took 30 seconds to load the overview waveform while the other deck was playing. The upper waveform appeared very quickly.

Is my hardware to weak or can the code be optimized?

[toomuch (toomuch)]

I had a little private party on the weekend, where I had insane loading times for BPM and waveforms. I did not stop the time but it interrupted my "mixing flow"...

[RJ Skerry-Ryan (rryan)]

Hey toomuch,

Was it possible you accidentally loaded a very long track into a deck? If you load, for example, a live set or otherwise hour long or more track then that could tie up the analyzer for quite a while as it analyzes the entire track. Any tracks that you load after that hour long one will have to wait for the hour-long one to finish before they can be analyzed.

Your hardware looks plenty fast to me -- so seeing analysis take longer than 10 seconds or so is abnormal. Could you try with a Linux live CD to see if the problem is contained to Windows 7 x64?

[toomuch (toomuch)]

No, I did not accidentally load a very long track into a deck. But for the record: You cannot abort a track scanning by loading another track to that very deck?

Sooo, I did a bit of testing with an Ubuntu-Live-CD (11.04) and Mixxx 1.10.0beta1. Results (loading times in seconds) are as follows:

Filesize Ubuntu Win7
8 6 16
3,5 3 16
5 4 14
8 10 13
10 13 22
24 15 40
19 12 38

Playing two files leads to 58% CPU load. While loading it rises to 126%.

So it looks like a Win7 issue. On Win7 I did not have any full screen flash or something similar going. :/

[Sean M. Pappalardo (pegasus-renegadetech)]

This problem started awhile ago when we decreased the GUI timer frequency. For some reason, the track analysis time is directly linked to the GUI refresh rate, on Windows at least. If we can decouple these, I think that will solve the problem.

[RJ Skerry-Ryan (rryan)]

Yes that's right that there is no way for loading a track to clear or
jump-to-the-front of the queue of previously loaded tracks to analyze.

We set the analyzer queue thread priority to IdlePriority.

toomuch, on Linux have you configured your /etc/security/limits.conf to
allow your username to change the priority of threads? By default many
distros are configured to not allow this. If you haven't then that might
explain it. Your Linux machine is unable to lower the priority of the
analyzer thread while you are able to on Windows.

On Mon, Dec 5, 2011 at 3:37 PM, Sean M. Pappalardo <
<email address hidden>> wrote:

> This problem started awhile ago when we decreased the GUI timer
> frequency. For some reason, the track analysis time is directly linked
> to the GUI refresh rate, on Windows at least. If we can decouple these,
> I think that will solve the problem.
>
> --
> You received this bug notification because you are a member of Mixxx
> Development Team, which is subscribed to Mixxx.
> https://bugs.launchpad.net/bugs/888817
>
> Title:
> Loading info takes too long (waveforms and BPM)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mixxx/+bug/888817/+subscriptions
>

[toomuch (toomuch)]

I did not change anything, just put in the live CD, installed Mixxx and tested.

[RJ Skerry-Ryan (rryan)]

OK, seems like that could be the case then.

On Mon, Dec 5, 2011 at 4:18 PM, toomuch <email address hidden> wrote:

> I did not change anything, just put in the live CD, installed Mixxx and
> tested.
>
> --
> You received this bug notification because you are a member of Mixxx
> Development Team, which is subscribed to Mixxx.
> https://bugs.launchpad.net/bugs/888817
>
> Title:
> Loading info takes too long (waveforms and BPM)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mixxx/+bug/888817/+subscriptions
>

[RJ Skerry-Ryan (rryan)]

Maybe we should make the analyzer priority user configurable via
preferences. IdlePriority isn't doing people with quad-cores any favors.

On Tue, Dec 6, 2011 at 11:39 AM, RJ Ryan <email address hidden> wrote:

> OK, seems like that could be the case then.
>
>
> On Mon, Dec 5, 2011 at 4:18 PM, toomuch <email address hidden> wrote:
>
>> I did not change anything, just put in the live CD, installed Mixxx and
>> tested.
>>
>> --
>> You received this bug notification because you are a member of Mixxx
>> Development Team, which is subscribed to Mixxx.
>> https://bugs.launchpad.net/bugs/888817
>>
>> Title:
>> Loading info takes too long (waveforms and BPM)
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/mixxx/+bug/888817/+subscriptions
>>
>
>

[Sean M. Pappalardo (pegasus-renegadetech)]

Adding a prefs option sounds good to me. Also IdlePriority doesn't make much sense to me at first glance though since A) when Mixxx is running, not much is idle and B) the analyzer blocks other things. I would suggest defaulting it to whatever is just above Idle.

And RJ, can you speak to the analyzer-tied-to-GUI-refresh-rate thing? I distinctly remember analysis taking MUCH longer on Windows when we reduced it from ~200Hz to 30, then you bumped it to ~60 for the time being to make this less noticeable.

[RJ Skerry-Ryan (rryan)]

There's no reason I can think of that the analyzer would be gated by the
GUI refresh unless a QueuedConnection connected a signal from the analyzer
to something that blocks on a GUI refresh. (which I checked yesterday and
it doesn't).

You might be thinking of when we used to have a usleep in the analyzer to
make it back off from hogging the CPU. This resulted in particularly
abysmal analyzer performance on Windows relative to Linux/Mac. We don't
have that anymore.

Also the GUI refresh rate was never 200Hz I don't think. We've always been
in the 30-60 range.

On Tue, Dec 6, 2011 at 12:23 PM, Sean M. Pappalardo <
<email address hidden>> wrote:

> Adding a prefs option sounds good to me. Also IdlePriority doesn't make
> much sense to me at first glance though since A) when Mixxx is running,
> not much is idle and B) the analyzer blocks other things. I would
> suggest defaulting it to whatever is just above Idle.
>
> And RJ, can you speak to the analyzer-tied-to-GUI-refresh-rate thing? I
> distinctly remember analysis taking MUCH longer on Windows when we
> reduced it from ~200Hz to 30, then you bumped it to ~60 for the time
> being to make this less noticeable.
>
> --
> You received this bug notification because you are a member of Mixxx
> Development Team, which is subscribed to Mixxx.
> https://bugs.launchpad.net/bugs/888817
>
> Title:
> Loading info takes too long (waveforms and BPM)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mixxx/+bug/888817/+subscriptions
>

[Owen Williams (ywwg)]

Is there an easy way to clean out the database so that it redoes all of the track analysis? Just resetting "header_parsed" doesn't do it. It's hard to test the timings if I keep having to load different tracks.

[RJ Skerry-Ryan (rryan)]

You need to clear the bpm, beats, beats_version, wavesummaryhex, and
replaygain columns to blank to trigger a full re-scan of overview, BPM, and
replaygain.

On Tue, Dec 6, 2011 at 2:03 PM, Owen Williams <email address hidden> wrote:

> Is there an easy way to clean out the database so that it redoes all of
> the track analysis? Just resetting "header_parsed" doesn't do it. It's
> hard to test the timings if I keep having to load different tracks.
>
> --
> You received this bug notification because you are a member of Mixxx
> Development Team, which is subscribed to Mixxx.
> https://bugs.launchpad.net/bugs/888817
>
> Title:
> Loading info takes too long (waveforms and BPM)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mixxx/+bug/888817/+subscriptions
>

[Owen Williams (ywwg)]

Maybe we should do that as part of Reload Metadata? I never understood quite what that feature was for if it didn't clear out all those values.

[toomuch (toomuch)]

Maybe this bug #688916 will give more info on my setup.

[toomuch (toomuch)]

Anything new?

[toomuch (toomuch)]

Please close this bug. It seems it was a performance problem. Though it was different when I started using Mixxx. Unfortunately I can't tell since when this behaviour started. I now have another laptop.

[Swiftb0y (swiftb0y)]

Mixxx now uses GitHub for bug tracking. This bug has been migrated to:
https://github.com/mixxxdj/mixxx/issues/6103