make BPM/waveform analysis data portable, **especially important for live distros**

[RAWRR]

When I've spent hours analyzing my collection I'd like to be able to grab that data and transfer it to other machines with the same or similar songs, so I don't have to do it all over again, especially if the machines I'm transferring it to are slower than the one I used for the analysis and scanning will subsequently take longer.

I recently tried to copy all the data from my analysis directory to its analogue on another computer hoping that it would be read, but the Mixxx of the recipient computer seemed completely unaware of it, no BPM data was displayed at all and the waveforms dd not seem rendered any earlier. I think the recipient Mixxx in this description was totally unaware of it even though it had all kinds of data sitting in the analysis folder.

**EDIT 2015 04 12**

It has come to my attention that this bug as originally suggested, meaning being able to move the analysis data, and then recognize it wherever it ends up, is an especially important feature for live distros deploying Mixxx. As the system now stands, a live distro cannot use pre-rendered analysis data except under very restrictive circumstances that require a lot of end-user customization, something live environments are not typically engineered for. At best, they would have to employ some kind of Union overlay ghosting the analysis directory into the .Mixxx directory in the live tree, and even then the paths would potentially cause problems.

Analysis data is very important. See bug #1431168.

[Daniel Schürmann (daschuer)]

Hi RAWRR,

thank you for the bug report.

My personal solution for this is like that:
* I have all my tracks on a external HD
* I start Mixxx with the cli option
mixxx --settingsPath to/my/external/HD
* so I can share all track metadata cues and playlists between two or more machines.

Does it suits to you as well?
Mybe we should make this feature visible in preferences?

@Max: have you more infos about the corner cases of this?

[RJ Skerry-Ryan (rryan)]

It should work if you use the same library /and/ the path to the files is
identical across the two machines.

On Thu, Nov 29, 2012 at 5:52 PM, Daniel Schürmann <
<email address hidden>> wrote:

> Hi RAWRR,
>
> thank you for the bug report.
>
> My personal solution for this is like that:
> * I have all my tracks on a external HD
> * I start Mixxx with the cli option
> mixxx --settingsPath to/my/external/HD
> * so I can share all track metadata cues and playlists between two or more
> machines.
>
> Does it suits to you as well?
> Mybe we should make this feature visible in preferences?
>
> @Max: have you more infos about the corner cases of this?
>
>
>
>
>
> ** Changed in: mixxx
> Status: New => Confirmed
>
> --
> You received this bug notification because you are a member of Mixxx
> Development Team, which is subscribed to Mixxx.
> https://bugs.launchpad.net/bugs/1084759
>
> Title:
> make BPM/waveform analysis data portable
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mixxx/+bug/1084759/+subscriptions
>

[RAWRR (rawrr)]

@ Daniel:

It can't suit me. Unfortunately, none of my externals have enough capacity to take responsibility for even most of my relevant collection, much less all. I've always wanted a master collection drive, external, it would order my files much more efficiently. But I'm not getting one for the foreseeable future.

Even so, I'm all for adding access to features via preferences, and I'm sure doing that would be useful to many people.

@ RJ:

I'm pretty sure that I had the same filepaths. But if I didn't, it would have been because of different usernames, and the analysis data is in the user folder. If I wanted to transfer my analysis date to any computer account with a different username there'd be a problem, even if it is true that identical paths will work in separate installations. Also, the tree changes for my tracks from computer to computer; sometimes I dj with a smaller usb drive and a borrowed computer, or use a usb if a computer that I'm using has different music than I want to dj with for that particular gig.

This could be solved partially by having the data stored in the installation directory if that directory is the Program Files folder, though of course that wouldn't mend the potentially varying paths for the tracks (unless the analysis data has some kind of hash to be able to recognize an mp3 or whatever type upon input). Also, files are personal data, and in the unlikely instance that multiple users existed in one dj-employed system, technically that analysis data would be personal and having it in a shared directory like Program Files would be a breach of compartmentalization.

It could also be solved by adding the data to the same directory as the tracks they pertain to, as mentioned earlier, though I know people could be loathe to clutter up often already cluttered music repositories with a bunch of strange files. But it still would solve all the issues I've stated, certainly, and make the data definitively portable, including the possible eventuality of the files being given to a completely different person.

Another option is to have the analysis data directory positionable from the options menu, as Daniel maybe refers (yes, it would also be cool to have the settings path be portable and changeable in the options gui). It would have to be coordinated with the request that Mixxx makes to the user for the music directory each time it starts in a strange place; the one caveat here is that if all else in this system remained the same, that tree within the music directory *would* have to be the same *within* the music directory the user points to. This is not a terribly unfriendly compromise, I would do it. And it is probably the simplest and most immediate solution. But there would have to be a mouseover in the relevant part of the dialog/preferences, or info somewhere, reminding the user that the tree must match the tree that the analysis was originally performed on or Mixxx won't read it.

I knew this one could point toward a potentially complex rearrangement. Just for everyone's information, I know there is a lot of high priority work to do and I don't necessarily expect people to jump on wishlist items like this immed...

[RAWRR (rawrr)]

Sorry, RJ, if it looked like that whole last part was just @ you, just the first paragraph after your @ was. I forgot to add a space or divider or something.

[Daniel Schürmann (daschuer)]

Hi RAWRR,

Thank you for your long post and the including ideas.

I think this Bug covers also some cases of this:
"database replication and mobile crates"
Bug #918233

[RAWRR (rawrr)]

Hmm... they do seem similar. Though they request different solutions, the ultimate need expressed seems the same. It still seems to me that your idea of having it user optional via preferences gui to set database (and settings) location is the quickest stable hookup, with any better options left to be implemented some other time in the future.

Seriously, sorry about the length of the post. I laughed in disbelief at it when I saw it after it posted 9_9

[RAWRR (rawrr)]

So another detail to this situation I've just discovered. I just deleted my mixxxdb.sqlite so that I could clean up my library, and now, after scanning my library again (not analyzing it, just scanning it) the analysis data is not there in Mixxx. Its still there in my analysis folder, all the hulking 3.27gb of it, but Mixxx is apparently oblivious.

This approximately duplicates the situation Mixxx would be in if I had another installation on another computer that had *all* the same file trees, same Mixxx version, everything. Huge quantity of data, no usefulness. Also, this implies that if you were to port your analysis data over to another machine with things as they are now, you would have to port the mixxxdb.sqlite that was present at the time of the analysis right on over, too.

So, yeah.

[RJ Skerry-Ryan (rryan)]

I just discovered '--settingsPath' does not affect the 'analysis' folder Mixxx uses so waveforms would not be used from a custom settings path. I fixed this in lp:mixxx/1.11 r3727.

I think that we have other bugs that cover things you mentioned RAWRR. For example, making library paths relative so that you can plug a library + Mixxx database into another computer and have it still be able to find its music.

It could be a lot easier but the waveform data is now 'portable' in the same sense that the database / config is portable by using a custom --settingsPath so I'll mark this fixed now. We could use some new bugs to cover how to make this process more user friendly :).

[RJ Skerry-Ryan (rryan)]

Eh, nevermind. I guess it's not fair to mark this fixed :).

If you use a custom library/config using --settingsPath on a new computer, the path to the track has to be identical.

So if you use /media/foo for your library on computer A, on computer B the tracks need to be in /media/foo as well. If they are then the waveform/beat data will load fine.

[RAWRR (rawrr)]

So let me repeat what I think I'm hearing back to you for confirmation:

Analysis data is now portable, but only providing that any track trees using that analysis data are identical to the ones used by the Mixxx installation the analysis was originally performed by?

If so, this is much closer.

As for user friendliness, I did mention already that I thought having to maintain consistently identical trees wouldn't be the worst of options, at least until use of relative paths can be implemented... I think it even to some extent might be intuitive to many users, at least those acquainted with paths/trees. And again, including a tooltip on mouseover in the preferences menu reminding this detail to the user seems like as complete a solution as can be bought at this time; text might read something like:

 "BPM and waveform Analysis data can be copied to and used with another Mixxx installation, provided that the relevant music tracks are stored in paths identical to those of the computer the analysis was originally performed with".

Anyhow, I'm amped that this is getting attention! I'll work on ideas for new bugs :P

[RJ Skerry-Ryan (rryan)]

@RAWRR -- yep, everything you said is correct.

The library scanner has a feature that is supposed to detect 'moved' folders and files. If your external music HD is mounted on one computer at /media/foo and on the other computer it's /media/bar then if nothing else changed and you open Mixxx with --settingsPath pointing to your settings folder that you brought over from the first computer, change the Library path in the Preferences to /media/bar, then hit re-scan. Mixxx should be able to detect that every file has moved and re-attach the files with their orphaned metadata (beatgrid/waveform/hotcues/playlists/crates/artist/title/album/etc).

I've never tried this though, but I think it should work.

[Daniel Schürmann (daschuer)]

Related:
Bug #1001299
Bug #918233

[Tuukka Pasanen (pasanen-tuukka)]

Should something like XSPF used with this bug? It supports extensions so cue points etc could but to those. More on XSPF would be found on: http://en.wikipedia.org/wiki/XML_Shareable_Playlist_Format or http://www.xspf.org/specs/

[RAWRR (rawrr)]

Ilkka Tuohela wrote at the mailing list:

"I think these two bugs should be implemented with a workflow like this:

- select a folder to export files to (‘create external mixxx library’ for example), let’s say
  /media/external/exported

- mixxx creates empty database to the location, for example
  /media/external/exported/.mixxx/mixxx.sqlite

- if successful, the exported library is shown in the mixxx sidebar with ‘unmount' icon

- user drags tracks from the library to the folder in sidebar, just like to a playlist
  -> mixxx copies the tracks to the folder and updates database in external directory
     -> the external DB must contain path in internal DB to match the tracks later
     -> tag changes must be timestamped to allow merging of changes back
  -> user can create folders, which are folders in the export directory, and mixxx has
      relative paths to the tracks in the exported directory

- user unmount the external library from sidebar
  -> exported database is closed and the entry disappears from mixxx sidebar

- on another (or same) mixxx machine, user chooses ‘import external library’
  -> the library is shown in mixxx sidebar, and changes can be imported (but maybe not
      automatically? Maybe option in sidebar context menu or in import dialog?)
  -> tracks can be played normally from the sidebar external library folder

I see no point exporting cue info or waveforms in json or xml formats, because the data is internal to mixxx."

[RAWRR (rawrr)]

This workflow has a lot of really good thinking, especially:

"- select a folder to export files to (‘create external mixxx library’ for example)"
"- mixxx creates empty database to the location"
"-> tag changes must be timestamped to allow merging of changes back"
(this latter feature might be better for telling which changes are the most up-to-date)
"- on another (or same) mixxx machine, user chooses ‘import external library’
  -> the library is shown in mixxx sidebar, and changes can be imported (but maybe not
      automatically? Maybe option in sidebar context menu or in import dialog?)"

But I did not understand these parts:

"-> user can create folders, which are folders in the export directory, and mixxx has relative paths to the tracks in the exported directory"
"- user unmount the external library from sidebar
  -> exported database is closed and the entry disappears from mixxx sidebar"
" -> tracks can be played normally from the sidebar external library folder"

And how will this work:

"-> the external DB must contain path in internal DB to match the tracks later"

...if portable internal DB locations, i.e. Mixxx installed on a usb drive, have some random internal DB path? Do we assert that this external DB must include a fallback/default internal DB path, and that any portable Mixxx must be installed at the root ,or some specific path, for its internal DB in a usb drive or other portable install location?

[RAWRR (rawrr)]

And from Daniel at the mailing list:

"Hi Tuukka, Hi Ilkka,

thank you for picking up this topic.

Here some unsorted comments:

---

XSPF:
It looks like this is extensible and because of this a good candidate to store Mixxx database info in a standard format. We may start by offering a simple xspf playlist export similar to what we do with m3u right now.
This may expand later with additional meta informations.

The next step is to allow to copy the tracks along with these playlists/crates. This way most use-cases are covered. I am just worrying that export might be pretty easy but not re-import.

---

/media/external/exported/.mixxx/mixxx.sqlite:
I like this idea very much, because it is exactly my current workflow using the --settingsPath Option.

I have all my tracks on my desktop PC and on a external HDD, big enough to carry the entire database.
Before I go to a gig, I just do:
rsync -av ~/Musik /media/Musik --delete
Since I use clementine for tagging, I search for tracks changed since last Gig, Put them in a Playlist and
Select "Reload Metadata"
After the gig I do
rsync -av /media/Musik ~/Musik

All changes that will streamline this work flow are very welcome :-)

Since there is a high risk of a corrupt sqlite DB when you remove
accidentally the USB cable. I put the whole .mixxx folder under git
control.
This means: the working sqlite DB should allways be on a fixed mounted
drive.

I am not scared about the possible bit incompatibility. We must Fix the
Mixxx version anyway, and if it turn out sqlite breaks the compatibility.
We may distribute the sqlite.so along with Mixxx.

---

"

[RAWRR (rawrr)]

*BUMP*

[Swiftb0y (swiftb0y)]

Mixxx now uses GitHub for bug tracking. This bug has been migrated to:
https://github.com/mixxxdj/mixxx/issues/6728