Library - missing files after update to 2.0.0-rc1

[Max Beiersdorfer]

I updated to the new mixxx-2.0.0-rc1 release (git5678) on Mac OSX 10.9.5. After I updated and rescanned library, a lot of often used mp3s are now listed under the category "Missing files" and are excluded from the search function.
The paths are still correct. I can rescan the library, delete and recreate a new library without my often used mp3s being inserted correctly. They are always missing. But if I go to the properties of one of these mp3s listed under "missing files" I can read and edit the mp3 tags.
When I go back to the official 1.12 beta release (git5442) and load my backed up library files everything's fine! With the old beta (git5442) I can rescan the library and all of my mp3 files are listed correctly.

MacBook 13" Aluminium, Late 2008
Operating system: MAC OS X 10.9.5
CPU: Core 2 Duo 2 GHz
RAM: 4 GB DDR3
Video: NVidia Geforce 9400M
Sound: Hercules DJ Console RMX 2

[RJ Skerry-Ryan (rryan)]

Hi Max -- this doesn't sound good at all.

I think you're the second person to report this -- unless you're the same person posting on our Facebook page about it.

Would you be willing to share your Mixxx database file?

It's stored in ~/Library/Application Support/mixxxdb.sqlite

You can upload it here as an attachment or email me directly <email address hidden> with it.

[RJ Skerry-Ryan (rryan)]

If it wouldn't be too much trouble, it would be super useful to see a log of the first run of Mixxx 2.0.0 that marks all these files missing.

1) Revert to backed up library.
2) Run with mixxx-2.0.0-rc1 release (git5678), rescan library, shut down
3) Upload ~/Library/Application Support/Mixxx/mixxx.log and ~/Library/Application Support/Mixxx/mixxxdb.sqlite

It would also be nice to see the copy of your backed up library to see what it's like in good condition.

[Max Beiersdorfer (max-beiersdorfer)]

Hi RJ Ryan,

I posted this problem on Facebook, too. So I think I'm the same person. I'm using Mixxx for 2 years now and I often updated to nightly builds etc.
There were never problems concerning the library.

First of all, here's a copy of my backed up library (good condition) as attachment.

[Max Beiersdorfer (max-beiersdorfer)]

...And now here comes the log-file after running mixxx-2.0.0-rc1 release (git5678) and rescanning the library.
A lot of missing files were detected, but the mp3 files are all at its paths and I can edit their tags.

[Max Beiersdorfer (max-beiersdorfer)]

...and finally the new database file with missing files.

[Sean M. Pappalardo (pegasus-renegadetech)]

Any chance you can test with Master (what will be Mixxx 2.1) also?

[Max Beiersdorfer (max-beiersdorfer)]

I can test it with Master, that's no problem. Which version/build should I take? Debug/release gitxxxx?

[Max Beiersdorfer (max-beiersdorfer)]

Ok, I tested it with master/debug/mixxx-2.1-alpha-pre-master-git5497-debug-macintel64.dmg

The same files were detected as "missing files". I have to say it's a very big library I think and *.wav and *.mp3 were detected as "missing files" at the same time.

[Uwe Klotz (uklotzde-deactivatedaccount)]

It looks like you that your library directory "/Users/stevejobs/Music" is a symbolic link that points to "/Volumes/Transcend"?

In the track_locations table all filenames start with "/Users/stevejobs/Music/...", but most directories start with "/Volumes/Transcend/...".

Unfortunately I don't recognize any pattern in the track_locations table why 15189 out of 24564 files have been marked as deleted.

[Uwe Klotz (uklotzde-deactivatedaccount)]

I compared this to my own Mixxx library:
* My library directory "/home/uk/Music/Collection" is a symbolic link to a directory on some external volume
* All filenames (column = "location") and directories (column = "directory") in the table "track_locations" start with "/home/uk/Music/Collection/..." consistently

Mixxx should use the absolute path (including symbolic links) for referring files in the library. The canonical path (without symbolic links) is only used at runtime and should not be stored in the database! I don't know if this is a specific OS X issue and how this happened? The directories in your old library are definitely invalid and don't match the library directory.

If you add files to your library by rescanning everything should be fine. Please don't use the browse view, since this has various issues as I recently discovered!

Dirty workaround to repair your library:
* Replace directory="Volumes/Transcend/..." with directory="/Users/stevejobs/Music/..." in the track_locations table
* Reset fs_deleted=0 in the track_locations table

There is already a bug for safer file handling: https://bugs.launchpad.net/mixxx/+bug/1451778

[Max Beiersdorfer (max-beiersdorfer)]

Hi Uwe,

thank you for the information! The reference "/Volumes/Transcend" is an external drive, which I connected to the MacBook a few times but doesn't use with Mixxx. I don't know where this symbolic link comes from!
Thank you for the workaround, but the next time I rescan the library, the files will be missing again. What I already tried was to start from zero (no library) and I added the path "/Users/stevejobs/Music" as a library search directory. After I scanned the library, the same missing files were detected as before.
I don't know if I understood you 100%, but I think that the wrong reference to the external drive then comes from my MAC OSX. Is that right? How can this be possible? My music folder is a normal folder with all my music files in it. I copied the music from another laptop via the external drive to the folder "/Users/stevejobs/Music" 2 years ago or so.
Other dj software or iTunes finds all my music files in that folder.

[Uwe Klotz (uklotzde-deactivatedaccount)]

I'm not familiar with OS X and its file system so maybe someone else has an idea of what is going on here?

Copying files from an external medium mounted as a volume seems to be the root cause. It's very strange that Mixxx still picks up a reference to this volume even if it is no longer available. The files should have been copied physically without a reference to the original files which doesn't exist any more. Maybe related to sandboxing in OS X?

[RJ Skerry-Ryan (rryan)]

> Maybe related to sandboxing in OS X?

Only mac app store builds are sandboxed so I don't believe that's related here.

[Sébastien BLAISOT (sblaisot)]

maybe related to https://github.com/mixxxdj/mixxx/pull/708

[Sébastien BLAISOT (sblaisot)]

Can you test with http://downloads.mixxx.org/builds/1.12/release/old/mixxx-1.12.0-beta1-1.12-git5431-release-macintel64.dmg

this is the last version before the 3 symlink-related pull requests that can impact you library (maybe) because yoru library looks like a symlink (different directory and track pathes)

[Sébastien BLAISOT (sblaisot)]

the two other PR I'm referring to are
https://github.com/mixxxdj/mixxx/pull/581
and
https://github.com/mixxxdj/mixxx/pull/587

[Max Beiersdorfer (max-beiersdorfer)]

I searched for symlinks on my macbook but there were no symlinks found concerning my music folder, which is used for the library scan. I never created any symlinks on my music folders to an external drive or something like that. I don't understand where the link to my external drive comes from, mentioned inside the database file.

I now testet the library scan with http://downloads.mixxx.org/builds/1.12/release/old/mixxx-1.12.0-beta1-1.12-git5431-release-macintel64.dmg, and with this version everything is fine. All files are found. Nothing is mentioned under "missing files".

Until now I used the 1.12 beta1 release git5442, which is newer than the mentioned download link I think and with this version everything's fine, too.

[Sébastien BLAISOT (sblaisot)]

ok, we made a progress.

We introduced some code to handle libraries with symlinks, and in particular circular symlinks.

You don't have symlinks, but as Uwe mentioned, your library just contains references to different pathes for directory and tracks which can be interpreted by this new code as "symlink". Probably you just found out a bug in that code the hard way.

Last working version is probably http://downloads.mixxx.org/builds/1.12/release/old/mixxx-1.12.0-beta1-1.12-git5557-release-macintel64.dmg if my assumption is correct.

probably git5571 will be the first build breaking your library (there is no available build between 5557 and 5571) because it is the first build including this change:
https://github.com/mixxxdj/mixxx/pull/708

[Max Beiersdorfer (max-beiersdorfer)]

Ah ok, thank you for the explanation.
I now tried to do the workaround described by Uwe (open mixxxdb.sqlite in textedit and replace directories) but it didn't work. After editing the database I get an error message when I start Mixxx:
"Ihr Datenbankschema konnte nicht auf Version 24 aktualisiert werden. Ihre Datei mixxxdb.sqlite ist möglicherweise beschädigt."

[Daniel Schürmann (daschuer)]

Fixed Database

[Daniel Schürmann (daschuer)]

You need an sqlite database editor to change the path. The attachment contains the fixed DB.

It really strange why it could happen that location and directory contains different base paths.
We use QString QFileInfo::absoluteFilePath() for location and QString QFileInfo::absolutePath() const for directory.
The symlink check code is not aware of this.

What happens if you remove the mixxx.sqlite file and start Mixxx 2.0.0 new?
Will it still have two different paths?

[Max Beiersdorfer (max-beiersdorfer)]

Hi Daniel,

thank you for fixing my database and for all the information! I tried to load your fixed library but there were still a lot of wrong directories in it, so that after the rescan a lot of files were detected as "missing files" again.

I guessed something like that, that I cannot edit the library simply opening it by textedit! But your info with with the sqlite database editor was very helpful. I took my original database file, downloaded the sqlite browser and changed all wrong directories to the correct paths. I don't know when and why so many directories were linked wrong.

Then I took the new mixxx-2.0.0-rc1 release (git5678) and rescanned the library. Now all my files are listed and found correctly!
Everything's fine now with the repaired database file!

I previously removed the mixxx.sqlite completely and inserted a new search directory -> my music folder! All files were found, nothing mentioned under "missing files".

So I think Mixxx works correctly and my database file was messed up and the "missing files" came up with the new version of Mixxx.
Now I know more about the library in mixxx, thank you very much! And sorry for the time!

I'm a dj and an electrical engineer, if I had more time, I would like to support/join the mixxx development team. It's a really great project and program. I exclusively use Mixxx these days!

[Owen Williams (ywwg)]

Thanks for your patience and suffering :). We'll keep an eye out for this problem and see if we can figure out some way to mitigate it so other users don't have to delete their libraries and start over.

[raskolnikov (raskolnikov)]

I am sad to say that... this affects me too! :-(

My library is a folder with symlinks to various locations. I wonder if that might be part of the problem. Also, in desperation, I renamed one of the symlinks and then renamed it back. It was not fixed. Also, updating the library takes ages, and I see debug messages about the library manager trying to detect moves for all the missing tracks.

My next step is to use the fact that now multiple library locations can be added to avoid the symlinks. I hope this helps. I don't want to delete the library because I have multiple playlists of sentimental value :')

[raskolnikov (raskolnikov)]

I attach my library, in case it helps.

My library is at /home/raskolnikov/media/mpd/music. It contains various symlinks:

$ ls ~/media/mpd/music/ -lh
total 4.0K
drwxr-xr-x 3 raskolnikov raskolnikov 4.0K Jun 22 2013 mixxx
lrwxrwxrwx 1 raskolnikov raskolnikov 36 Jun 7 2013 hd-alexandria -> /media/raskolnikov/alexandria/musica
lrwxrwxrwx 1 raskolnikov audio 25 Mar 4 2011 hd-raskolnikov -> /media/raskolnikov/musica
lrwxrwxrwx 1 raskolnikov audio 29 Mar 3 2011 media -> /home/raskolnikov/media/music
lrwxrwxrwx 1 raskolnikov raskolnikov 28 Jul 5 11:20 sync -> /home/raskolnikov/sync/music

[raskolnikov (raskolnikov)]

Update: adding the locations in the symlinks as separate library folders (even without removing the original merged location) did fix the problem! Great!

Note that it took *ages* for the library to update. On the command line I could see that most of the time was spent finding "successors" for the "moved" files. I had to leave the computer on all night because even after few hours it still needed more time.

Also, the library is even bigger now (130 MB sqlite file) and using the "Search" box is, admittedly, a bit slow. I feel that it is slower than in 1.12, but maybe I am biased since, as you can guess from me updating to 2.0 so late, I haven't DJed much in the last few months.

Btw, I love that Mixxx actually tries to detect file moves now! This has also fixed the path of some files that were _actually_ moved around and that where thus "broken" in old playilists.

Thanks a lot team!

Anyways, I do reckon symlinks should be supported. While I understand that this is only a power-user feature, people do us symlinks for this purpose and also for creating categories and stuff...

[raskolnikov (raskolnikov)]

Sorry guys for all the noise, but I have another update: now my library has lots of duplicate files (it seems, that the ones that were not missing before, are now duplicate). It is weird, because the symlink was expanded, so I have for many files, two separate entries with the same "location" in the library list. I am going to try something: go back and re-do the step that fixed the library, but removing the old library in the same step... let's see if it fixes it. Otherwise I might need some SQL savyness to manually fix it...

[Daniel Schürmann (daschuer)]

Thank you for your input. Mixxx does support symlinks.
Mixxx 2.0 receives some love fixing symlink issues.

The main discussion can be found here:
https://github.com/mixxxdj/mixxx/pull/708

I think I am suffering similar issues:

My library is on /home/daniel/Musik In Musik, there is a symlink, that point to may external HDD, which is sometimes available.

I move tracks around including remaining regularly, and end up with a long list of missing tracks. It bugs me for some time but I not had the time to work or investigate the fix.

Do you? Any help is very appreciated.

Unfortunately this bug, original labelled "Library - missing files after update to 2.0.0-rc1" slightly becomes a meta-bug, which cannot be handled.

It would be a great help, if you could try to separate you findings into several bugs.

If you have fun and time to help some more you may do the next big step towards a solution and provide a step by step list how to reproduce the issue.
This will help us a lot.

But be careful not to mess up your library even more. First backup your /home/<user>/.mixxx folder, to be always able to reproduce the old state. Mixxx supports changing the .mixxx folder by command line. So you might want to create a app starter with
mixxx --settingsPath PATH
where PATH is your alternative library path for testing.
Than you should prepare a testing folder structure that reflects your original library structure but only contains a few tracks.
When you have managed to reproduce an issue, please attach the mixxx.log file from your test run.

Thank you very much.

[raskolnikov (raskolnikov)]

Hi Daniel!

Thanks a lot. After all this trying, the state with many duplicates is the "best" state. I would like to work on a Python script or something to fix it -- remove the duplicates, etc. This script could evolve into a general tool for trying to diagnose and recover various potential library messups.

I will try to separate my problem into sepparate, reproducible bugs. Sadly I have lost my pre-1.9 .mixxx folder, but maybe I can achieve something starting from scratch and starting from nothing.

In my experiments these days, I have also noticed, that the library gets into "worse" non-recoverable states when the "Cancel" option is used during the scan, specially if it is used during the "detecting moves" phase.

However, before I do any further hacking involving digging in the code, I would like to learn a bit more about the SQL schema and the library code design. I remember reading some of that a couple of years ago from the dev mailing list, but do you have any relevant pointers to documentation at hand, that might guide me in the right direction?

Thanks!

[Daniel Schürmann (daschuer)]

Thank you for offering your work. A phyton tool for fixing library problems would be nice. But it would be much more straight forward to fix the issues in the C++ domain.

It will be probably also easier because you can access the original data.

Unfortunately we have not much docu about the library but I will do my best to support you.
Just ask.

[raskolnikov (raskolnikov)]

Sounds good. I will try to look deeper into the C++ code later this week. Maybe adding a "de-duplicate" phase after the "detect moves" phase would do the trick to fix my library. But I should also look at why the library got the duplicates in the first place and prevent that. I have other stuff do before though (I want to update the Mixco scripts to work with 2.0). I'll ping you on IRC when I look further into it, what's your nick there?

[Daniel Schürmann (daschuer)]

Thank you. My IRC name is daschuer, but I am only there by appointment via mail.
My mail is <IRC_name>@mixxx.org

Some time ago I have started to implement a re-locate feature but it was never finished.
I am still think that this is a nice addition, because all magic de-duplicate code can't beat the users decision in hard cases.

Maybe this helps you to get into the code:
http://bazaar.launchpad.net/~daschuer/mixxx/daschuers_trunk/revision/2859
http://bazaar.launchpad.net/~daschuer/mixxx/daschuers_trunk/view/head:/mixxx/src/library/dao/trackdao.cpp#L1316

Here are some general hints:
http://www.mixxx.org/wiki/doku.php/bugfix_workflow

[ronso0 (ronso0)]

@uklotzde Do you think this is fixed by now?

[Launchpad Janitor (janitor)]

[Expired for Mixxx because there has been no activity for 60 days.]

[Swiftb0y (swiftb0y)]

Mixxx now uses GitHub for bug tracking. This bug has been migrated to:
https://github.com/mixxxdj/mixxx/issues/8369