save and restore search queries for library

[RJ Skerry-Ryan]

Provide a way to save a current search query as a Smart Playlist/Crate that updates based on that query. Will be more useful once we have search operators. I actually think this only makes sense to do a Smart Crate because a Smart Playlist is essentially a crate.

[rob (another-rob)]

It would be useful if smart crates ran their queries once at the start of the mixxx session, and cached the results for the duration of the session, rather than running the crate-construction query every time you open the crate - I have a very large music collection, and searches of the whole collection aren't very fast - if these smart crates cached their contents somehow, that would really speed up searches within the crates, since I would only be searching a subset of my library.

[RJ Skerry-Ryan (rryan)]

I'd suggest something slightly different -- that they are cached but only update when tracks in the Mixxx library change. This way if you had a crate that was like [playcount>5] and you played a track in the same session bumping its playcount from 5 to 6 then it would appear in the crate.

[rob (another-rob)]

That works for me.

[Daniel Schürmann (daschuer)]

Bug #1228789

[naught101 (naught101)]

Just noting that both Zotero and Thunderbird do this quite well, if anyone is looking for UI inspiration.

Also, it would be really useful to have a couple of default smart crates - particularly "Songs not in a crate", and "Never played songs". They'd be really helpful for organising a library initially.

[Uwe Klotz (uklotzde-deactivatedaccount)]

More ideas on (and beyond) the 'smart crates' feature.

Dynamic content
- Low-level: Textual search query, directly editable by power users
- High-level (optional): Query editor
   - Parse an existing textual query
   - Conveniently edit and reorder the individual terms of the query
   - Format the results as a textual query
- Query execution: On-the-fly or cache results in database for faster access?
   - Mandatory: On-the-fly
   - Optional: Caching is an optimization and could be implemented later

Customizable column configuration
- Column configuration consists of
   - Ordering and width of the displayed columns
   - Column sorting
- Use the current column configuration of the library view for new smart crates
- Provide an option to reset the customized column configuration of a crate to the current column configuration of the library view
- See also: https://bugs.launchpad.net/mixxx/+bug/957693

Adhoc filtering
- Further restrict the contents of a smart crate by an adhoc query
- Example: Restrict the displayed content of a smart create to a certain bpm range by "bpm:>119 bpm:<131"

Mixxx query syntax: Add crate containment property
- Introduce a containment property that evaluates to true if the track is contained in the named (smart) crate
- Example query: "crate:Summer crate:Warmup rating:>=4"

Mixxx query syntax: Add new operators
- Currently Mixxx only supports an implicit AND operator
- Add support for explicit AND operator
  - How to achieve backward compatibility?
- Add support for OR operator
- Add support for NOT operator
- Support grouping of terms by braces: (term1 AND term2) OR term3

[Ferran Pujol (ferranpujol)]

A filter that evaluates to true for songs that are in a particular crate and multi-parent crates are essentially the same (A child crate is a crate that selects tracks that match its query AND its parents query). They both present a problem when cross-referencing crates, e.g:
Crate A = songs that are not in crate B
Crate B = songs that are not in crate A

This cannot be evaluated. Did you thought about this when you came up with this ideas? How could we solve this in a way that is efficient and is easy for the user to understand why he/she can't configure a crate the way he/she wanted? Some users might get confused about why their crate doesn't work. In fact this would introduce the users with the task of "debugging" a smart crate expression, as some complicated ones could lead to this kind of evaluation cycles.

If we limit the crates structure to be a tree (just one parent per crate), we can easily prevent this issues. In fact, in this case the only problematic operation would be to move a crate down one of its child branches. This can be managed just by breaking all the child relations of that create before moving it down one of its branches.

We could keep the “is in crate” filter if we don't support it on smart crates queries but just in the search field and as a library column.

So, is it really worth to support multiple parent crates? What are the use cases that would benefit from that and which couldn't be achieved in a reasonably easy way with a tree-like crate structure?

[Daniel Schürmann (daschuer)]

I am not sure if I get the issue correct, but I think it would be the best to separate the crates and smart crates.

Think of crates in terms of file system folders containing symlinks to the original files, manually setup by the user.
We may allow to manually group crates or create a new crate within a upper crate.
It sounds reasonable to limit this to a tree structure only, just like it would be in a file system.
.. or like a vinyl DJ that puts a sorter into his real crate.

Smart crates are for me sticky filtered filtered views of the library. They do not need to be applied on other filtered views.
They also do not need to apply on Crates or playlists, if we allow to set a filter "member of crate/playlist" Bug #1402133
Maybe we should name it not smart crate.

A related aspect of this bug is the hierarchical view of the library. We have some bugs that propose different nature of such a hierarchical tree.
I have collected the browse related bugs here: Bug #412453
Here is the multi level crate Bug #671632 and virtual folders: Bug #1228789

For me, a complete solution will feature this:
* Smart crates = Sticky filters (focused by this bug)
* A hierarchically organized Library view
** by file location (in addition to our RAW browse feature)
** by Artist -> Album -> Track
* Crates tree, allow to put on or more crate into a Meta crate
** allows to build genre trees

[Ferran Pujol (ferranpujol)]

Hi Daniel! Let me focus on the crates, which is what I'm working on :)

I agree with you that if this is implemented then we don't that kind of parent child relation of crates I described (child crates only match tracks that are in its parent).

Not to confuse this with crates folders which are just a display feature, i.e. they don't change the behavior nor the content of the crates. It is a feature common to both smart and regular crates. As a bonus, selecting a folder could show in the library all the tracks of its sub-folders together, but for this to be useful, smart crates and regular crates should not be in separate sections of the browser (I don't see a problem in that, provided that smart crates have a distinctive icon).

But my main concern now is the "member of crate/playlist" filter:
If crate A references crate B via this filter, and crate B references crate A, neither crate can be evaluated. How could we handle this elegantly and making it easy for the user to understand whats the problem? Imagine a user configuring a smart crate and seeing a pop-up saying that he/she just created a circular reference between smart crates...weird.

[Daniel Schürmann (daschuer)]

> Let me focus on the crates, which is what I'm working on :)

Focusing on a doable topic is a good idea and a requirement for a GSoC projekt.

All these crates natures are confusing. I hope we can clarify that somehow.
Did you mean "Let me focus on "smart crates"?

"Crates" are for me it is just a place where the user can collect manually some tracks.
Lets stick with this term, since it is what we have in Mixxx right now. It is implemented as an own persistent library association table.

In the bug title we have "Smart Crates". We may replace it a more more significant term like "Filter" or "Dynamic ... "
(Uwe please correct me) This will be stored as a search phase, which is applied to the library every time the user opens a "Smart Crate"

And a third term we have the fixed library "hierarchy". This might be an auto generated set of filters descending into the natural library structure.

At a "Crate", the user is responsible for each single track in the content.
At "Smart crates" Mixxx is responsible for the content and the user can control this by a fancy filter sting or similar.
In the "Hierarchy" we have a natural tree build form the track meta-data, not changeable by the user.

We may merge all this together, but IMHO it is a good Idea to keep it separate.

For me there is no need to have a filter on "Smart Crates". The user might wish to search for a track in a "smart crate", but such add-hock searches, basically extend the filter phrase of the original saved filter.

The "member of crate/playlist" feature can be limited to "crates". If we wish to allow to filter for "member of smart crate", this will just merge the two filter phrases of both smart crates.

"Smart playlists": I think we do not need them, since we have already the Auto-DJ crates and the Add Random feature.
It would be great if we can finally add a "Smart crate" to the Auto-DJ, but that is a separate issue.

[Ferran Pujol (ferranpujol)]

Hi Daniel, I agree in almost every point. Thank you for clarifying the concepts, I'll use your terminology from now on.

Regarding the member of crate/playlist filter, I'm sorry to keep insisting on the same thing but my concerns are about the user experience.

If users want to use this filter with a smart crate and we don't allow it, they might wonder why they can't. Sure we can inform of that in the manual or even show an info pop-up if they try to do this, but they might feel frustrated.

On the other hand, if we allow the filter to match smart crates, the issue gets even more difficult. As soon as an user tries to do some smart crate configuration that would lead to the reference cycle situation, we should warn him/her and let him/her know which smart crates are affected.

I asked if someone had any idea on how to handle this.

[Daniel Schürmann (daschuer)]

If users want to use this filter with a smart crate ...

I think we can allow it. The interface can be similar to the crate filter, but the backend will be different, since we will probably have no association table for a smart crate.
I hope Uwe can jump in here, since I am not a database expert.

[Be (be.ing)]

Saving and recalling arbitrary queries in the library redesign branch takes care of this. https://github.com/mixxxdj/mixxx/pull/1117

[naught101 (naught101)]

Are you sure you posted the right pull request there Be? It doesn't really discuss filtering/searching/smart crates other than filtering in relation to AutoDJ...

Daniel: re filtering on smart crates. I think this would be very useful. For example, if I have a "genre:dubstep" smart crate, I would like to be able to filter on key:Am. It doesn't make sense to have 12 key smart-crates for each genre though.

Also, it would be really useful to have "filters macros", which are saved filters that can be added to the current filter. For example a "mid-tempo" filter macro might be "bpm:>100 bpm:<120" or similar. The ability to go into any crate, then quickly apply that filter would be great (it's maybe not the best example, because I could do it with sorting by BPM, I guess. I feel like there are likely to be other examples though).

This could probably be done just as easily by having a search-bar history feature, which would operate similarly to a web-browser's form/address bar auto-fill functions (where it provides a drop-down with matches from previous entries).

[Be (be.ing)]

Yes, saving and restoring search queries accomplishes the same thing but doesn't use the same terminology or interface as other software like Serato. IMO it is better because it could potentially be operated easily from the keyboard.

The "filter macro" idea seems really useful. I have been thinking it would be useful to be able to name saved search queries so typing those user-defined names would be a shorthand for the full query. So instead of using your mouse to open the menu of saved queries, looking for the one you want, then clicking it, you could just focus the search bar (Ctrl + F with en_US locale, not sure about other locales) and type "midtempo", potentially as part of a larger query. We may want to add a new syntax for invoking saved queries to clearly distinguish them from doing a plain text search, for example maybe "query: midtempo" or "q: midtempo".

Keeping a history for queries and autofilling queries are good ideas too! I think making the search query functionality similar to what users are already familiar with from web browsers is a good idea and could make all these functionalities quite intuitive. I don't think we should constrain the design to resembling web browsers because the purpose is not quite the same, but drawing that comparison and taking inspiration from the design of web browsers is a good idea.

[Owen Williams (ywwg)]

what's the status of this feature / gsoc project? Is it still alive?

[Be (be.ing)]

This is entangled with the massive library redesign branch which still has major performance issues. :(
https://github.com/mixxxdj/mixxx/pull/1726

[Owen Williams (ywwg)]

is there any way to break that huge PR into smaller pieces that can be committed one at a time? ~17000 line change is ... a lot

[Owen Williams (ywwg)]

or can we make it so the library is a runtime or compile-time selectable option? That way the new code can live in the tree but we don't lose the working one

[Swiftb0y (swiftb0y)]

Mixxx now uses GitHub for bug tracking. This bug has been migrated to:
https://github.com/mixxxdj/mixxx/issues/5575