Support NI Stems

[Owen Williams]

See http://djworx.com/stems-open-format-multi-track-songs-from-ni

Obviously there is nothing concrete yet -- no format, no specs, etc, but the basic outline is known: four tracks in one file that play in sync and can be independently looped / cued / etc. Supporting these files requires some analog for Traktor's remix decks, which seem to just be like four regular decks or samplers that are always at the same bpm.

Full support will require major work, but we might be able to get bare-minimum support in (load either full mix or individual stems) with hopefully not too much effort. Likely the actual audio encoding will be something preexisting like FLAC.

Actual design should happen on https://blueprints.launchpad.net/mixxx/+spec/ni-stems

[Sean M. Pappalardo (pegasus-renegadetech)]

Why limit it to NI? We were taking about doing this years ago: http://mixxx.org/forums/viewtopic.php?f=1&t=3018

[Sean M. Pappalardo (pegasus-renegadetech)]

Also https://blueprints.launchpad.net/mixxx/+spec/multi-channel-files

[Owen Williams (ywwg)]

NI announced a specific file format for this situation, so this bug is about what would be required to support that format. Yes some of the features needed to support this format could also apply to multichannel files in general.

[Sean M. Pappalardo (pegasus-renegadetech)]

I suspect all of the same blueprints apply. Once that work is done, this bug would just involve adding support for another file format. (I just wanted to call your attention to the existing blueprints so you don't duplicate work.)

[Owen Williams (ywwg)]

Cool good to know

[Ferran Pujol (ferranpujol)]

I've thought about some kind of remix decks counterpart several times last year. Remix decks are nothing more than 4 mini decks (with its individual filter and volume controls) that are routed to the same standard mixer channel. We support n-decks each with its full mixer channel, so we have actually already overcome that. Even if we provide a GUI with 32 decks with few controls for space reasons, users could still map their controllers to use the full mixer controls for each Mixxx deck.

The other two main features of remix decks are:
1) The 4 mini-decks are hold together under a "meta deck". Tracks or samples can individually be loaded to each mini deck. But besides that, a special library entity that is nothing more than a group of tracks can be loaded to the "meta deck". That automatically loads the corresponding sample to each mini deck.

2) Each mini deck is an ableton-like collection of slots, instead of a single deck. There can be up to 16 samples loaded to the same mini-deck that can be alternatively triggered.

Possible solutions:
1) Support for NI Stems or other multi channel file format seems like the way to go to store related tracks together in the library easily. However, we are missing this "meta deck", or the way to group 4 decks together so they can be loaded with related content at once.

2) We don't really need this ableton-like slots if we have a solid work-flow of cue-points with this multi channel files.

Additional "secondary" remix decks features:
The 4 mini decks in a remix deck are always in sync. Also, if the meta deck is scratched, all its mini decks are scratched accordingly. The same happens with transport and looping controls.

[Daniel Schürmann (daschuer)]

I think you N-Deck approach does not relay fit to Stems files because of the missing common transport region.
Is it possible to think about stamps like AC3 files where we always have a common transport?

Currently Mixxx decks support the left and right channel.
In a future version we may add support for n-additional channels.
Has a Stam file 8 channels from this point of view?
2 x vocals / 2 x drums / 2 x base / 2 x guitar
compared to AC3 with 6 channels left / right / center / sub / rear left / rear right

For me this use case make sense:
Load a Stem file in one Mixxx deck. Normally play it with all channels mixed down to stereo. For Stem Mixing each deck has a advanced Mixing region where you can apply gain / EQ / effects to individual Stam channels or select only a set of channels from a stam file.
So you are able to produce fancy transitions by adding just the vocals to an other track, or loop just the drums and base channels.

If you need individual transport for each stam channel for a fancy remix, you can load the same stam file to a second deck and select a different set of channels. Since bot decks have the same beatgrid, playing them in sync should already work.

[Daniel Schürmann (daschuer)]

We will have a problem to distribute a NI-Stem enabled version of Mixxx, because it is mp4 based.
Maybe we can push an royalty free stam format.

[Be (be.ing)]

Here is a video from NI talking all about Stems: https://www.youtube.com/watch?v=MntkrfNAQ6w

It will use an MP4 container with either AAC or ALAC codecs. Additionally, there will be some metadata specifying details about compression for each stem to mimick mastering when they are mixed differently. It seems to be the responsibility of the program playing the Stems file to apply the compression on-the-fly.

[Be (be.ing)]

A developer toolkit will be available from NI sometime this summer from here: http://www.stems-music.com/stems-is-for-developers/index.html

They are inviting developers who plan to implement Stems support to get in touch with them by emailing <email address hidden>

[Owen Williams (ywwg)]

Yeah when I watched the video I realized I could hear sidechaining, and suddenly the whole thing got way more complicated.

[Owen Williams (ywwg)]

This screenshot reveals some of the details about the (sidechain?) compression options available: http://www.residentadvisor.net/news.aspx?id=31019

[Ferran Pujol (ferranpujol)]

The steam creator is already available for download:
http://www.stems-music.com/stem-creator-tool/
El dia 24/08/2015 20:45, "Owen Williams" <email address hidden> va escriure:

> This screenshot reveals some of the details about the (sidechain?)
> compression options available:
> http://www.residentadvisor.net/news.aspx?id=31019
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1438746
>
> Title:
> Support NI Stems
>
> Status in Mixxx:
> Confirmed
>
> Bug description:
> See http://djworx.com/stems-open-format-multi-track-songs-from-ni
>
> Obviously there is nothing concrete yet -- no format, no specs, etc,
> but the basic outline is known: four tracks in one file that play in
> sync and can be independently looped / cued / etc. Supporting these
> files requires some analog for Traktor's remix decks, which seem to
> just be like four regular decks or samplers that are always at the
> same bpm.
>
> Full support will require major work, but we might be able to get
> bare-minimum support in (load either full mix or individual stems)
> with hopefully not too much effort. Likely the actual audio encoding
> will be something preexisting like FLAC.
>
> Actual design should happen on
> https://blueprints.launchpad.net/mixxx/+spec/ni-stems
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mixxx/+bug/1438746/+subscriptions
>

[Owen Williams (ywwg)]

I'm in contact with the Stems people, I'll let you know what I find out.

[Timmie (timmie)]

@daschuer
>We will have a problem to distribute a NI-Stem enabled version of Mixxx, because it is mp4 based.
>Maybe we can push an royalty free stam format.

Would FOSS not be allowed to use mp4?

@ywwg:
any outcome from the discussion with Stems people?

[Daniel Schürmann (daschuer)]

A Stem file is a five tracks mp4 container usually encoded in AAC 256 kbps VBR or ALAC.

If you like to distribute Mixxx with an AAC decoder like libfaad, you have to pay for the included patents.

ALAC was released under the Apache license which contains a patent grant.
ALAC was just added to libsndfile which is already used by Mixxx.

[Be (be.ing)]

The Stems SDK is finally available, but it's not free software:
http://www.stems-music.com/sdkagreement/

:(

[Be (be.ing)]

There's still the possibility of reverse engineering or clean room design, but we should seek a lawyer's advice before trying that.

[Sean M. Pappalardo (pegasus-renegadetech)]

That agreement reads to me that the Stems format itself is open, but the DSP processing library is the licensed portion.

[Be (be.ing)]

I don't think so:
Any information supplied by Native Instruments or obtained by Recipient, as permitted hereunder, may only be used for the purposes as described herein and may not be used to create any software which is substantially similar to the expression of the software provided.

I don't see any documentation for the format available without agreeing to that.

[RJ Skerry-Ryan (rryan)]

We can reverse engineer this format from files alone -- anyone who wants to be involved in supporting stems in Mixxx, do not download or agree to the SDK terms.

[RJ Skerry-Ryan (rryan)]

> We will have a problem to distribute a NI-Stem enabled version of Mixxx, because it is mp4 based.
> Maybe we can push an royalty free stam format.

Publishing our own open format is certainly an option we should consider if NI is unwilling to make the terms of using their format acceptable.

MP4/AAC is no problem on Windows and Mac. It's just problematic on Linux -- and some distros will give their users the option to include libmp4v2 / libfaad. That's the choice of the user and the distro. At least we can offer Debian packages that depend on libmp4v2 via our PPA and website -- even if Debian cuts them out when included in their repos that's their choice.

[neufeind (neufeind)]

I just also found some older information on a wiki on the mixxx-site that tried to summarize work needed (or at least one way to do it). Is that still up-to-date? Otherwise maybe it could be adjusted:
https://www.mixxx.org/wiki/doku.php/multi-channel_file_mixing

Thread on the forum:
https://www.mixxx.org/forums/viewtopic.php?f=1&t=3018

[Daniel Schürmann (daschuer)]

It is almost up to date even though it is form 2012.
Do you have fun to adopt some of the work?

[Michael Z Freeman (michael-z-freeman)]

I saw R J Ryan's comment above. What exactly is the conflict with their license / binary blob and how is that solved by reverse engineering ? Actually why do they even have some DSP closed source code of some kind at all ? I thought it was just MP4 with 5 audio streams in it so I don't see what wonderful IP invention they could be protecting. Anyway the alternative is to support a truly open STEM format (FLAC, and so on) and then its probably very easy for the Mixxx user to convert from MP4 STEM to FLAC (or whatever) STEM.

[Michael Z Freeman (michael-z-freeman)]

I just checked and did not realise that Mixxx can already import multi channel FLAC ? At least according to the bottom of this page: https://blueprints.launchpad.net/mixxx/+spec/ni-stems

I just did a quick test of Audacity and it imports all 5 stereo streams OK from MP4 STEM, some free one's and one free download from NI themselves. So Audacity have either found a way round the restrictive license or have reverse engineered the format.

[Be (be.ing)]

As far as I understand the files are standard MP4 files with multiple channels, but the DSP library is needed to properly process them for playback. I'm not sure what exactly it does.

[T D (clidx)]

I'm not so sure how enforceable their "if you've seen these SDK terms, you can't use any info we give you for anything other than what we want you to do with it" clause is. The format is absolutely nothing more than MP4 and AAC with no extensions to either; I would be very surprised if they were granted permission to sublicense MP4 in this manner. Also I had already figured out the implementation (lucky guess) before reading anything that NI themselves had written, having just found out about STEMs about an hour ago.
However I suppose this would need to be tested legally first to avoid lawsuit later.

Has there been any further thinking on this 3 years later?

[Be (be.ing)]

> Has there been any further thinking on this 3 years later?

No.

[Swiftb0y (swiftb0y)]

Mixxx now uses GitHub for bug tracking. This bug has been migrated to:
https://github.com/mixxxdj/mixxx/issues/7935