MIDI queue skips sending some data on Linux

[Sean M. Pappalardo]

In Linux (tested in Debian Squeeze and Ubuntu) using ALSA for MIDI (ALSA and OSS for sound,) with MIDI scripting, some MIDI output gets discarded (or corrupted, hence discarded) before it reaches the device, usually when signals are connected/disconnected and a series of commands is sent to the MIDI device. This is a problem on advanced controllers because it results in unchanged modes, incomplete LED fields, etc.

An example of this occurs at 2:08 in http://www.youtube.com/watch?v=qfkJnTqIeAw when the three sliders should have red dots in the center. Note that I had to switch to FX then back to EQ to get the desired behavior.

This does not happen on Windows.

Possibly related: Albert found with CoreMIDI that the calls were asynchronous and he had to queue them somehow. Perhaps this applies to ALSA as well?

[Sean M. Pappalardo (pegasus-renegadetech)]

This also an issue with the SCS.1 series since so much data is sent to them. Testing with the SCS.1m using FFADO's test-scs utility (in their trunk,) the VU meters on the mixer don't work (they just light the bottom LED) and when the init function asks the device to report all of its knob positions, hardly any of them take effect, and one is just plain wrong (deck 1 pitch jumps to max) which suggests that Mixxx's ALSA implementation also can't handle too much data on the input side either.

In addition, Mixxx doesn't connect its MIDI output to the SCS.1m (yet it does with the SCS.3d and other USB devices. I had to manually connect it using qjackctl's ALSA tab.)

I'm convinced bug 374665 is related to this as well.

None of this is a problem on Windows. In short, midiobjectalsaseq needs to be gone over with a fine-toothed comb, and I don't have the experience or knowledge to do it. Unless Albert's PortMIDI implementation will be ready soon...

[Albert Santoni (gamegod)]

I played around with increasing the input and output memory pool sizes for ALSA sequencer with no noticeable change. It turns out we were using non-blocking mode when opening the ALSA sequencer, which is what I think we wanted to use. I tried blocking mode, but also found no change.

The test case I'm using is:
1) Launch Mixxx
2) Hit EQ button on SCS.3d.
Result: The three middle lights don't light up. They should, but they don't.

One helpful thing I could use from you Sean is a tiny bit of troubleshooting. I'd like hard data showing that Mixxx isn't sending the correct MIDI messages. You can get data on this by:
1) Launching Mixxx
2) Run "aseqdump --list" and find out the port number for Mixxx.
3) Run "aseqdump -p [portnum]" using the port number you just found.
4) Switch to the EQ mode in Mixxx, and capture the output from aseqdump.
5) Compare that output with what you're sending in the script, and verify that there's a real mismatch. If you could show me what the expected sequence of MIDI messages is, that would help too.

Thanks!
Albert

[Sean M. Pappalardo (pegasus-renegadetech)]

Sorry for the delay. I just tested as you asked and the data IS being sent by Mixxx, as I expected, it's just not making it to the device intact. The output is exactly the same when it works and when it doesn't.

[Sean M. Pappalardo (pegasus-renegadetech)]

Also, funnily enough, it worked more reliably while it was dumping output to my console screen. I had to try a number of times before it glitched.

[Albert Santoni (gamegod)]

Ok, so this is an important discovery we've made then - I can't see
how this is a bug in Mixxx then. I was also worried that the messages
might be being delivered in the wrong order, but the data you captured
seems to show it in the correct order. I wonder if there's some
kernel-level MIDI buffer in the USB stack that's getting full and
dropping our messages. (The USB MIDI code might not let us send faster
than the MIDI spec's baud rate.)

I think devs in #alsa might be interested to hear about this....

Thanks a ton for the data, this is very helpful...
Albert

On Sat, Jun 13, 2009 at 9:25 PM, Pegasus<email address hidden> wrote:
> Sorry for the delay. I just tested as you asked and the data IS being
> sent by Mixxx, as I expected, it's just not making it to the device
> intact. The output is exactly the same when it works and when it
> doesn't.
>
> ** Attachment added: "MIDI output"
>   http://launchpadlibrarian.net/27885853/MIDI%20output
>
> --
> MIDI queue skips sending some data on Linux
> https://bugs.launchpad.net/bugs/342952
> You received this bug notification because you are a member of Mixxx
> Development Team, which is subscribed to Mixxx.
>

[Sean M. Pappalardo (pegasus-renegadetech)]

The problem is _greatly_ reduced when running a real time kernel (but it still exists.) Just tested on Ubuntu Studio 9.04 with rt and non-rt kernels.

[Sean M. Pappalardo (pegasus-renegadetech)]

I just compared the data sent by Mixxx and that sent by vkeybd and Mixxx is frequently sending two bytes when it is supposed to send three, causing many Note and CC messages to be corrupt. Someone with ALSA experience needs to look at this or risk me tearing midiobjectalsaseq apart!

[Sean M. Pappalardo (pegasus-renegadetech)]

I just committed a new and improved midiObjectAlsa in r2442. Testing with that for about a half hour, I didn't get any skipped data on the SCS.3d. The downside is that RawMIDI uses polling which takes up alot of CPU.
Could others please test with this implementation? Build mixxx with scons rawmidi=1 debug=1

[Sean M. Pappalardo (pegasus-renegadetech)]

Logged ALSA bug #4606: https://bugtrack.alsa-project.org/alsa-bug/view.php?id=4606

[Sean M. Pappalardo (pegasus-renegadetech)]

Waiting on ALSA developers' input.

[RJ Skerry-Ryan (rryan)]

This only happens to me when switching between FX and EQ, whether using rawmidi or alsaseq.

First the symptoms:

The center lights that are used for EQ parameter and FX parameter reporting sometimes do not light correctly on deck changes. If I switch from FX to EQ, and the eq lights do not light properly, then hitting EQ once more will light them correctly. (same for switching from EQ to FX). Switching from loop to EQ and loop to FX always light their respective lights properly.

I setup the FX*LEDS and EQ*LEDS functions in the script to print their arguments to sendShortMsg, and I also added a print in MidiObject::sendShortMsg to print the calls it was receiving from script. I found that when the lights do not light on deck changes, the command to light the LEDs is shortly preceded by a command to turn them off.

If I comment the lines that turn off the leds (around script line 167-169) then the problem does not happen! Switching between EQ and FX works fine. Of course, now the lights are sometimes lit improperly because I turned off the part where you turn them off.

This gives some important insight into the problem. It's almost as if the MIDI subsystem is treating the two events -- turning off the lights and turning them on -- as two simultaneous events, and the order that they are processed (or sent by the MIDI subsystem) to the SCS3d is non-deterministic.

One thing we should try is using the ALSA sequencer to timestamp the midi messages such that all the messages we send have different timestamps.

[RJ Skerry-Ryan (rryan)]

It's also worth mentioning that adding usleep(10) at the end of sendMidiShortMsg makes the problem go away for me.

[RJ Skerry-Ryan (rryan)]

Found another workaround that I committed to lp:mixxx/1.7

MIDI short messages are now queued using event queueing, and they are scheduled 1 nanosecond apart. System exclusive messages are still done via direct queueing. The sequencer is still opened in non-blocking mode.

Sean and I both confirm that the problem no longer occurs with the workaround. We should continue to look into the root cause behind this, but for release purposes, I'm marking this fix committed.

[RJ Skerry-Ryan (rryan)]

Marked as confirmed in Mixxx because we aren't really sure if this is truly fixed -- we just worked around it.

[Sean M. Pappalardo (pegasus-renegadetech)]

I still see this problem in trunk r2327 (pre-release v1.8.0). I found that the MIDI Device thread was executing script functions instead of the MidiScriptEngine one, and fixed that in features_scriptTimers (which is awaiting merge to trunk pending approval) and that helped, but the issue is not fully resolved even then.

[Sean M. Pappalardo (pegasus-renegadetech)]

features_scriptTimers was merged to trunk in r2367.

[SZÉKELYI Szabolcs (cc-mail)]

I'm still seeing something like this in 1.11.0 on Linux with Denon DN-SC2000. All of a sudden all Mixxx->DN-SC2000 communication breaks, LEDs don't update. The controller continues to work however, but without feedback.

[Sean M. Pappalardo (pegasus-renegadetech)]

That's strange. Mixxx 1.11 introduced an entirely new controller framework that eliminates the problems this bug mentioned.

Can you please run Mixxx from the command line with the --controllerDebug parameter, then use the controller until it stops working correctly, then attach a text file with the last few screens of console output?

[Swiftb0y (swiftb0y)]

Mixxx now uses GitHub for bug tracking. This bug has been migrated to:
https://github.com/mixxxdj/mixxx/issues/5116