I've tried Mixxx 2.3 and 2.2.4 on Fedora Linux tweaking as many settings as possible as well as on Ubuntu and there is just no solution to the issue I've seen of the beat being through around by the beatmatching process.... Latency is down and I can get the scratch function to work using external timecoders with Serato Vinyl Controls v2, but the AI keeps trying to advance the track forward. I paused the audio and tweaked some settings and once I resumed play the track jumped ahead about 15 to 30 seconds as if it didn't like being delayed. I thought it was just latency and some other issues for a few days now, but I'm really starting to think that the program is trying to do the thinking for the dj instead of letting the dj let loose...
When I was using a mouse and the virtual timecoders provided by the interface some years back, they were at least more responsive in releasing the audio on spot, but now I can physically see the software fighting me for position on the track as I try and simply move the vinyls back and forth.... It will be really smooth for a moment and then it ends up jumping ahead or all the way back to the beginning of the track.
Do you have quantize enabled on the deck? What about sync? Can you post a screenshot?