Waveform 2.0 is jerking: GPU V-Sync

Bug #981199 reported by Daniel Schürmann on 2012-04-13
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Daniel Schürmann
Daniel Schürmann
Daniel Schürmann

Bug Description

I have noticed that the waveform 2.0 is jerking on my system.

Ubuntu Oneiric 32bit
Pentium Dual E2180 @ 2.00 GHz
Radeon HD 4350 with radeon driver
lp:mixxx #3012

I am not sure if it is an hardware performance problem, because the CPU usage is at 40% the same as with waveform 1.0.
The selected and the current frame rate are 30 fps. It is possible to have both at 60 fps without additional issues.

The jerking happens about 3 times per second. It looks like if there where some frames refreshed with ~5 fps although 30 fps is selected.

This was tested with "Filtered (GL)" and Simple (GL).

Filtered (GLSL) produces no waveform, only | bars. If I can see this correct, at least without jerking.
"Filtered" has a frame rate of ~5 fps at all.

Related branches

Daniel Schürmann (daschuer) wrote :

Attached you find a sequence with a jerk.

I am not sure how significant is, but maybe it helps.

It was generated by desktoprecorder, converted to avi and picked single steps with avidemux.
It seems, that frame 3 is jumping to far ahead and because of this, frame 4 is jumping back.
In the last 3 frames the waveform stops.

I am really not sure if this is the truth or if this is the result from reencoding.

Daniel Schürmann (daschuer) wrote :

On other test:
Mixxx 1.11.0-alpha bzr 3011 x64; Ubuntu Maverick 64; DELL Latitude E6510; i5 M560 @ 2.67 GHz; 3.7 GiB RAM; GPU NVS 3100M with nvidia driver.

It was generated by desktoprecorder.
The single images where generated with:

ffmpeg -an -ss 0:0:2 -t 00:00:02.93 -i out.ogv -f image2 thumb%d.jpg

I can not confirm jumping on this system, but I can see a significant jitter.

jus (jus) wrote :

I can confirm with MacOSX 10.6.8 and Intel HD Graphics 3000 GPU there is a stutter in the waveform movement. I can reach max 30-40 fps. The fps dialog in the preferences shows a stalling here and then just as described in #1.

On a 1st gen Macbook with MacOSX 10.5.8 and Intel GMA 950 i can reach max 8-10 fps with 90% CPU
Ok its its old but runs Mixxx 1.10 and other audio soft just fine until.

Alban (albeu) wrote :

I'm seeing a similar problem, but mostly right after starting the playback. This seems related to the log feature which do some update in the library. Disabling the log solved the unsmooth start on my netbook.

RJ Skerry-Ryan (rryan) on 2012-04-30
Changed in mixxx:
milestone: none → 1.11.0
tags: added: waveform
Daniel Schürmann (daschuer) wrote :

Hi Alban,

thank you, for the bug report. I think your issue is slightly different because the original bug happens continuously.
I have just opened a separate bug Bug #991717, you may add a comment there.

jus (jus) wrote :

Unfortunately even with the latest 1.11 beta1 this bug is still valid on my system (see #3)

Changed in mixxx:
status: New → Confirmed
importance: Undecided → Medium
Changed in mixxx:
assignee: nobody → Daniel Schürmann (daschuer)
status: Confirmed → In Progress
Daniel Schürmann (daschuer) wrote :

I have prepared a solution that corrects the visual play position with the audio latency and the video frame rate.
Unlike the waveform 1.0 solution it does not use a separate polling thread. It bypasses the Qt event queue by using static visual play positions.

Unfortunately this solution suffers bug #884705.

The solution does not solve the entire problem. There are good results on my netbook and especially with long latencies but no visual improvements on my desktop.

See: lp:~daschuer/mixxx/daschuers_trunk

I think the other part of the problem might be solved by render vsync.

Useful information:

Daniel Schürmann (daschuer) wrote :

In #1 I was facing a jump back of the waveform.
This can actually occur due to a QT Bug
fixed in Qt 4.8

Daniel Schürmann (daschuer) wrote :

Port audio offers functions to synchronize the waveform to the Sound stream:

This is best to solve the issues from bug #884705.

Daniel Schürmann (daschuer) wrote :
jus (jus) wrote :

This bug is still with the latest trunk and makes the waveforms somewhat unappealing and hard to look at.
I noticed that Mixxx eats considerably higher CPU when switching from a 2 deck to a 4 deck skin.

Using the OpenGL Profiler on MacOSX 10.6.8 i found that the waveform frame rate in the Mixxx preferences is possibly per deck.

Examples (4 tracks playing, Filtered GL waveform):
2 Deck skin, Waveform setting = 10 fps, Measured = 20 fps, CPU = 25%
4 Deck skin, Waveform setting = 10 fps, Measured = 40 fps, CPU = 33%
2 Deck skin, Waveform setting = 30 fps, Measured = 60 fps, CPU = 32%
4 Deck skin, Waveform setting = 30 fps, Measured = 120 fps, CPU = 47%
2 Deck skin, Waveform setting = 30 fps, Measured = 60 fps, CPU = 38%
4 Deck skin, Waveform setting = 30 fps, Measured = stall (210 fps peak), CPU = 60%

Either the profiler is wrong, i use it just not right or something is odd here.

Playing a little more with the OpenGl Profiler:
More than 20 % GL Time is spend in CGLFlushDrawable() , invoked exactly when the frame rate drops as described in the original post.

Break after: CGLFlushDrawable()
 Context: 0x1051fa600
 GL_RENDERER: Intel HD Graphics 3000 OpenGL Engine
 kCGLCPCurrentRendererID: 16925441 (0x01024301)
 Virtual Screen: 0
 kCGLCPGPUFragmentProcessing: GL_TRUE
 kCGLCPGPUVertexProcessing: GL_TRUE
Function call stack:
 0: 0x7fff8136e26d in CGLFlushDrawable in <OpenGL>
 1: 0x10022b741 in WaveformWidgetFactory::refresh() at stl_vector.h: 400
 2: 0x100d5793d in QApplicationPrivate::notify_helper(QObject*, QEvent*) in <QtGui>
 3: 0x100d5ddc4 in QApplication::notify(QObject*, QEvent*) in <QtGui>
 4: 0x100aa017c in QCoreApplication::notifyInternal(QObject*, QEvent*) in <QtCore>
 5: 0x100d57b4c in qt_sendSpontaneousEvent(QObject*, QEvent*) in <QtGui>
 6: 0x100d0ee0c in QEventDispatcherMacPrivate::activateTimer(__CFRunLoopTimer*, void*) in <QtGui>
 7: 0x7fff813cdbb8 in _CFRunLoopRun in <CoreFoundation>
 8: 0x7fff813cbd8f in CFRunLoopRunSpecific in <CoreFoundation>
 9: 0x7fff8109a74e in RunCurrentEventLoopInMode in <HIToolbox>
 10: 0x7fff8109a553 in ReceiveNextEventCommon in <HIToolbox>
 11: 0x7fff8109a40c in BlockUntilNextEventMatchingListInMode in <HIToolbox>
 12: 0x7fff8216aeb2 in DPSNextEvent in <AppKit>
 13: 0x7fff8216a801 in -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] in <AppKit>
 14: 0x7fff8213068f in -[NSApplication run] in <AppKit>
 15: 0x100d10900 in QEventDispatcherMac::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) in <QtGui>
 16: 0x100a9f094 in QEventLoop::processEvents(QFlags<QEventLoop::ProcessEventsFlag>) in <QtCore>
 17: 0x100a9f444 in QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>) in <QtCore>
 18: 0x100aa1b2c in QCoreApplication::exec() in <QtCore>
 19: 0x1000e3ed2 in main at main.cpp: 302
 20: 0x00000003 in <mixxx>

Hope this is somehow useful to track this bug down.

Owen Williams (ywwg) wrote :

This is because each OpenGL widget has its own OpenGL context. This bug won't get fixed until we figure out how to share contexts among widgets, which has something to do with creating a large image buffer and stenciling it into separate pieces for each widget.

Daniel Schürmann (daschuer) wrote :

Yes, we can save time when using a single OpenGL context. This will also help to use the V-sync facilitys of open GL.

I have tried to nail down this bug and it turns out to be more then one overlapping issues.

1) refresh() is not called reliable in time, sometimes we miss a v-Sync slot.
** the best results might be achievable if we are V-synced by a kind of high prioriti multi media timer.
** I have played arround with this issue but wit no success.
** I Think this is caused by other events processed from the GUI Thread. Since we have only one GUI Thread they will not yield for the refesch() event.
** I have tried to use a dedicated open GL Thread but without a better result.
2) we have to be sure that the Pixel Buffers are swaped in time.
** I have moved the postRender() call to the beginning of refresh()
** and I have changed it in a way that QT swaps emediatly and not when it thinks its best.
3) we have to be sure, that the new frame is rendered in time
** I am not sure how to read the results from Jus. Is CGLFlushDrawable called for every frame?
** Does it take longer in certain situations so we miss finish rendering in time?
4) we have to be sure that the audio is in sync with the video.
** this is not the case in trunk version
** I have prepared a solution with a VisaulPlayPosition Class that is able to sync them better.

I think 2-3 might be be solved in my current lp:~daschuer/mixxx/daschuers_trunk but I still have not a significant improvement.
I am still looking for a good solution for 1! Any Ideas?

By the way:
We may also save time if we pre-draw the entire waveform and diplay only the needed part or only repaint the frams that are moving in the view.

Daniel Schürmann (daschuer) wrote :

I think a part of the solution could be a separate openGL thread.
Without it, the GUI thread waits for finishing every GL command and long functions GUI are preventing to render the next frame in time.


Thomas Vincent (vrince) wrote :

We are facing known issues here using opengl in separate viewports with rendering update triggered by Qt event loop.
Please stop saying it's a context sharing issue because waveform widgets are already sharing context.
Like Qt dev team is doing, we need to make bigger commitment like "going full opengl" to be able to take advantage of hardware acceleration and completely rethink how waveform and the complete GUI is rendered. We will do it anyway sooner or later when we gonna port mixxx to Qt5 ...

Smooth animation in Qt is an active research project, please see some recent dev blog about it :

Owen Williams (ywwg) wrote :

As per https://bugs.launchpad.net/mixxx/+bug/1030313 I was under the impression that context sharing is currently not functioning at all.

Thomas Vincent (vrince) wrote :

Last time I debug some stuff all widgets were sharing the same context properly. It could be broken by now ...
To make the context valid we need to 'show' the first OpenGl widget before trying to share its context with others. This is the most frequent mistake about invalid context.

RJ Skerry-Ryan (rryan) wrote :
Download full text (4.3 KiB)

RE: Thomas and Owen -- the context sharing was indeed broken before. It's fixed now in that every QGLWidget reports that it is sharing the context.

RE: all

I've been messing around w/ the waveform a lot and I have some thoughts I wanted to get your opinions on:

1) We should use repaint() instead of update() on the widgets in WaveformWidgetAbstract::render.

update() will schedule the widget to receive a paintevent while repaint() will immediately repaint it. update() is basically provided so that you can call it many times per second and it will only repaint the widget once. We are running a rendering timer so we are already careful to only repaint it at our update rate so I think the extra delays caused by update() aren't necessary.

I'm also considering

2) We should turn VSync on

I've found that turning on vsync/swapInterval helps reduce tearing and corruption a fair amount. I originally turned this off thinking it was a performance issue but it reduces the tearing I see.

3) We should render in a thread (in Mixxx 1.12)

Rendering in a thread helped the responsiveness of the waveform a ton for me. I could scroll the library back and forth as fast as I could and the waveform stayed solid at 40fps. I had to turn off almost all the rendering steps that use QPixmap since that isn't safe for use in a thread. Now that we are switching away from QPixmap to QImage this should make it easier to switch to a thread.

If all the widgets share the same GL context then it's fine to use them in the same thread and as of Qt 4.8 it's safe to use a QPainter on a QGLWidget from a thread:

Daniel -- you mentioned you tried this. Did you see similar improvements in immunity to the Qt main thread being busy?

4) Non-smooth movement of the waveform. I call this "lagging" (as opposed to jerking, which I believe is a separate problem)

As you surely see, the waveform does not move smoothly. It moves with intermittent sluggishness.. every few seconds it will slow down or speed up a little bit. The slowdown seems much less sharp than the "jerking" so I think it's a separate issue.

To test what causes this, I created a GLWidget class in my stop_being_a_jerk branch. Look at LateNight 1280x800 to see it in action. It just draws parallel lines and moves them forward. The GLWidget always animates smoothly and you can create 4 of them and they all still animate smoothly at the desired framerate.

An important tool in testing was the ability to separate Mixxx's engine from the waveforms. I unhooked the playposition in WaveformWidgetRenderer and just incremented playposition a fixed amount each time. The result for me is it moves smoothly and with no apparent lag. It still jerked every now and then and still had visual tearing and corruption (which I found was improved with turning on vsync). See the attached playposition.patch.

What I conclude from this is that waveform lagging is caused by numerical issues on our side, not by Qt or OpenGL.

5) Jerking

I can't figure this one out. Every now and then the waveform shifts violently forward or backward as in Daniel's screenshot. I've checked that various differ...


Daniel Schürmann (daschuer) wrote :

Hi RJ,

Thank you for your analysis. Here are my comments:

> 1) We should use repaint() instead of update() on the widgets in WaveformWidgetAbstract::render.

> 2) We should turn VSync on
Yes! This is the only way to know when the frame is displayed on the screen
Unfortunately we hit the mess of different VSync methods in different driver versions of different vendors and OSs.

> 3) We should render in a thread (in Mixxx 1.12)
This could be an options if all used drivers where bug free, which seems not be the case now a days.
I don't have managed to run a threaded version with QT and my accessible test machines :-(
For now, I would prefer the other way round: to ban all long unbreakable (uncooperative) slots into a worker thread.
All slots longer than ~5 ms are critical in those terms.
We should consider to introduce a database worker thread. (This might solve other db issues as well)

> 4) lagging
When I understand it right, I have solution for this in my branch lp:~daschuer/mixxx/daschuers_trunk.
The visualplayposition and vsyncthread class are able to sync the Display and Sound. I am still fighting with the right Vsync approach for the different drivers. But at least for the Linux mesa drivers I have a working approach.

> 5) Jerking
> shifting backwards.
IMO this must be a QT event queue issue, I have tought this was solved with QT 4.8 (Comment #8). Did you see it in Qt4.8 as well?
Bypassing the Control Objects and the QT event queue by my visual play position class solves the problem.

* lagging: un-smoothness due to missing frames caused by overloaded CPU
** move long slots drom the GUI Thread
** Improve rendering speed
* jerking: un-smoothness due to wrong frame at wrong time
** VSync rendering and sync with Soundcard
** Sync all timer driven GUI slots to VSync
** Bypass Control Objects and Qt Event Queue

Owen Williams (ywwg) wrote :

I agree with the discussion so far. Interestingly I've never seen the jerking you guys describe (certainly nothing like the screenshot). VSync is critical and as long as we're updating every widget for every sync, that should be on. For a long time vsync was disabled because the contexts were broken. (The spinnies also need to be checked that they are using the same context, btw).

As far as waveform performance goes, how hard would it be to do a scrolling/damage model instead of full repainting on every frame? If the waveform is scrolling left, and the framerate is high, I could imagine the paint could be vastly simplified:

1. take previous render, shift it left X pixels
2. draw width-X new columns of waveform.

I bet there are even APIs to do the scrolling since it's such a common use-case. And no, this can't work when the zoom level has changed, but for most controller-based mixing redrawing the whole waveform is a waste of time.

Daniel Schürmann (daschuer) wrote :

Setting glFormat.setSwapInterval(1) is not that convenient as it might looks.
With current trunk version I have now a maximum frame rate of 20 on my Linux desktop with ATI GPU and radeon driver. :-(

On Linux this end in a call to glXWaitVideoSyncSGI which lets the calling task sleep until a Vsync happens.
The implementation is GPU/Driver depended. There are cases reported where glXWaitVideoSyncSGI might burn CPU cycles.
I have looked up source from other applications and I have found many workarounds for issues with it.

IMO the only case where it is useful if we have a single thread for each widget.
I have tried to test that within the Mixxx codebase but I have not managed to create a version without segfaults on all of my testing devices.

The current mesa drivers has a more intelligent anti tearing approach which simply avoid swapping the buffer during vsync.
This is the default setting in current Linux distributions .
I have successful managed to measure it out and sync the rendering against it.

But for Mixxx we need a solution for all platforms.
Thats what I am working on.

Unfortunately my work is not in a cleaned up condition, but If you like you can give it a try.

Owen Williams (ywwg) wrote :

Yeah I also get the 20fps problem -- each widget is getting its own vsync. This is with nvidia proprietary drivers.

RJ Skerry-Ryan (rryan) wrote :

OK, since I believe OS X is always composited with OpenGL, it's safe to turn on vsync for __APPLE__.

I wonder what experiences are like on Windows? I would imagine better than Linux.

RJ Skerry-Ryan (rryan) wrote :

Owen/Daniel : What versions of Qt are you seeing this on and what OS?

Daniel Schürmann (daschuer) wrote :

It was tested with Ubuntu Precise and Qt 4.8 and radeon driver 1:7.0.99

I wonder why apple has so different results.
Is QGLContext::swapBuffers() called from qgl_x11.cpp?
and is finally line #974 glXWaitVideoSyncSGI(interval + 1, (counter + interval) % (interval + 1), &counter); ?

It must be either glXWaitVideoSyncSGI is broken or something magical happens.
Can you measure the maximum time for glXWaitVideoSyncSGI with Mac?
On Linux it has a maximum value of up to 16 ms

Can you also attach the output of glxinfo?

The mesa drivers by default delays the buffer swap internal by default to avoid tearing. This is in generally a better approach but this will delay the following draw on the buffer until the first swap is actually finished. (glXSwapIntervalMESA)

If we rely on an CPU driven ms timer, we will always have those lasy waits (hopefully implemented as sleep()) in our GUI thread.
Because of that I have introduced an additional thread which managed the waiting for sync /mixxx/src/waveform/vsyncthread.h

RJ Skerry-Ryan (rryan) wrote :

OS X doesn't use X11 and it has its own API for interacting with OpenGL called the Apple Graphics Library (AGL). I believe glxinfo is only relevant to X11.


Qt uses this API as you'll find in qt/src/opengl/qgl_mac.mm:

void QGLContext::swapBuffers() const {
  // ...

If you search aglSwapBuffers on the page above, you'll find they mention "synchronizing" with the vsync. It's unclear whether they mean they will sleep if you call this but my guess is they have a vsync-sync thread inside of the AGL API that delays the swap until vsync. That way the user-thread is not impacted which would match what I am seeing.

RJ Skerry-Ryan (rryan) wrote :

I noticed the Qt blog post from earlier also suggested making a wait-for-vsync thread. Does this work? The problem on Linux is that we can't control the Qt version. On Windows/Mac we are Qt 4.8.3 which from my reading of various Qt blogs and release notes is the 4.8.x is the first version where it's truly safe to do things like this in other threads.

Daniel Schürmann (daschuer) wrote :

I think the lowest Qt version we should support is Qt 4.6
From Qt 4.8 QPainter can be used from other than the GUI thread.
In Qt 4.6 you can paint from different thread than the GUI thread if you don't use QPainter but pure gl calls.
It is required to call this for thread save X11 calls:
#ifdef Q_WS_X11
But anyhow. In my test I hit different bugs causing segfaults in the gl or driver library. So my preferred solution would be to do painting in the GUI thread and only do waiting in the vsync thread.

It sounds like this is implemented like glXSwapIntervalMESA.
What happens with Apple when you draw and swap with maximum speed?
Is the render rate fixed on 60 fps than?
If yes, when actually occurs the wait?
With glXSwapIntervalMESA it occours when you initialise the new Painter for the next frame.
Is this also the case in AGL?

I use this fact to sync Waveform to sound. See:

int GLWaveformWidget::render() {
    PerformanceTimer timer;
    int t1;
    //int t2, t3;
    // QPainter makes QGLContext::currentContext() == context()
    // this may delayed until previous buffer swap finished
    QPainter painter(this);
    t1 = timer.restart();
    draw(&painter, NULL);
    //t2 = timer.restart();
    // glFinish();
    //t3 = timer.restart();
    //qDebug() << "GLVSyncTestWidget "<< t1 << t2 << t3;
    return t1/1000; // return timer for painter setup

Owen Williams (ywwg) wrote :

I'm on ubuntu precise with compiz window managing.

So if I'm reading correctly, on newer versions of QT it should be possible to have a single thread whose job it is to wait for vsync and then a signal back to the main gui thread when that happens? So swapinterval would still be set to 0 but the timing of the calls would be driven by that vsync signal instead of the current qtimer. If that's the case, could we fall back to the qtimer for older versions of QT and just let there be tearing? The coding difference between the two shouldn't be too much in that case.

Daniel Schürmann (daschuer) wrote :

Yes Owen, correct. Those timer is ready in lp:~daschuer/mixxx/daschuers_trunk.
It has the advantage that we can have a higher resolution then qtimers ms.
Unfortunately it is based on the CPU clock while V-Sync is based on the GPU/Disdplay clock.

What are older version of Qt? I think these VSync strategies should all work on the "old" Qt 4.6. version. There is a bigger issue with the driver versions.

Looking at the Ubuntu Qt versions:

I think since Precise is LTS we should suggest users upgrade and only
support 4.8.1. Considering Linux is such a small portion of our user-base
and they tend to be more technically-abled this is probably fine. I think
it's a tough decision but ultimately we need to spend our limited developer
time pushing Mixxx (and the Linux desktop) forward and not waste effort on
fixing legacy bugs.

On Tue, Dec 11, 2012 at 12:05 PM, Daniel Schürmann <
<email address hidden>> wrote:

> Yes Owen, correct. Those timer is ready in
> lp:~daschuer/mixxx/daschuers_trunk.
> It has the advantage that we can have a higher resolution then qtimers ms.
> Unfortunately it is based on the CPU clock while V-Sync is based on the
> GPU/Disdplay clock.
> What are older version of Qt? I think these VSync strategies should all
> work on the "old" Qt 4.6. version. There is a bigger issue with the
> driver versions.
> --
> You received this bug notification because you are a member of Mixxx
> Development Team, which is subscribed to Mixxx.
> https://bugs.launchpad.net/bugs/981199
> Title:
> Waveform 2.0 is jerking
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mixxx/+bug/981199/+subscriptions

Daniel Schürmann (daschuer) wrote :

I think I was not clear enough: The issues are not a question of Qt 4.6 or Qt 4.8. Qt 4.6 is just fine if we use QPainter only from the GUI thread what we will most likely do for 1.11.
For 1.12 we can than drop Qt 4.6 support because Lucid falls out of support April 2013.

RJ Skerry-Ryan (rryan) wrote :

This is going to get fixed more satisfactorily in 1.12.0.

Changed in mixxx:
milestone: 1.11.0 → none
Daniel Schürmann (daschuer) wrote :

I would like to keep this open.
Currently, only the CPU timer synced version is merged.

A best version would be a V-Synced version with the GPU clock.
I have prepared some working code in:

The other as pecked are long "slots" in the GUI thread, that are causing jerking.
The non blocking database project is a great benefit for this.

summary: - Waveform 2.0 is jerking
+ Waveform 2.0 is jerking: GPU V-Sync
Owen Williams (ywwg) wrote :

Agree, the waveform is better but it's not vsynced for real.

RJ Skerry-Ryan (rryan) wrote :

Marking fixed in 1.12.0 and leaving open in 1.13.0 to get it off the active bugs list (I'm trying to make the list of open 1.12.0 bugs actually useful).

RJ Skerry-Ryan (rryan) on 2018-09-20
no longer affects: mixxx/2.1
Changed in mixxx:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Related blueprints