gstreamer-vaapi fails with vdpau-va-driver, Nvidia and Radeon GPUs

[nkef]

[Impact]
Gstreamer fails when gstreamer-vaapi hardware decoding is used trough vdpau-va-driver when using nvidia, nouveau, ati or radeonsi drivers with the following error message:
gst-launch-1.0: vdpau_decode.c:1264: vdpau_EndPicture: Assertion `obj_buffer' failed

[Test Case]
sudo apt-get install gstreamer1.0-vaapi mesa-vdpau-drivers vdpau-va-driver
gst-launch-1.0 test_file.mp4

The video should play now, and in any case no crash should happen now.

[Regression Potential]
The changes are based on git commits now included in the latest stable release (0.7.4), and does not change the way things work, but ensure that the buffers are correctly allocated and deleted.
Thus the regression potential is very low.

--

Gstreamer fails when gstreamer-vaapi hardware decoding is used trough vdpau-va-driver with NVIDIA binary drivers with the following error message:

gst-launch-0.10: vdpau_decode.c:1264: vdpau_EndPicture: Assertion `obj_buffer' failed

Example pipeline:
gst-launch-0.10 -vvv filesrc location=test.h64 ! queue ! h264parse ! queue ! vaapidecode ! vaapisink fullscreen=yes

The bug was produced in 13.10

vdpau-va-driver:amd64 0.7.3-2ubuntu1
libvdpau1:amd64 0.6-2
nvidia-319 319.32-0ubuntu6
gstreamer0.10-x:amd64 0.10.36-1.1ubuntu
gstreamer0.10-vaapi:amd64 0.3.6-0ubuntu4

The bug was also reproduced in 13.04
13.04, with default system packages.

A workaround only applicable to gstreamer-0.10
Example pipeline:
gst-launch-0.10 -vvv filesrc location=test.h64 ! queue ! h264parse ! queue ! vaapidecode use-ffmpeg=true ! vaapisink fullscreen=yes

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Marco Trevisan (Treviño) (3v1n0)]

I can confirm the same using radeon CPU, here's the backtrace I get.

[Martin Pitt (pitti)]

Sponsored utopic fix.

[Martin Pitt (pitti)]

Trusty SRU uploaded to review queue.

[Martin Pitt (pitti)]

Note that this FTBFSed in utopic, so you'll need a followup upload to fix that.

[Marco Trevisan (Treviño) (3v1n0)]

Thanks!

Do you have a log of the failure?
I suppose it's this issue http://lists.freedesktop.org/archives/libva/2013-June/001762.html right?

[Brian Murray (brian-murray)]

You can find the amd64 failure here:

https://launchpadlibrarian.net/174862648/buildlog_ubuntu-utopic-amd64.vdpau-video_0.7.3-2ubuntu2_FAILEDTOBUILD.txt.gz

[Brian Murray (brian-murray)]

Thanks for uploading the fix for this bug report to -proposed. However, when reviewing the package in -proposed and the details of this bug report I noticed that the bug description is missing information required for the SRU process. You can find full details at http://wiki.ubuntu.com/StableReleaseUpdates#Procedure but essentially this bug is missing some of the following: a statement of impact, a test case and details regarding the regression potential. Thanks in advance!

[Marco Trevisan (Treviño) (3v1n0)]

Sorry, I've added them now.

[Brian Murray (brian-murray)]

Hello nkef, or anyone else affected,

Accepted vdpau-video into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/vdpau-video/0.7.3-2ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

This also failed to build in trusty-proposed.

[Marco Trevisan (Treviño) (3v1n0)]

Ok, lp:~3v1n0/ubuntu/trusty/vdpau-video/remove-va-constants/+merge/219436 should fix it (it's the same patch I mentioned above and based on the same that it's included in utopic version).

[d.w. harks (dwink)]

Any update on this? Would love to see this make it into trusty-updates.

[Chris Halse Rogers (raof)]

Hello nkef, or anyone else affected,

Accepted vdpau-video into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/vdpau-video/0.7.3-2ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Mathew Hodson (mhodson)]

I don't know how to test this. This is the output I get when I run the test case.

$ gst-launch-1.0 Linux.mp4
ERROR: pipeline could not be constructed: Unrecoverable syntax error while parsing pipeline Linux.mp4 .

[Marco Trevisan (Treviño) (3v1n0)]

Check that you've all the main gstreamer-plugins-* packages

[Mathew Hodson (mhodson)]

I think the correct test case is to run

gst-launch-1.0 filesrc location=Linux.mp4 ! qtdemux ! vaapidecode ! vaapisink

I used LIBVA_DRIVER_NAME=vdpau to work around bug #964040, because r600_drv_video.so is not provided. The video played about once in five attempts. Other times, I got a black window and an error message.

$ LIBVA_DRIVER_NAME=vdpau gst-launch-1.0 filesrc location=Linux.mp4 ! qtdemux ! vaapidecode ! vaapisink
libva info: VA-API version 0.35.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'vdpau'
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/vdpau_drv_video.so
libva info: Found init function __vaDriverInit_0_32
libva info: va_openDriver() returns 0
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Got context from element 'vaapidecode0': gst.vaapi.Display=context, display=(GstVaapiDisplay)NULL;
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Caught SIGSEGV
Spinning. Please run 'gdb gst-launch-1.0 6561' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.

$ LIBVA_DRIVER_NAME=vdpau gst-launch-1.0 filesrc location=Linux.mp4 ! qtdemux ! vaapidecode ! vaapisink
libva info: VA-API version 0.35.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'vdpau'
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/vdpau_drv_video.so
libva info: Found init function __vaDriverInit_0_32
libva info: va_openDriver() returns 0
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Got context from element 'vaapidecode0': gst.vaapi.Display=context, display=(GstVaapiDisplay)NULL;
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
vdpau_video: VdpPresentationQueueDisplay(): status 3: An invalid handle value was provided. Either the handle does not exist at all, or refers to an object of an incorrect type.

[Marco Trevisan (Treviño) (3v1n0)]

That seems a different issue to me, if the crash underlined by the error:
  vdpau_EndPicture: Assertion `obj_buffer'

Does not happen to you, then this specific bug is fixed.
See also https://bugzilla.gnome.org/show_bug.cgi?id=729578

[Colin Watson (cjwatson)]

The verification of the Stable Release Update for vdpau-video has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package vdpau-video - 0.7.3-2ubuntu1.2

---------------
vdpau-video (0.7.3-2ubuntu1.2) trusty; urgency=medium

  * Add libva-constants.patch to fix FTB with current libva

vdpau-video (0.7.3-2ubuntu1.1) trusty-proposed; urgency=medium

  * Added patch-set from upstream to fix crash when using
    gstreamer-vaapi (LP: #1222790)
 -- Marco Trevisan (Trevino) <email address hidden> Mon, 30 Jun 2014 13:06:46 +0100