VA-API fails to initialize in a Gnome Shell Wayland session

[Daniel van Vugt]

The Intel i965 VA-API driver works well in Xorg sessions (Unity7 and Gnome Shell). But it refuses to load when in a Wayland session:

$ totem bbb_sunflower_1080p_60fps_normal.mp4
libva error: va_getDriverName() failed with unknown libva error,driver_name=(null)

$ env LIBVA_DRIVER_NAME=i965 totem bbb_sunflower_1080p_60fps_normal.mp4
totem: intel_driver.c:112: intel_driver_init: Assertion `VA_CHECK_DRM_AUTH_TYPE(ctx, VA_DRM_AUTH_DRI1) || VA_CHECK_DRM_AUTH_TYPE(ctx, VA_DRM_AUTH_DRI2) || VA_CHECK_DRM_AUTH_TYPE(ctx, VA_DRM_AUTH_CUSTOM)' failed.
Aborted (core dumped)

$ gst-play-1.0 bbb_sunflower_1080p_60fps_normal.mp4
Press 'k' to see a list of keyboard shortcuts.
Now playing /home/dan/Videos/bbb_sunflower_1080p_60fps_normal.mp4
Redistribute latency...
Redistribute latency...
ERROR Internal error: could not render surface for file:///home/dan/Videos/bbb_sunflower_1080p_60fps_normal.mp4
ERROR debug information: ../../../gst/vaapi/gstvaapisink.c(1482): gst_vaapisink_show_frame_unlocked (): /GstPlayBin:playbin/GstPlaySink:playsink/GstBin:vbin/GstVaapiSink:vaapisink0

WORKAROUNDS:

 * Use weston instead of gnome-shell; or
 * $ env -uDISPLAY totem ... # but creates corruption -> bug 1701463
 * $ env GST_GL_WINDOW=wayland totem ... # but creates corruption -> bug 1701463
 * $ env GST_GL_WINDOW=wayland gst-play-1.0 --videosink glimagesink ...
 * $ env -uDISPLAY mpv --hwdec --opengl-backend=wayland ...

ProblemType: Bug
DistroRelease: Ubuntu 17.10
Package: i965-va-driver 1.8.1-1
ProcVersionSignature: Ubuntu 4.10.0-22.24-generic 4.10.15
Uname: Linux 4.10.0-22-generic x86_64
ApportVersion: 2.20.5-0ubuntu4
Architecture: amd64
Date: Fri Jun 16 13:40:38 2017
InstallationDate: Installed on 2017-05-03 (44 days ago)
InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Alpha amd64 (20170502)
SourcePackage: intel-vaapi-driver
UpgradeStatus: No upgrade log present (probably fresh install)

[Sebastian Ramacher (s-ramacher)]

Bugs in gstreamer-vaapi and totem according to upstream.

[Daniel van Vugt (vanvugt)]

A fix for mpv is pending upstream:
https://github.com/mpv-player/mpv/pull/4556

[Daniel van Vugt (vanvugt)]

It seems we're loosely in a deadlock on this one...

The i965 VA driver is failing to load because it's using Xwayland. Xwayland only supports DRI3, while VA-API only supports DRI2. VA-API does support native Wayland, but that's not being tried by the video player apps as first preference. You could call it a bug that the apps choose X instead of Wayland, and that's easily fixable, but according to the mpv developers people prefer X in the mpv case because a native Wayland app has no titlebar or window decorations. But by defaulting to X, it uses Xwayland which does not support the same DRI interface as VA-API. Hence VA-API initialization fails.

So there are at least a few possible ways forward:
 * Implement DRI3 support in libva (unlikely to happen soon); or
 * Implement DRI2 support in Xwayland (unlikely to happen ever); or
 * Modify each player app to prioritize checking for Wayland before checking for X11 (rejected by mpv); or
 * Modify each player app to try vaInitialize multiple times (different display types) until one works.

[Daniel van Vugt (vanvugt)]

Here's my first attempt at a fix for totem. Although fixing this bug in totem then triggers corruption (bug 1701463) as well as higher CPU, so still needs more verification to see if the higher CPU is just a side-effect of poor plugin choice or related to the corruption...

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "fix-1698287-for-totem-via-gstreamer-vaapi.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Daniel van Vugt (vanvugt)]

Without sounding too defeated, I think it might be a good idea for us to not use the patch in comment #5 any time soon. The reason is that while the patch is correct, it triggers bigger issues in clutter-gst. And that starts to make me think I'm going too deep down this rabbit hole.

It probably doesn't make sense to fix gstreamer/clutter stuff if it's too time consuming, when we already know gstreamer lacks a variety of working demuxers, and is already so unpopular that it's not part of the long term plan anyway (https://wiki.ubuntu.com/IntelQuickSyncVideo).

The only real problem is that 'totem' is still our default video player. And if clutter-gst is a mandatory requirement for that then I either need to not give up yet, or we should consider changing default video players in future (ideally to something that's more purely FFmpeg instead of GStreamer).

[Daniel van Vugt (vanvugt)]

On second thoughts, if we keep this bug about decoding flaws, and deal with all the display flaws elsewhere then we might be able to close some tasks here...

[Daniel van Vugt (vanvugt)]

More progress: gst-play-1.0 fails on Gnome Shell Wayland only (and not in Weston) because of a mutter problem (or arguably the gstreamer-vaapi plugin assuming that mutter should support it):

https://bugzilla.gnome.org/show_bug.cgi?id=775698

[Doug McMahon (mc3man)]

vaapi in mpv in a wayland session was broken due to this commit
https://github.com/mpv-player/mpv/commit/bba08e38ff9842c4edf9da6a1d54066a01acd982

Testing here a reversion to allow, seems ok for at least me to date, as in
https://launchpad.net/~mc3man/+archive/ubuntu/mpv-tests/+files/mpv_2%3A0.25.0+git1~artful3_2%3A0.25.0+git1~artful4.diff.gz

[Daniel van Vugt (vanvugt)]

Here is the "big" patch I used in testing (includes makefile fix that I suspect archive doesn't need, but my machine does).

[Daniel van Vugt (vanvugt)]

Here's the preferred clean fix. Doesn't build on my machine, but I suspect it will build in archive :S

[Sebastien Bacher (seb128)]

The simple version of the patch builds fine in a ppa, sponsoring that one

[Doug McMahon (mc3man)]

Regarding the mpv in artful:
You can currently get some vaapi support in wayland session, you'd need to use
hwdec=vaapi-copy
Not as effective as hwdec=vaapi but I don't think they'll be changing that (commit noted), in near future.

[Launchpad Janitor (janitor)]

This bug was fixed in the package gst-plugins-bad1.0 - 1.12.2-1ubuntu2

---------------
gst-plugins-bad1.0 (1.12.2-1ubuntu2) artful; urgency=medium

  * Prefer native Wayland over X11/Xwayland if both are available. This
    fixes VA-API initialization failure in Gnome Shell Wayland sessions
    (LP: #1698287).

 -- Daniel van Vugt <email address hidden> Tue, 08 Aug 2017 17:08:09 +0800

[Daniel van Vugt (vanvugt)]

Doug,

A couple of my fixes have landed today, as well as ffmpeg 3.3 which fixes most of mpv's acceleration problems. However your comment reminds me there's another bug with mpv I need to open and fix... mpv will by default try to use the X11 display provided by Xwayland and ends up with vaapi-copy. You can work around that trivially and get hwdec=vaapi proper working again by hiding Xwayland:

   env -uDISPLAY mpv --hwdec myvideo.mp4

[Daniel van Vugt (vanvugt)]

I am going to work on additional patches so that by the time 17.10 is released all you will need is:

   mpv myvideo.mp4

[Daniel van Vugt (vanvugt)]

Actually comment #16 doesn't need a new bug. This bug will do fine for that. See comments #3 - #4.

[Daniel van Vugt (vanvugt)]

Doug, and all:

https://bugs.launchpad.net/ubuntu/+source/mpv/+bug/1708102/comments/3

[Daniel van Vugt (vanvugt)]

Update:

* totem is fixed by the gst-plugins-bad fix released today.
* mpv is fixed by the patch proposed in bug 1708102.
* chromium needs to either be ported to native Wayland, or for libva to be enhanced with DRI3 support. There are no workarounds for Chromium other than to avoid Wayland and use Xorg sessions.
* gst-play-1.0 needs an enhancement to mutter linked above, but the workaround mentioned above is also simple.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mpv - 0.26.0-3ubuntu1

---------------
mpv (0.26.0-3ubuntu1) artful; urgency=medium

  * Prefer Wayland over X11, so if both are present (e.g. Xwayland) then
    VA-API gets a display type that works (LP: #1698287)
  * Enable hardware acceleration by default (LP: #1708102)

 -- Daniel van Vugt <email address hidden> Wed, 09 Aug 2017 16:38:55 +0800

[Doug McMahon (mc3man)]

In Regards to mpv:
While vaapi has now been enabled please note that in a 'wayland' session mpv will not work properly with vaapi from the nautilus context menu. Only audio will work, no video.

The issue seems to be --player-operation-mode=pseudo-gui
Not sure what you all can do about this..

(- for my ppa users I'll provide a 2nd .desktop that will allow context menu/vaapi/opengl & produce expected results
Likely named mpv wayland-only or thereabouts, basically -

[Desktop Entry]
Type=Application
Name=mpv wayland-only
Icon=/usr/share/icons/hicolor/scalable/apps/mpv.svg
TryExec=/usr/bin/mpv
Exec=mpv --player-operation-mode=cplayer -- %U
Terminal=false
NoDisplay=true
Comment=Only for context menu in wayland session & vaapi
Categories=AudioVideo;Audio;Video;Player;TV;
MimeType=blah, blah,

[Doug McMahon (mc3man)]

Please ignore above comment, my bad.
(- to save some time on fresh install of 17.10 I copied mpv.conf from 16.04
It has an autofit-larger option for the psuedo-gui that isn't compatible with wayland.
Sorry..

[Iain Lane (laney)]

What is the plan for this "Forwarded: no" patch in gst-plugins-bad?

[Daniel van Vugt (vanvugt)]

It appears forwarding was blocked by uncertainty around comments made by the upstream reporter saying he would not accept it (although I'm not sure he himself is upstream):

https://bugzilla.gnome.org/show_bug.cgi?id=783169#c7

At the time I ignored that comment because it's not a good enough reason to reject the approach. The patch we use in comment #12 above is trivial and perfectly solves this bug. However the upstream comment is about the same approach not solving other issues that might arise from a user setting his/her own *_BACKEND environment variables.

I don't think that upstream comment should be considered a blocker. While valid, it is not pragmatic and the patch we use is perfectly safe and closes the bug. Upstream really is asking us to solve larger gstreamer design problems which Ubuntu isn't interested in solving right now.

[Daniel van Vugt (vanvugt)]

Andrea,

What errors do you see in VLC relating to this bug?

[Andrea Agnolin (agno94)]

Generate video h264 file (however I get the same behavior for every file I tested).
$ ffmpeg -f lavfi -i color=white;fade=t=out:d=10 -t 10 -c:v libx264 test.mkv

Within the "Ubuntu" session

Selecting the option "Decoder VA-API throught DRM"
$ vlc test.mkv
VLC media player 2.2.6 Umbrella (revision 2.2.6-0-g1aae78981c)
[000055d996caf9d8] core libvlc: Esecuzione di vlc con l'interfaccia predefinita. Usa 'cvlc' per utilizzare vlc senza interfaccia.
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 1
libva error: va_getDriverName() failed with operation failed,driver_name=i965
[00007f9ecc000c18] vaapi_drm generic error: Failed to initialize the VAAPI device
Failed to open VDPAU backend libvdpau_nvidia.so: impossibile aprire il file oggetto condiviso: File o directory non esistente

Within the "Ubuntu on Xorg" session

Selecting the option "Decoder VA-API throught X11"
$ vlc test.mkv
VLC media player 2.2.6 Umbrella (revision 2.2.6-0-g1aae78981c)
[0000563a5bfb79d8] core libvlc: Esecuzione di vlc con l'interfaccia predefinita. Usa 'cvlc' per utilizzare vlc senza interfaccia.
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns -1
libva error: va_getDriverName() failed with unknown libva error,driver_name=(null)
[00007f8ff4000c18] vaapi_x11 generic error: Failed to initialize the VAAPI device
Failed to open VDPAU backend libvdpau_nvidia.so: impossibile aprire il file oggetto condiviso: File o directory non esistente
QObject::~QObject: Timers cannot be stopped from another thread

Selecting the option "Decoder VA-API throught DRM"
$ vlc test.mkv
VLC media player 2.2.6 Umbrella (revision 2.2.6-0-g1aae78981c)
[000055adbab999d8] core libvlc: Esecuzione di vlc con l'interfaccia predefinita. Usa 'cvlc' per utilizzare vlc senza interfaccia.
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_40
libva info: va_openDriver() returns 0
[00007f88b4c0d8b8] avcodec decoder: Using Intel i965 driver for Intel(R) Haswell Mobile - 1.8.3 for hardware decoding.
Failed to open VDPAU backend libvdpau_i965.so: impossibile aprire il file oggetto condiviso: File o directory non esistente
QObject::~QObject: Timers cannot be stopped from another thread

Selecting the option "Decoder VA-API throught X11"
$ vlc test.mkv
VLC media player 2.2.6 Umbrella (revision 2.2.6-0-g1aae78981c)
[00005636cc1249d8] core libvlc: Esecuzione di vlc con l'interfaccia predefinita. Usa 'cvlc' per utilizzare vlc senza interfaccia.
libva info: VA-API version 0.40.0
libva info: va_getDriverName() returns 0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_0_40
libva info: va_openDriver() returns 0
[00007fbd00c0d8b8] avcodec decoder: Using Intel i965 driver for Intel(R) Haswell Mobile - 1.8.3 for hardware decoding.
Failed to open VDPAU backend libvdpau_i965.so: impossibile aprire il file oggetto condiviso: File o directory non esistente

For other bigger files there is a noticeable difference in the CPU usage between the Xorg and Wayland sess...

[Daniel van Vugt (vanvugt)]

Thanks. I learned something new.

Yes, as you discovered I would only expect VLC hardware decoding to work in a "Ubuntu on Xorg" session with the VLC option "VA-API video decoder via X11". Because that's the only combination of session and VLC option that's even theoretically compatible at all. Although performance in this mode is still relatively poor compared to MPV or even Totem. So I'd recommend those first.

Using "VA-API video decoder via DRM" I would not expect to work under Wayland. And it doesn't.

It sounds like someone needs to log an enhancement/bug with VLC upstream to add a new option "VA-API video decoder via Wayland". Because that's the missing option (where the possible options are https://github.com/01org/libva/blob/master/va/va_backend.h#L39 ).

[Daniel van Vugt (vanvugt)]

Andrea, it appears bug 1716800 already covers the problem to some extent. I suggest you subscribe there instead.

This bug is already conflated by different issues in different players.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Iain Lane (laney)]

Daniel - with 1.13 -gl has moved from -bad to -base. I think that b769f022619db7b46851986494621492342eaafb is the same as your fix, so I have dropped your patch. Please let me know if that's wrong somehow.

[Daniel van Vugt (vanvugt)]

Thanks Laney. It was in the back of my mind -- I knew upstream had done a fix identical to mine, but they just moved it. I had not stopped to go looking for it...

I can verify everything seems to be working with the latest bionic updates.

[Daniel van Vugt (vanvugt)]

Oops. I wasn't testing the new packages yet. I guess we're waiting on:

gst-plugins-base1.0 (1.13.1-1) experimental; urgency=medium

[Iain Lane (laney)]

On Fri, Mar 02, 2018 at 02:42:41AM -0000, Daniel van Vugt wrote:
> Oops. I wasn't testing the new packages yet. I guess we're waiting on:
>
> gst-plugins-base1.0 (1.13.1-1) experimental; urgency=medium

You could probably (carefully!) extract it from bionic-proposed if you
want to test it that way.

--
Iain Lane [ <email address hidden> ]
Debian Developer [ <email address hidden> ]
Ubuntu Developer [ <email address hidden> ]

[Daniel van Vugt (vanvugt)]

I might do that but:
 (a) We would have to exclude the old fix (upgrade plugins-bad) at the same time; and
 (b) It is almost EOW here. I can probably bet on everything just working by next week :)

[Doug McMahon (mc3man)]

This seems fixed as of today's updates but does beg the question of how would most users even know to install the gstreamer1.0-vaapi package?
It doesn't show up here in ubuntu software no matter what I search..

[Daniel van Vugt (vanvugt)]

Doug,

Ideally it will be automatically installed, or at least gets installed with ubuntu-restricted-addons. See: https://wiki.ubuntu.com/IntelQuickSyncVideo

[Doug McMahon (mc3man)]

On 03/05/2018 09:02 PM, Daniel van Vugt wrote:
> Doug,
>
> Ideally it will be automatically installed, or at least gets installed
> with ubuntu-restricted-addons. See:
> https://wiki.ubuntu.com/IntelQuickSyncVideo
>
Good to know about the add-ons package, been quite awhile here that I've
used it. Generally on new install I'm just trying to play media &
looking to see if the gst-packagekit > ubuntu-software reacts properly,
ect. (- which it's doing a decent job atm, always some room for
improvement..
I'd think many users also follow that path, i.e add or access media then
follow the pop up if support isn't already available.

[Daniel van Vugt (vanvugt)]

Libva 2.17.0 added support for DRI3: https://github.com/intel/libva/pull/614 so that's in lunar

[Daniel van Vugt (vanvugt)]

Chromium is a "Won't Fix". Under Xwayland it should work automatically with the introduction of libva 2.17.0. And under native Wayland there is no bug.

[Daniel van Vugt (vanvugt)]

Actually we can say Fix Committed to Chromium now because it's a snap and the fixed libva has been added to it:

https://code.launchpad.net/~hectorcao/chromium-browser/+git/chromium-browser/+merge/440311