@floe -- thanks for the extra info! Unfortunately I'm not sure this is a gstreamer issue (or more precisely, there may well be a gstreamer issue on amd64, but it doesn't appear to be the case on arm64 with the Pi Desktop image). gstreamer1.0-vaapi isn't seeded on the Pi Desktop images, and indeed running your gst-launch reproducer plays the video happily with no mention of gstvaapi in the console output. Just in case it's useful, here's the output I saw:

GST_DEBUG=3 gst-launch-1.0 playbin uri=file:///$HOME/big_buck_bunny_720p_surround.mp4
Setting pipeline to PAUSED ...
0:00:00.045303856 31845 0xaaaaf25f9240 WARN                 basesrc gstbasesrc.c:3688:gst_base_src_start_complete:<source> pad not activated yet
0:00:00.046280739 31845 0xaaaaf25f9240 WARN                 basesrc gstbasesrc.c:3688:gst_base_src_start_complete:<source> pad not activated yet
Pipeline is PREROLLING ...
0:00:00.096946452 31845 0xaaaaf260eb60 WARN                 qtdemux qtdemux.c:3121:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 1
0:00:00.097408468 31845 0xaaaaf260eb60 WARN                 qtdemux qtdemux.c:3121:qtdemux_parse_trex:<qtdemux0> failed to find fragment defaults for stream 2
Redistribute latency...
Redistribute latency...
Redistribute latency...
Redistribute latency...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstPulseSinkClock
Redistribute latency...

About 9 seconds in I closed the window (which was correctly playing the video, and audio), which produced the output:

0:00:09.474384373 31845 0xffff902bc120 WARN             xvimagesink xvimagesink.c:568:gst_xv_image_sink_handle_xevents:<xvimagesink0> error: Output window was closed
ERROR: from element /GstPlayBin:playbin0/GstPlaySink:playsink/GstBin:vbin/GstXvImageSink:xvimagesink0: Output window was closed
Additional debug info:
../sys/xvimage/xvimagesink.c(568): gst_xv_image_sink_handle_xevents (): /GstPlayBin:playbin0/GstPlaySink:playsink/GstBin:vbin/GstXvImageSink:xvimagesink0
Execution ended after 0:00:09.137470426
Setting pipeline to NULL ...
0:00:09.477803649 31845 0xffff800609e0 WARN             xvimagesink xvimagesink.c:1045:gst_xv_image_sink_show_frame:<xvimagesink0> could not output image - no window
0:00:09.478389497 31845 0xaaaaf260eb60 WARN                 qtdemux qtdemux.c:6747:gst_qtdemux_loop:<qtdemux0> error: Internal data stream error.
0:00:09.478468497 31845 0xaaaaf260eb60 WARN                 qtdemux qtdemux.c:6747:gst_qtdemux_loop:<qtdemux0> error: streaming stopped, reason error (-5)
ERROR: from element /GstPlayBin:playbin0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstQTDemux:qtdemux0: Internal data stream error.
Additional debug info:
../gst/isomp4/qtdemux.c(6747): gst_qtdemux_loop (): /GstPlayBin:playbin0/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/GstQTDemux:qtdemux0:
streaming stopped, reason error (-5)
Freeing pipeline ...

Admittedly, this was on my usual Pi desktop; I'll try it on an entirely fresh card in a bit (as I need to do that to respond to seb128's request for a proper stacktrace), but I suspect the result will be similar. Still, it may be worth opening a separate bug for the gstvaapi case?