X crashes on GPU access by userspace apps, Hardware Acceleration

[Pritam Baral]

An upgrade to Xorg 1.11, and a restart later, when I tried to play a video in VLC, X crashed. I looked into /var/log/Xorg.0.log.old and found a segfault in /usr/bin/X. The fault seemed to be with fglrx, so I disabled Hardware Acceleration in VLC (Video Overlay). All was fine now without Hardware Acceleration. When I opened chrome://gpu in Google Chrome, it showed:

unable to boot the gpu process

while previously it used to have full Hardware Accelaration (except 2d canvas)

I enable Hardware Acceleration in VLC, and it crashes again. I open any game that uses Hardware Accelaration: it crashes again.

System:
Ubuntu 12.04 (all updates installed, freshly booted)
ATI Mobility Radeon HD 5470 (any other card of the fglrx family will work too)

Steps to reproduce:
1) install the xserver-xorg pakages (with X 1.11 being the latest, and the cause here.)
2) install fglrx package (either from the repo, or built by the binary from ati.com, same issue)
3) enable Overlay in VLC's video settings
4) Play any video (preferably .264 encoded, they have most wide Hardware support.)

I'm not sure whether to classsify it as a security vulnerabilty (a non-privileged process being able to crash the display server, and take all other open apps with it.)

[Pritam Baral (pritambaral)]

Just reinstalled Precise Alpha 1, installed fglrx and vlc, and ran multiple videos with Overlay enabled. No crash. So I guess this bug can be said to be present in X 1.11 and not in 1.10