nux::IOpenGLShaderProgram::Begin crashes on nVidia 10de:00fd

[Sergio Gelato]

After upgrading to precise and activating nvidia-173 (173.14.35) on a machine with the following graphics card:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation NV37GL [Quadro PCI-E Series] [10de:00fd] (rev a2)

/usr/bin/compiz started to crash with the exact same symptoms as in bug #768178. (I'm filing a new bug since that one is marked as a duplicate of a quite different bug that is itself marked as invalid.) This isn't too surprising since the graphics card is a close relative of the GeForce FX series, various flavours of which are already blacklisted in nux.

Forcing Unity 2D works around the problem, so I've applied the attached patch on top of nux 2.14.1-0ubuntu1. Maybe one could set _use_glsl_shaders = false in NuxGraphics/GraphicsEngine.cpp instead but I don't have the time to test this conjecture. glGetString(GL_RENDERER) returns "Quadro PCI-E Series/PCI/SSE2" for this card and driver.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Add 10de:00fd to the blacklist for Unity" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

[Sergio Gelato (sergio-gelato)]

I've upgraded the OS on the affected computer to trusty and found that the problem is still present in trusty's version of nux. Same symptoms, and my patch still applies.

[Sergio Gelato (sergio-gelato)]

 tag -patch
 done

* Sergio Gelato [2014-10-07 06:27:49 +0000]:
> I've upgraded the OS on the affected computer to trusty and found that
> the problem is still present in trusty's version of nux. Same symptoms,
> and my patch still applies.

I'll take that back: the patch still applies but is no longer useful since
it no longer results in compiz-2d being invoked. compiz starts anyway, the
LIBGL_ALWAYS_SOFTWARE setting is not applicable because I'm using nVidia's
proprietary OpenGL libraries rather than Mesa, and so I get a crash. That
means one needs to fix the actual cause of the crash.

[Sergio Gelato (sergio-gelato)]

I can't seem to persuade apport to upload my crash file for this bug, so here
is what I see as the gist of it.

The crash is due to the method nux::IOpenGLShaderProgram::Begin() being
invoked on a null object. It occurs while trying to read field _OpenGLID,
and the failing instruction (i386) is
 mov 0x44(%eax),%eax
with %eax being 0 at this point.

This method is invoked from nux::GraphicsEngine::QRP_GLSL_1Tex(), and simple
inspection of the source code (NuxGraphics/RenderingPipeGLSL.cpp) reveals
that if the DeviceTexture argument isn't of a type derived from
nux::IOpenGLTexture2D::StaticObjectType the ShaderProg local variable will
indeed hold a null value. So the solution is either to extend this method
so that it can handle other device texture types, or to make sure it never
gets invoked with a texture type it does not support.

In my case the additional texture type that needs to be supported turns out
to be IOpenGLRectangleTexture. The code to do this is present but commented
out in the source. It seems to always have been commented out since the
code was first added to the bzr repository (revision 9). I assume that
uncommenting it will take me into uncharted territory?

Here is the stack trace:

#0 0xb2db9873 in nux::IOpenGLShaderProgram::Begin() () from /usr/lib/i386-linux-gnu/libnux-graphics-4.0.so.0
No symbol table info available.
#1 0xb2dc97d4 in nux::GraphicsEngine::QRP_GLSL_1Tex(int, int, int, int, nux::ObjectPtr<nux::IOpenGLBaseTexture>, nux::TexCoordXForm&, nux::color::Color const&) () from /usr/lib/i386-linux-gnu/libnux-graphics-4.0.so.0
No symbol table info available.
#2 0xb2dc2e2a in nux::GraphicsEngine::QRP_1Tex(int, int, int, int, nux::ObjectPtr<nux::IOpenGLBaseTexture>, nux::TexCoordXForm&, nux::color::Color const&) () from /usr/lib/i386-linux-gnu/libnux-graphics-4.0.so.0
No symbol table info available.
#3 0xb06088f3 in nux::TextureLayer::Renderlayer(nux::GraphicsEngine&) () from /usr/lib/i386-linux-gnu/libnux-4.0.so.0
No symbol table info available.
#4 0xb0602077 in nux::BasePainter::RenderSinglePaintLayer(nux::GraphicsEngine&, nux::Rect, nux::AbstractPaintLayer*) () from /usr/lib/i386-linux-gnu/libnux-4.0.so.0
No symbol table info available.
#5 0xb111ff3d in unity::panel::PanelView::Draw(nux::GraphicsEngine&, bool) () from /usr/lib/compiz/libunityshell.so
No symbol table info available.
#6 0xb061ff67 in nux::View::ProcessDraw(nux::GraphicsEngine&, bool) () from /usr/lib/i386-linux-gnu/libnux-4.0.so.0
No symbol table info available.
#7 0xb05fe513 in nux::Layout::ProcessDraw(nux::GraphicsEngine&, bool) () from /usr/lib/i386-linux-gnu/libnux-4.0.so.0
No symbol table info available.
#8 0xb05d89a5 in nux::BaseWindow::DrawContent(nux::GraphicsEngine&, bool) () from /usr/lib/i386-linux-gnu/libnux-4.0.so.0
No symbol table info available.
#9 0xb061fa32 in nux::View::ProcessDraw(nux::GraphicsEngine&, bool) () from /usr/lib/i386-linux-gnu/libnux-4.0.so.0
No symbol table info available.
#10 0xb06236b6 in nux::WindowCompositor::RenderTopViewContent(nux::BaseWindow*, bool) () from /usr/lib/i386-linux-gnu/libnux-4.0.so.0
No symbol table info available.
#11 0xb0629020 in nux::WindowCompositor::RenderTopViews(bool, std::list<n...