dlopen(libGL.so) resolves to mesa rather than nvidia

[Peter Würtz]

I'm having trouble with a combination of NVIDIA + Python + Qt + Opengl.

I tried using a QGraphicsView on a QGLWidget. I'm getting a white window and errors like this these:
QGLShader: could not create shader
Vertex shader for simpleShaderProg (MainVertexShader &PositionOnlyVertexShader) failed to compile

This is an example application triggering the problem:
http://pastebin.com/R0aa8ejs

The 'same' program works flawlessly when using C++/Qt. I'm seeing the exact behavior when using PySide instead of PyQt4 by the way. I'm also seeing this error when trying the original demo application from python-qt4-doc. Also, calling
QtGui.QApplication.setGraphicsSystem("opengl")
produces the same errors.

I'm experiencing this problems on 11.10 and 12.04 with the ubuntu-provided nvidia drivers (where 12.04 includes the most recent driver for now). After installing the driver using the original NVidia installer, the applications work as expected.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Peter Würtz (pwuertz)]

Still seeing this in 12.10

[Peter Würtz (pwuertz)]

Problem still exists in 13.10 and also affects QML/QtQuick applications on Qt5.

[Peter Würtz (pwuertz)]

Sorry, I meant QML/QtQuick applications on PyQt5.

[Peter Würtz (pwuertz)]

Workaround:
The shader creation problem completely vanishes when importing PyOpenGL additionally to PyQt. It doesn't even matter if it's imported before or after PyQt, just before the first shader is created.

When I add:
from OpenGL import GL
the example from my first post works.

Someone from the PyQt mailing list suggested that this is caused by an error when python dynamically loads the GL library.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dmitry Shachnev (mitya57)]

As discussed in bug 1241988, this is rather an issue with drivers themselves.

[Dmitry Shachnev (mitya57)]

Might be related to bug 1248642.

[Peter Würtz (pwuertz)]

Hm, this bug here existed already in 2012, whereas bug 1248642 is a regression from a change in binutils in 2013. Also, this missing libpthread.so.0 link problem shouldn't apply since the main application, python, is definitely linked to it:

$ objdump -pR /usr/bin/python | grep libpthread
  NEEDED libpthread.so.0

[Peter Würtz (pwuertz)]

Here is another, more minimalistic script that demonstrates the shader compilation bug on NVIDIA and its workaround.

[Jörn Schönyan (joern-schoenyan)]

I can confirm that bug in PyQt5/QML. I hope that it can be fixed, as I'm working on a project for future Lubuntu versions and want to get rid of additional dependencies (python3-opengl). Interesting: It doesn't seem to exist on hybrid notebooks a.k.a. Optimus in combination with bumblebee.

[Scott Kitterman (kitterman)]

Since this works fine on non-nvidia systems, why does anyone think this is anything other than a driver issue?

[B. Clausius (barcc)]

Since this works fine with Qt, why does anyone think this is anything other than a PyQt issue?

[Scott Kitterman (kitterman)]

On Tuesday, August 12, 2014 05:13:54 you wrote:
> Since this works fine with Qt, why does anyone think this is anything
> other than a PyQt issue?

OK, provide a working Qt test case to go with the PyQt test case already
provided and I'll see what upstream has to say.

[Peter Würtz (pwuertz)]

Please also have a look at my example and workaround in #10. As described in #5, the problem seems to be an error when python dynamically loads the required opengl libraries. Unfortunately we still don't know what exactly goes wrong and why it only affects python qt bindings loading the ubuntu-packaged nvidia driver.

[Peter Würtz (pwuertz)]

Ok, I think I found the problem.

Apparently the PyQt application ends up loading the Mesa GL library (/usr/lib/x86_64-linux-gnu/mesa/libGL.so.1.2.0) instead of the NVidia driver (/usr/lib/nvidia-331/libGL.so.331.38) that is loaded when importing PyOpenGL first.

The difference seems to be the library name when calling dlopen(). PyOpenGL tries to import "libGL.so.1", whereas PyQt imports "libGL.so", without the ".1" suffix.

A simple C application calling dlopen() and checking /proc/<pid>/maps shows that "libGL.so" points to the Mesa driver, whereas "libGL.so.1" ends up loading the NVidia driver.

So this is a bug in Ubuntu's NVidia package I assume?

[Peter Würtz (pwuertz)]

The first entry of "libGL.so" in "ldconfig -p" points to "/usr/lib/x86_64-linux-gnu/libGL.so", which is a symlink to the mesa driver. After deleting that link pyqt works fine as expected. That symlink belongs to the "libgl1-mesa-dev" package and seems to overrule the libGL.so entries in ldconfig for the nvidia driver.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[B. Clausius (barcc)]

Thanks Peter Würtz! So it seems the workaround can be simplified to

import ctypes
ctypes.CDLL('libGL.so.1', ctypes.RTLD_GLOBAL)

or

import ctypes.util
ctypes.CDLL(ctypes.util.find_library('GL'), ctypes.RTLD_GLOBAL)

[Jörn Schönyan (joern-schoenyan)]

barcc, that second approach won't work.

import ctypes
from ctypes import util
ctypes.CDLL(util.find_library('GL'), ctypes.RTLD_GLOBAL)

That should be better, but I can't reproduce the bug myself, likely because it's an Optimus notebook.

[Peter Würtz (pwuertz)]

*sigh* late 2015, still not fixed in Ubuntu

[Thomas Karl Pietrowski (thopiekar)]

Still here on Wily

[JianFeng Liu (liujianfeng1994)]

why not just dlopen libGL.so.1
I've mailed to the packager of nvidia driver in debian and got the answer: "pyqt should dlopen libGL.so.1".

[Peter Würtz (pwuertz)]

I don't even understand the argument. Lets turn it around/right again: Why not just dlopen libGL.so? Any application should be able to dlopen libGL.so.

"libGL.so" is the library name everyone agrees to and everyone expects to find, which is exactly why this file exists in the first place, doesn't it? One should expect that this primary library name points to the most reasonable and (before all else) WORKING OpenGL library. If a distribution decides to do some name mangling for maintaining multiple versions or vendors it is of course free to do so. Please name it "libGL.so.foo" or "libGl.so.bar" if you want, buy why is it necessary for pyqt to know about your naming decision and most importantly: Why is it necessary to stick with a broken(!) default configuration?

It baffles me that it took 4 years to boil it down to one line of not giving a damn.

[kosti (ilayn)]

Just to provide another user-case:

https://github.com/spyder-ide/spyder/issues/3226

I don't even understand the problem unfortunately.

[ChenYunChih (cba840910)]

This works!!

https://github.com/Ultimaker/Cura/pull/131

[Peter Würtz (pwuertz)]

Still not fixed after almost 5 years. Problem and solution are well known.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Tempuser (tempuser999)]

Also there seems to be no interest in fixing this, for the record:

Problem and solution are still present/working in Ubuntu 17.10. and Nvidia driver 384.

[Julian Gilbey (jdg)]

I have found the source of the bug and provided a patch: see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=913287

[Julian Gilbey (jdg)]

And the Qt team are now working on implementing the patch, so this should be resolving in a forthcoming Qt release :-) See https://bugreports.qt.io/browse/QTBUG-71488