libclang1 has wrong standard include paths

[Corentin SCHREIBER]

Bug affects all versions of libclang1 I could test (3.3, 3.4 and 3.5).
The precise version of libclang I have is 1:3.4-1ubuntu3.
Running Ubuntu 14.04 (Linux Mint 17.1).

When building a minimal libclang program (refgen.cpp, attached) and using the "-print-search-dirs" command line argument to output the standard header search directories, libclang outputs the following:
programs: =:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../x86_64-linux-gnu/bin
libraries: =../lib/clang/3.4:/usr/lib/gcc/x86_64-linux-gnu/4.8:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu:/lib/x86_64-linux-gnu:/lib/../lib64:/usr/lib/x86_64-linux-gnu:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../..:/lib:/usr/lib

Notice the leading "../lib/clang/3.4" in "libraries", which does not make any sense and is likely missing part of the path. Notice also the leading ":" in "programs", which probably doesn't hurt, but looks suspicious.

For reference, running clang (3.4) with the same command line argument outputs:
programs: =/usr/bin:/usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../../x86_64-linux-gnu/bin
libraries: =/usr/bin/../lib/clang/3.4:/usr/bin/../lib/gcc/x86_64-linux-gnu/4.8:/usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu:/lib/x86_64-linux-gnu:/lib/../lib64:/usr/lib/x86_64-linux-gnu:/usr/bin/../lib/gcc/x86_64-linux-gnu/4.8/../../..:/lib:/usr/lib

One can see here that the leading "/usr/bin/" has probably been stripped somehow from the "programs" and "libraries" search paths in libclang1.

The end result is that libclang1 does not find clang's standard headers, and falls back to using GCC's, which it does not support.
In my case, this means that if I try to parse even the simplest of files (test.cpp, attached) with this libclang1 and C++11 support enabled, the parsing fails because libclang1 is failing to understand GCC's SSE intrinsics in /usr/lib/gcc/x86_64-linux-gnu/4.8/include/ia32intrin.h, generating lots of errors:
/usr/lib/gcc/x86_64-linux-gnu/4.8/include/ia32intrin.h:41:10: error: use of undeclared identifier '__builtin_ia32_bsrsi'
/usr/lib/gcc/x86_64-linux-gnu/4.8/include/ia32intrin.h:89:10: error: use of undeclared identifier '__builtin_ia32_rdpmc'
/usr/lib/gcc/x86_64-linux-gnu/4.8/include/ia32intrin.h:97:10: error: use of undeclared identifier '__builtin_ia32_rdtsc'
/usr/lib/gcc/x86_64-linux-gnu/4.8/include/ia32intrin.h:105:10: error: use of undeclared identifier '__builtin_ia32_rdtscp'
/usr/lib/gcc/x86_64-linux-gnu/4.8/include/ia32intrin.h:113:10: error: use of undeclared identifier '__builtin_ia32_rolqi'

[Corentin SCHREIBER (cschreib)]

Compiling the minimal libclang program attached above using:
clang refgen.cpp -std=c++11 -I"/usr/lib/llvm-3.4/include" -lclang-3.4 -lstdc++ -o refgen
(replacing clang by gcc here does not change anything)

Invoking the libclang program on the test file attached above using:
./refgen test.cpp -std=c++11

[Corentin SCHREIBER (cschreib)]

Sorry, I meant to report this bug to libclang1-3.4, not llvm-toolchain-3.4.

[Corentin SCHREIBER (cschreib)]

I have also tried with the libclang python bindings, in case I was doing something wrong with the C interface. I am still getting these errors (see attached python script, to be run on the example file linked above).

[Corentin SCHREIBER (cschreib)]

Also, I confirm this is an issue of the standard include paths: if I create manually a symlink to "/usr/lib" in "../lib" (whichever directory I am currently in), then the programs attached above do work correctly...

[Corentin SCHREIBER (cschreib)]

This issue could be linked to this entry of the llvm libtooling FAQ:
http://clang.llvm.org/docs/LibTooling.html#libtooling-builtin-includes

Quoting:
"Clang tools need their builtin headers and search for them the same way Clang does. Thus, the default location to look for builtin headers is in a path $(dirname /path/to/tool)/../lib/clang/3.3/include relative to the tool binary. This works out-of-the-box for tools running from llvm’s toplevel binary directory after building clang-headers, or if the tool is running from the binary directory of a clang install next to the clang binary."

I guess this could explain the "../lib/clang/3.4" in libclang.

I have another machine running quantal, and on which I could install libclang1-3.3 and libclang1-3.4 from the llvm PPA (ppa:h-rayflood/llvm). Here too, the library search dirs start with the same "../lib/clang/", but these versions work perfectly. There must be something different in the way they are built, however from what I could gather this PPA is just taking the packages from Debian without modification:
http://http.debian.net/debian/pool/main/l/llvm-toolchain-3.4

I will try this approach in lucid, replacing the packages provided by Ubuntu and report back.

[Corentin SCHREIBER (cschreib)]

(sorry, I meant of course "trusty", not "lucid", in the above post)

I confirm that the libclang from the above PPA (1:3.4.2-debian13trusty1) does work correctly in trusty. In other words, when I compile the example programs attached to this bug report with the libclang from this PPA, the errors about ia32intrin.h are gone and parsing is fully functional. I don't know if this is due to 1) a more recent upstream libclang version (3.4.2 vs 3.4), 2) a more recent debian base, or 3) tweaks specific to this PPA.

If someone familiar with the debian/ubuntu build system could take a look at the source of this PPA:
https://github.com/hATrayflood/llvm-ppa

... and compare to what is done for the official ubuntu packages, that would be great. I have tried myself, but the thing is so complex and globally unknown to me that I am completely lost. From what I could gather, it seems that the maintainer of the PPA actually has made a few modifications to the original debian packages, as can be seen from this diff file:
https://github.com/hATrayflood/llvm-ppa/blob/master/trusty/llvm-toolchain-3.4/debian.diff

Also, can someone else actually reproduce this issue on their side?

NB: Using the PPA libclang, the builtin search directories are similar to the original ubuntu version. So that lead was a dead end. Here is the output of -print-search-dirs:
programs: =:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../x86_64-linux-gnu/bin
libraries: =../lib/clang/3.4.2:/usr/lib/gcc/x86_64-linux-gnu/4.8:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu:/lib/x86_64-linux-gnu:/lib/../lib64:/usr/lib/x86_64-linux-gnu:/usr/lib/gcc/x86_64-linux-gnu/4.8/../../..:/lib:/usr/lib

[Corentin SCHREIBER (cschreib)]

I could spot the difference between the two versions. It can be seen by running:
./refgen test.cpp -std=c++11 -v

With the ubuntu libclang, the output I get is (stripped to the essential bits):
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/x86_64-linux-gnu"
ignoring nonexistent directory "../lib/clang/3.4/include"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8
 /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/backward
 /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/x86_64-linux-gnu/c++/4.8
 /usr/local/include
 /usr/lib/gcc/x86_64-linux-gnu/4.8/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.

However, with the PPA libclang, the output is:
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.8
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/x86_64-linux-gnu"
ignoring nonexistent directory "../lib/clang/3.4.2/include"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8
 /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/backward
 /usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/x86_64-linux-gnu/c++/4.8
 /usr/include/clang/3.4.2/include
 /usr/local/include
 /usr/lib/gcc/x86_64-linux-gnu/4.8/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.

It is essentially identical, except that the PPA version has one additional line that makes it all work:
 /usr/include/clang/3.4.2/include

Now where this line came from, or why was it removed in the official ubuntu version, I have no clue... I will contact the author of the PPA to see if he could help.

[Corentin SCHREIBER (cschreib)]

Sorry for the flood of message. I just want to report that the issue can be fixed temporarilly by adding this missing include path manually using the command line option: -isystem /usr/include/clang/3.4/include

Note that the usual "-I /usr/include/clang/3.4/include" will not work, since the GCC headers will be found first.