install deps header files

[Matteo Bertozzi]

Our application depends on cython lxml (etreepublic.pxd)
We need to specify the header path (gcc -I) of libxml2,
but we've to ship libxml2 headers (and other deps with the lxml installation).

I've attached a simple patch, that copies headers to lxml/include/
folder, and I've added a get_include() method that returns include paths.

[scoder (scoder)]

Did you test this patch with Python 2.3?

Stefan

[Matteo Bertozzi (theo-bertozzi)]

Added patch to work with python 2.3

[scoder (scoder)]

The "package_data" option is not available in distutils on Python 2.3. You
need to use "data_files" there.

Stefan

[scoder (scoder)]

Tried it, doesn't work.

The problem is that libxml2 and especially libxslt may be installed in standard locations, which "xslt-config --cflags" does not include in its output. On my system, for example, the header files of libxslt are not included in the distro when using your patch as they are installed under /usr/include as /usr/include/libxslt/*.h

I don't know of any "standard" way to figure out the standard include directories, especially not in a portable way. The search order must match what the C compiler uses when being run, otherwise, the setup.py script could end up finding a separate installation somewhere else and copying that instead of the actual installation that is being used for the compilation.

Also, your patch *always* includes the header files in the distribution, not only for static builds. I don't think that's necessary (or even a good idea). If the libraries are upgraded on a system, it may well be better to use the corresponding newer header files instead of the ones that lxml was originally built with.

[Giovanni Bajo (giovannibajo)]

"cpp -v" dumps the include search paths, that can be parsed somehow. The full path to cpp can be found with "$CC --print-prog-name=cpp" (try it with something different from /usr/bin/gcc to see that it works). I think that, for Linux, it is sufficient to fall back to parsing cpp -v whenever xslt-config doesn't return an include path. For Windows, it shouldn't be a problem since libxml/xslt are not standard libraries and are manually installed anyway.

I would also add a self-check so that setup.py bails if it cannot find libxslt.h/libxml2.h, instead of silently avoiding bundling them.

[Matteo Bertozzi (theo-bertozzi)]

If the header files are just copied only for static build,
I don't see the reason to lookup the headers path (cpp -v).

static build fetch libxml2 from a repo to lxml-src/lib/ and then compile
the source into build/tmp/libxml/, so the header files are placed in build/tmp/libxml2/include/

     python setup.py build --static-deps

or you can specify STATIC_LIBRARY_DIRS and STATIC_INCLUDE_DIRS
and you know exactly where headers files are placed.

am I wrong or have I missed something?

[Matteo Bertozzi (theo-bertozzi)]

I've attached a v3 patch that copies headers only if --static is specified, also works with python 2.3.

Tested with:
    LINUX: python setup.py build --static --static-deps install
    WINDOWS: python setup.py bdist_wininst --static

[scoder (scoder)]

I've pushed a modified version to github. Could you try if it works for you?

https://github.com/lxml/lxml/commit/a75e9c9c15a3c80b5d42113e6df7fc6f471108e4

Note that I decided to drop Py2.3 support. Sorry for not doing it before you started your work. on this.

[scoder (scoder)]

Oh, and it would be nice if "lxml.get_includes()" also worked for dynamically linked lxml versions, namely by checking if the header files are there and otherwise parsing the output of xslt-config, just like setupinfo.py currently does at build time.

Could you provide a patch for that as well?

[scoder (scoder)]

Implemented in lxml 2.3.1.