lxml compile osx10.9 mavericks cannot find libxml2 headers

[Gábor Farkas]

when compiling lxml on osx10.9 it cannot find the libxml headers, because they are in a slightly different folder.

in short, doing this before building it, fixes the issue:

xport CFLAGS=-I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/libxml2

the libxml2 folder contains the "libxml" subfolder, which contains the header-files.

i'm not sure how would be the best way to fix this.

full output from compiling:

$ /usr/bin/python setup.py build
Building lxml version 3.2.3.
Building without Cython.
Using build configuration of libxslt 1.1.28
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url'
  warnings.warn(msg)
running build
running build_py
copying src/lxml/includes/lxml-version.h -> build/lib.macosx-10.9-intel-2.7/lxml/includes
running build_ext
building 'lxml.etree' extension
cc -fno-strict-aliasing -fno-common -dynamic -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -mno-fused-madd -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -pipe -arch x86_64 -I/usr/include/libxml2 -I/private/tmp/hehe/lxml-3.2.3/src/lxml/includes -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.macosx-10.9-intel-2.7/src/lxml/lxml.etree.o -flat_namespace
clang: warning: argument unused during compilation: '-mno-fused-madd'
clang: warning: argument unused during compilation: '-flat_namespace'
In file included from src/lxml/lxml.etree.c:314:
/private/tmp/hehe/lxml-3.2.3/src/lxml/includes/etree_defs.h:9:10: fatal error: 'libxml/xmlversion.h' file not found
#include "libxml/xmlversion.h"
         ^
1 error generated.
error: command 'cc' failed with exit status 1

[Petr Olah (donthave)]

quick (and dirty) solution for me was adding the path to the C_INCLUDE_PATH - at least until its fixed

if you have libxml installed through brew it should look something like this:

export C_INCLUDE_PATH=$C_INCLUDE_PATH:/usr/local/Cellar/libxml2/2.9.1/include/libxml2

[Petr Olah (donthave)]

well the simple and right solution was just installing Command Line Tools, you have to do it manually

https://developer.apple.com/downloads/index.action?=command%20line%20tools#

hope it helps you as well

[Gábor Farkas (gabor-nekomancer)]

yes, i can confirm that this fixes the problem. so there is no need to change lxml it seems.

[scoder (scoder)]

If there is anything to add to the documentation to help users get this right, I'd take a pull request. Otherwise, ISTM that this issue can be closed.

[Felix Yen (fyen-a)]

never had a problem with 3.2.3, but 3.2.4 does not build for me on this platform (Mac 10.9) when i try to install lxml via buildout.

In file included from src/lxml/lxml.etree.c:314:
/var/folders/r0/5355nqqs29q6k02tk9nlbfz40000gp/T/easy_install-cR7vak/lxml-3.2.4/src/lxml/includes/etree_defs.h:9:10: fatal error:
      'libxml/xmlversion.h' file not found
#include "libxml/xmlversion.h"

[scoder (scoder)]

>> Felix Yen:
>>> the way my project is structured, a single
>>> buildout failure freezes all our third-party dependencies. so not being
>>> able to build lxml 3.2.4 via buildout hides any upgrades published for
>>> other packages. i am thinking about finding a way to refine our build
>>> system to freeze lxml to 3.2.3 without affecting other packages, but
>>> would prefer to use 3.2.4.
>>
>> Someone proposed to install the "command line tools", whatever that is. Did
>> you try that?
>
> not exactly … i installed the Xcode command-line tools earlier for other
> reasons. i couldn’t build 3.2.3 without them. in short, the tools are
> installed and everything was fine until 3.2.4 was released.

Are you sure that the lxml version is the only difference? No (partial) system upgrades in between? Still using the same XCode release?

lxml 3.2.3 still builds for you when you do exactly the same steps, right?

I wouldn't know any changes between 3.2.3 and 3.2.4 that could impact the build under MacOS-X.

[Felix Yen (fyen-a)]

obviously not sure, after failing to produce a small example of my build problem.

reinstalling the command-line tools ("xcode-select --install") fixed my environment. i had neglected to remove the old lxml 3.2.3 egg, so buildout cheerfully reused it.

very sorry, and thankful for your prompt responses!

[Tres Seaver (tseaver)]

I've just seen three failures[1][2][3] on the Zope Winbot following the 3.3.0 release, all showing the same error on Windows:

    c:\temp\easy_install-k1i48j\lxml-3.3.0\src\lxml\includes\etree_defs.h(9) : fatal error C1083: Cannot open include file: 'libxml/xmlversion.h': No such file or directory

[1] http://winbot.zope.org/builders/z3c.layer.ready2go_py_265_32/builds/523/steps/buildout/logs/stdio

[2] http://winbot.zope.org/builders/z3c.pagelet_py_265_32/builds/525/steps/buildout/logs/stdio

[3] http://winbot.zope.org/builders/z3c.contents_py_265_32/builds/524/steps/buildout/logs/stdio

[scoder (scoder)]

MS Windows has a completely different build setup, though. These two problems are unlikely to be related.

[Tres Seaver (tseaver)]

The point is that the '-I <path/to/libxml2.h>' stuff should be computed via distutils, and has somehow recently begun to fail on two different (non-Linux) platforms.

[scoder (scoder)]

It can't be "computed" by distutils on Windows because there is no installed libxml2. It's unpacked into a local build directory.

Please believe me, the Windows build is *not* related to the MacOS build here.

[Marius Gedminas (mgedmin)]

Tres: I think the winbot.zope.org problem is a different one: https://bugs.launchpad.net/lxml/+bug/1274413

[Marius Gedminas (mgedmin)]

Clarification for the different error message (missing libxml2.h rather than missing stdint.h):
- on winbot the 'wineggbuilder' job does the fancy static linking stuff that involves editing lxml's setup.py as described in http://lxml.de/build.html#static-linking-on-windows, then uploads the binary eggs (and wininsts) to PyPI
- the other winbot jobs expect to find binary lxml eggs on PyPI, fail, then try the standard distutils build which is not expected to succeed.

[scoder (scoder)]

Closing as "invalid" as this is not a problem that can be fixed by lxml.