lxml fails to install on pypy3

[Eeo Jun]

Hi, we have tried to support PyPy 3 for our python CI builds today, but it fails to install only on PyPy 3. I have tried restarting the build but to no avail. Here's the full log:

Using worker: worker-linux-docker-527b567d.prod.travis-ci.org:travis-linux-6

Build system information
Build language: python
Build image provisioning date and time
Thu Feb 5 15:09:33 UTC 2015
Operating System Details
Distributor ID: Ubuntu
Description: Ubuntu 12.04.5 LTS
Release: 12.04
Codename: precise
Linux Version
3.13.0-29-generic
Cookbooks Version
a68419e https://github.com/travis-ci/travis-cookbooks/tree/a68419e
GCC version
gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

LLVM version
clang version 3.4 (tags/RELEASE_34/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
Pre-installed Ruby versions
ruby-1.9.3-p551
Pre-installed Node.js versions
v0.10.36
Pre-installed Go versions
1.4.1
Redis version
redis-server 2.8.19
riak version
2.0.2
MongoDB version
MongoDB 2.4.12
CouchDB version
couchdb 1.6.1
Neo4j version
1.9.4
RabbitMQ Version
3.4.3
ElasticSearch version
1.4.0
Installed Sphinx versions
2.0.10
2.1.9
2.2.6
Default Sphinx version
2.2.6
Installed Firefox version
firefox 31.0esr
PhantomJS version
1.9.8
ant -version
Apache Ant(TM) version 1.8.2 compiled on December 3 2011
mvn -version
Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T17:29:23+00:00)
Maven home: /usr/local/maven
Java version: 1.7.0_76, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-7-oracle/jre
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "3.13.0-29-generic", arch: "amd64", family: "unix"

$ git clone --depth=50 --branch=master git://github.com/libextract/libextract.git libextract/libextract
Cloning into 'libextract/libextract'...
remote: Counting objects: 345, done.
remote: Compressing objects: 100% (236/236), done.
remote: Total 345 (delta 199), reused 250 (delta 106), pack-reused 0
Receiving objects: 100% (345/345), 148.54 KiB | 0 bytes/s, done.
Resolving deltas: 100% (199/199), done.
Checking connectivity... done.
$ cd libextract/libextract
$ git checkout -qf 5d4c332676ec064165b3148e1c115d675019a9a0

This job is running on container-based infrastructure, which does not allow use of 'sudo', setuid and setguid executables.
If you require sudo, add 'sudo: required' to your .travis.yml
See http://docs.travis-ci.com/user/workers/container-based-infrastructure/ for details.
$ source ~/virtualenv/pypy3/bin/activate
$ python --version
Python 3.2.5 (b2091e973da6, Oct 19 2014, 18:29:55)
[PyPy 2.4.0 with GCC 4.6.3]
$ pip --version
pip 6.0.7 from /home/travis/virtualenv/pypy3-2.4.0/site-packages (python 3.2)
$ pip install .
You are using pip version 6.0.7, however version 6.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Processing /home/travis/build/libextract/libextract
Collecting chardet>=2.3.0 (from libextract==0.0.0)
  Downloading chardet-2.3.0.tar.gz (164kB)
    100% |################################| 167kB 1.4MB/s
Collecting lxml>=3.4.2 (from libextract==0.0.0)
  Downloading lxml-3.4.2.tar.gz (3.5MB)
    100% |################################| 3.5MB 113kB/s
    Building lxml version 3.4.2.
    Building without Cython.
    Using build configuration of libxslt 1.1.26
    Building against libxml2/libxslt in the following directory: /usr/lib/x86_64-linux-gnu
    /opt/python/pypy3-2.4.0/lib-python/3/distutils/dist.py:257: UserWarning: Unknown distribution option: 'bugtrack_url'
      warnings.warn(msg)
Installing collected packages: libextract, lxml, chardet
  Running setup.py install for libextract
  Running setup.py install for lxml
    Building lxml version 3.4.2.
    Building without Cython.
    Using build configuration of libxslt 1.1.26
    Building against libxml2/libxslt in the following directory: /usr/lib/x86_64-linux-gnu
    building 'lxml.etree' extension
    cc -O2 -fPIC -Wimplicit -I/usr/include/libxml2 -I/tmp/pip-build-zhoeh6/lxml/src/lxml/includes -I/home/travis/virtualenv/pypy3-2.4.0/include -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-3.2/src/lxml/lxml.etree.o -w
    cc -shared build/temp.linux-x86_64-3.2/src/lxml/lxml.etree.o -L/usr/lib/x86_64-linux-gnu -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-x86_64-3.2/lxml/etree.pypy3-24.so
    building 'lxml.objectify' extension
    cc -O2 -fPIC -Wimplicit -I/usr/include/libxml2 -I/tmp/pip-build-zhoeh6/lxml/src/lxml/includes -I/home/travis/virtualenv/pypy3-2.4.0/include -c src/lxml/lxml.objectify.c -o build/temp.linux-x86_64-3.2/src/lxml/lxml.objectify.o -w
    src/lxml/lxml.objectify.c:30412:3: error: ‘__pyx_pw_4lxml_9objectify_13NumberElement_29__div__’ undeclared here (not in a function)
    src/lxml/lxml.objectify.c:30484:3: error: ‘__pyx_pw_4lxml_9objectify_13NumberElement_5__long__’ undeclared here (not in a function)
    src/lxml/lxml.objectify.c:30498:3: error: ‘__pyx_pw_4lxml_9objectify_13NumberElement_15__oct__’ undeclared here (not in a function)
    src/lxml/lxml.objectify.c:30505:3: error: ‘__pyx_pw_4lxml_9objectify_13NumberElement_17__hex__’ undeclared here (not in a function)
    /opt/python/pypy3-2.4.0/lib-python/3/distutils/dist.py:257: UserWarning: Unknown distribution option: 'bugtrack_url'
      warnings.warn(msg)
    error: command 'cc' failed with exit status 1
    Complete output from command /home/travis/virtualenv/pypy3-2.4.0/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-zhoeh6/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-bdds0h-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/travis/virtualenv/pypy3-2.4.0/include/site/python3.2:
    Building lxml version 3.4.2.

    Building without Cython.

    Using build configuration of libxslt 1.1.26

    Building against libxml2/libxslt in the following directory: /usr/lib/x86_64-linux-gnu

    running install

    running build

    running build_py

    creating build

    creating build/lib.linux-x86_64-3.2

    creating build/lib.linux-x86_64-3.2/lxml

    copying src/lxml/builder.py -> build/lib.linux-x86_64-3.2/lxml

    copying src/lxml/cssselect.py -> build/lib.linux-x86_64-3.2/lxml

    copying src/lxml/pyclasslookup.py -> build/lib.linux-x86_64-3.2/lxml

    copying src/lxml/usedoctest.py -> build/lib.linux-x86_64-3.2/lxml

    copying src/lxml/ElementInclude.py -> build/lib.linux-x86_64-3.2/lxml

    copying src/lxml/sax.py -> build/lib.linux-x86_64-3.2/lxml

    copying src/lxml/_elementpath.py -> build/lib.linux-x86_64-3.2/lxml

    copying src/lxml/__init__.py -> build/lib.linux-x86_64-3.2/lxml

    copying src/lxml/doctestcompare.py -> build/lib.linux-x86_64-3.2/lxml

    creating build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/__init__.py -> build/lib.linux-x86_64-3.2/lxml/includes

    creating build/lib.linux-x86_64-3.2/lxml/html

    copying src/lxml/html/_html5builder.py -> build/lib.linux-x86_64-3.2/lxml/html

    copying src/lxml/html/builder.py -> build/lib.linux-x86_64-3.2/lxml/html

    copying src/lxml/html/html5parser.py -> build/lib.linux-x86_64-3.2/lxml/html

    copying src/lxml/html/_diffcommand.py -> build/lib.linux-x86_64-3.2/lxml/html

    copying src/lxml/html/clean.py -> build/lib.linux-x86_64-3.2/lxml/html

    copying src/lxml/html/formfill.py -> build/lib.linux-x86_64-3.2/lxml/html

    copying src/lxml/html/usedoctest.py -> build/lib.linux-x86_64-3.2/lxml/html

    copying src/lxml/html/_setmixin.py -> build/lib.linux-x86_64-3.2/lxml/html

    copying src/lxml/html/soupparser.py -> build/lib.linux-x86_64-3.2/lxml/html

    copying src/lxml/html/ElementSoup.py -> build/lib.linux-x86_64-3.2/lxml/html

    copying src/lxml/html/__init__.py -> build/lib.linux-x86_64-3.2/lxml/html

    copying src/lxml/html/defs.py -> build/lib.linux-x86_64-3.2/lxml/html

    copying src/lxml/html/diff.py -> build/lib.linux-x86_64-3.2/lxml/html

    creating build/lib.linux-x86_64-3.2/lxml/isoschematron

    copying src/lxml/isoschematron/__init__.py -> build/lib.linux-x86_64-3.2/lxml/isoschematron

    copying src/lxml/lxml.etree.h -> build/lib.linux-x86_64-3.2/lxml

    copying src/lxml/lxml.etree_api.h -> build/lib.linux-x86_64-3.2/lxml

    copying src/lxml/includes/relaxng.pxd -> build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/xmlerror.pxd -> build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/etreepublic.pxd -> build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/c14n.pxd -> build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/dtdvalid.pxd -> build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/config.pxd -> build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/schematron.pxd -> build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/xpath.pxd -> build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/xmlparser.pxd -> build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/tree.pxd -> build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/xslt.pxd -> build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/htmlparser.pxd -> build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/xmlschema.pxd -> build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/uri.pxd -> build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/xinclude.pxd -> build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-3.2/lxml/includes

    copying src/lxml/includes/etree_defs.h -> build/lib.linux-x86_64-3.2/lxml/includes

    creating build/lib.linux-x86_64-3.2/lxml/isoschematron/resources

    creating build/lib.linux-x86_64-3.2/lxml/isoschematron/resources/rng

    copying src/lxml/isoschematron/resources/rng/iso-schematron.rng -> build/lib.linux-x86_64-3.2/lxml/isoschematron/resources/rng

    creating build/lib.linux-x86_64-3.2/lxml/isoschematron/resources/xsl

    copying src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl -> build/lib.linux-x86_64-3.2/lxml/isoschematron/resources/xsl

    copying src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl -> build/lib.linux-x86_64-3.2/lxml/isoschematron/resources/xsl

    creating build/lib.linux-x86_64-3.2/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_dsdl_include.xsl -> build/lib.linux-x86_64-3.2/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_message.xsl -> build/lib.linux-x86_64-3.2/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_svrl_for_xslt1.xsl -> build/lib.linux-x86_64-3.2/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_abstract_expand.xsl -> build/lib.linux-x86_64-3.2/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/iso_schematron_skeleton_for_xslt1.xsl -> build/lib.linux-x86_64-3.2/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

    copying src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/readme.txt -> build/lib.linux-x86_64-3.2/lxml/isoschematron/resources/xsl/iso-schematron-xslt1

    running build_ext

    building 'lxml.etree' extension

    creating build/temp.linux-x86_64-3.2

    creating build/temp.linux-x86_64-3.2/src

    creating build/temp.linux-x86_64-3.2/src/lxml

    cc -O2 -fPIC -Wimplicit -I/usr/include/libxml2 -I/tmp/pip-build-zhoeh6/lxml/src/lxml/includes -I/home/travis/virtualenv/pypy3-2.4.0/include -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-3.2/src/lxml/lxml.etree.o -w

    cc -shared build/temp.linux-x86_64-3.2/src/lxml/lxml.etree.o -L/usr/lib/x86_64-linux-gnu -lxslt -lexslt -lxml2 -lz -lm -o build/lib.linux-x86_64-3.2/lxml/etree.pypy3-24.so

    building 'lxml.objectify' extension

    cc -O2 -fPIC -Wimplicit -I/usr/include/libxml2 -I/tmp/pip-build-zhoeh6/lxml/src/lxml/includes -I/home/travis/virtualenv/pypy3-2.4.0/include -c src/lxml/lxml.objectify.c -o build/temp.linux-x86_64-3.2/src/lxml/lxml.objectify.o -w

    src/lxml/lxml.objectify.c:30412:3: error: ‘__pyx_pw_4lxml_9objectify_13NumberElement_29__div__’ undeclared here (not in a function)

    src/lxml/lxml.objectify.c:30484:3: error: ‘__pyx_pw_4lxml_9objectify_13NumberElement_5__long__’ undeclared here (not in a function)

    src/lxml/lxml.objectify.c:30498:3: error: ‘__pyx_pw_4lxml_9objectify_13NumberElement_15__oct__’ undeclared here (not in a function)

    src/lxml/lxml.objectify.c:30505:3: error: ‘__pyx_pw_4lxml_9objectify_13NumberElement_17__hex__’ undeclared here (not in a function)

    /opt/python/pypy3-2.4.0/lib-python/3/distutils/dist.py:257: UserWarning: Unknown distribution option: 'bugtrack_url'

      warnings.warn(msg)

    error: command 'cc' failed with exit status 1

    ----------------------------------------
    Command "/home/travis/virtualenv/pypy3-2.4.0/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-zhoeh6/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-bdds0h-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/travis/virtualenv/pypy3-2.4.0/include/site/python3.2" failed with error code 1 in /tmp/pip-build-zhoeh6/lxml

The command "pip install ." failed and exited with 1 during .

Your build has been stopped.

[scoder (scoder)]

Yes, these are known issues. They'll hopefully be fixed with the next release.

[scoder (scoder)]

Still, note that PyPy, and especially PyPy3, is rather buggy and incomplete when it comes to running native extensions, so it may or may not work for you.

[Eeo Jun (packwolf58)]

Noted. Any reason why PyPy is buggy when running native extensions?

On Fri, Apr 10, 2015 at 9:07 PM, scoder <email address hidden> wrote:

> Still, note that PyPy, and especially PyPy3, is rather buggy and
> incomplete when it comes to running native extensions, so it may or may
> not work for you.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1442427
>
> Title:
> lxml fails to install on pypy3
>
> Status in lxml - the Python XML toolkit:
> Confirmed
>
> Bug description:
> Hi, we have tried to support PyPy 3 for our python CI builds today,
> but it fails to install only on PyPy 3. I have tried restarting the
> build but to no avail. Here's the full log:
>
> Using worker: worker-linux-docker-527b567d.prod.travis-ci.org:travis-
> linux-6
>
> Build system information
> Build language: python
> Build image provisioning date and time
> Thu Feb 5 15:09:33 UTC 2015
> Operating System Details
> Distributor ID: Ubuntu
> Description: Ubuntu 12.04.5 LTS
> Release: 12.04
> Codename: precise
> Linux Version
> 3.13.0-29-generic
> Cookbooks Version
> a68419e https://github.com/travis-ci/travis-cookbooks/tree/a68419e
> GCC version
> gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
> Copyright (C) 2011 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions. There is
> NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
> PURPOSE.
>
> LLVM version
> clang version 3.4 (tags/RELEASE_34/final)
> Target: x86_64-unknown-linux-gnu
> Thread model: posix
> Pre-installed Ruby versions
> ruby-1.9.3-p551
> Pre-installed Node.js versions
> v0.10.36
> Pre-installed Go versions
> 1.4.1
> Redis version
> redis-server 2.8.19
> riak version
> 2.0.2
> MongoDB version
> MongoDB 2.4.12
> CouchDB version
> couchdb 1.6.1
> Neo4j version
> 1.9.4
> RabbitMQ Version
> 3.4.3
> ElasticSearch version
> 1.4.0
> Installed Sphinx versions
> 2.0.10
> 2.1.9
> 2.2.6
> Default Sphinx version
> 2.2.6
> Installed Firefox version
> firefox 31.0esr
> PhantomJS version
> 1.9.8
> ant -version
> Apache Ant(TM) version 1.8.2 compiled on December 3 2011
> mvn -version
> Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1;
> 2014-12-14T17:29:23+00:00)
> Maven home: /usr/local/maven
> Java version: 1.7.0_76, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-7-oracle/jre
> Default locale: en_US, platform encoding: ANSI_X3.4-1968
> OS name: "linux", version: "3.13.0-29-generic", arch: "amd64", family:
> "unix"
>
> $ git clone --depth=50 --branch=master git://
> github.com/libextract/libextract.git libextract/libextract
> Cloning into 'libextract/libextract'...
> remote: Counting objects: 345, done.
> remote: Compressing objects: 100% (236/236), done.
> remote: Total 345 (delta 199), reused 250 (delta 106), pack-reused 0
> Receiving objects: 100% (345/345), 148.54 KiB | 0 bytes/s, done.
> Resolving deltas: 100% (199/199), done.
> Checking connectivity... done.
> $ cd libextract/libextrac...

[scoder (scoder)]

This is the wrong place to ask for a reason. It's just buggy. Some things
don't work as expected, some don't work at all. Sometimes, working around
these bugs is easy, sometimes it's not easy and sometimes it's impossible.
Some bugs get fixed, others don't.

PyPy3 is particularly buggy as it is a fork of the PyPy2 implementation and
not everything has been converted and adapted for Python 3 semantics yet.
Other things have been wrongly adapted or got broken along the way. Work in
progress, I guess.

[scoder (scoder)]

Marking as "won't fix" because it's up to PyPy to come up with a working pypy3 release.