python3 tests fail

[clayg]

There's a py34 tox environment - but the tests don't pass

ubuntu@saio:/vagrant/pyeclib$ tox -e py34
GLOB sdist-make: /vagrant/pyeclib/setup.py
py34 create: /vagrant/pyeclib/.tox/py34
py34 installdeps: -r/vagrant/pyeclib/test-requirements.txt
py34 inst: /vagrant/pyeclib/.tox/dist/pyeclib-1.4.0.zip
py34 installed: appdirs==1.4.2,coverage==4.3.4,docutils==0.13.1,Jinja2==2.9.5,MarkupSafe==0.23,nose==1.3.7,packaging==16.8,pyeclib==1.4.0,Pygments==2.2.0,pyparsing==2.1.10,six==1.10.0,Sphinx==1.2.3
py34 runtests: PYTHONHASHSEED='385433183'
py34 runtests: commands[0] | nosetests -v test/
Failure: ImportError (/vagrant/pyeclib/pyeclib_c.so: undefined symbol: _Py_ZeroStruct) ... ERROR
Failure: ImportError (/vagrant/pyeclib/pyeclib_c.so: undefined symbol: _Py_ZeroStruct) ... ERROR

======================================================================
ERROR: Failure: ImportError (/vagrant/pyeclib/pyeclib_c.so: undefined symbol: _Py_ZeroStruct)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/vagrant/pyeclib/.tox/py34/lib/python3.4/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/vagrant/pyeclib/.tox/py34/lib/python3.4/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/vagrant/pyeclib/.tox/py34/lib/python3.4/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/vagrant/pyeclib/.tox/py34/lib/python3.4/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/vagrant/pyeclib/.tox/py34/lib/python3.4/imp.py", line 235, in load_module
    return load_source(name, filename, file)
  File "/vagrant/pyeclib/.tox/py34/lib/python3.4/imp.py", line 171, in load_source
    module = methods.load()
  File "<frozen importlib._bootstrap>", line 1220, in load
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/vagrant/pyeclib/test/test_pyeclib_api.py", line 34, in <module>
    from pyeclib.ec_iface import ECBackendNotSupported
  File "/vagrant/pyeclib/pyeclib/ec_iface.py", line 29, in <module>
    from pyeclib_c import get_liberasurecode_version
ImportError: /vagrant/pyeclib/pyeclib_c.so: undefined symbol: _Py_ZeroStruct

======================================================================
ERROR: Failure: ImportError (/vagrant/pyeclib/pyeclib_c.so: undefined symbol: _Py_ZeroStruct)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/vagrant/pyeclib/.tox/py34/lib/python3.4/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/vagrant/pyeclib/.tox/py34/lib/python3.4/site-packages/nose/loader.py", line 418, in loadTestsFromName
    addr.filename, addr.module)
  File "/vagrant/pyeclib/.tox/py34/lib/python3.4/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/vagrant/pyeclib/.tox/py34/lib/python3.4/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/vagrant/pyeclib/.tox/py34/lib/python3.4/imp.py", line 235, in load_module
    return load_source(name, filename, file)
  File "/vagrant/pyeclib/.tox/py34/lib/python3.4/imp.py", line 171, in load_source
    module = methods.load()
  File "<frozen importlib._bootstrap>", line 1220, in load
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1471, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/vagrant/pyeclib/test/test_pyeclib_c.py", line 31, in <module>
    import pyeclib_c
ImportError: /vagrant/pyeclib/pyeclib_c.so: undefined symbol: _Py_ZeroStruct

----------------------------------------------------------------------
Ran 2 tests in 0.048s

FAILED (errors=2)
ERROR: InvocationError: '/vagrant/pyeclib/.tox/py34/bin/nosetests -v test/'
____________________________________________________________________________ summary _____________________________________________________________________________
ERROR: py34: commands failed

[clayg (clay-gerrard)]

tox -re py34 also doesn't work

... at least not until you `rm pyeclib_c.so` which causes `./.unittests` to fail (and `from swift.common.storage_policy import POLICIES`)

We need to find some way to make tox not use the `pyeclib_c.so` from the cwd when running tests that way

[clayg (clay-gerrard)]

Gunna try to get something going with https://review.openstack.org/#/c/438756/1

[Kota Tsuyuzaki (tsuyuzaki-kota)]

Can I make sure how to reproduce this issue?

I checked either py34 or py35 within trusty and xenial environments but I could not reproduce it. Both env works well (all tests passed as expected) Looking at comment #1, did you locate the shared library object to somewhere? Or perhaps you did already installed pyeclib to py3 env? I had thought it might be a trouble around setting for virtualenv but I'm not entirely sure for now.

Because of that, i could not to make sure if the proposed fix is reasonable to fix the issue. Am I missing something?

[Tim Burke (1-tim-z)]

Repro: http://paste.openstack.org/show/600802/

The trouble seems to be if you install it for a py2 environment, then try to run tox for py3.

[clayg (clay-gerrard)]

I had been installing pyeclib on my development vm's with `sudo pip install -e .` - which should be similar to `sudo python setup.py develop` in that it puts a egg-link to the source tree from which you install into your system packages.

When installed this way a `pyeclib_c.so` file is produced in my virtual environment (for the python version which installed it).

When I run py3 tests from the directory containing the `pyeclib_c.so` built with py2 - and run tests with nosetests with default arguments - then I end up importing the py2 built pyeclib_c.so in the current directory [1]

I think this is mostly only an issue if you do a development install of pyeclib - but it's also terrifying that tox + nose might not test the code it installed in it's virtualenv depending on where you run it from.

I think this is the fix for this issue:

https://review.openstack.org/#/c/438770/

^ oh look i even remembered to closes bug!

1. KUDOS to tim for spotting this in the traceback, and then also for the repro output.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/438770
Committed: https://git.openstack.org/cgit/openstack/pyeclib/commit/?id=28b0abc51bc39bec697b0742c47dfdf601c4359d
Submitter: Jenkins
Branch: master

commit 28b0abc51bc39bec697b0742c47dfdf601c4359d
Author: Clay Gerrard <email address hidden>
Date: Mon Feb 27 18:23:17 2017 -0800

    Don't let nose play with our PYTHONPATH.

    Closes-Bug: #1668447

    Change-Id: I5a9844bad285d992b6a3a01a7fd95634d02960cd