testsuite and tox

[IAN DELANEY]

tox in gentoo is simply toxic when it comes to running tests. It woud be so much more sane to assure that the testsuite will pass on the system installed dependencies. While tox has its place, I avoid it like the plague in testsuites in python based packages as a gentoo dev package maintainer. Gentoo also has a rule of no connecting to the internet in a test phase. Sheeeeeeesh.

Sure nosetests is just another test runner, but it's my first port of call.

testuser@archtester /mnt/gen2/TmpDir/portage/dev-python/python-swiftclient-1.6.0/work/python-swiftclient-1.6.0 $ ./run_tests.sh
GLOB sdist-make: /mnt/gen2/TmpDir/portage/dev-python/python-swiftclient-1.6.0/work/python-swiftclient-1.6.0/setup.py
py27 create: /mnt/gen2/TmpDir/portage/dev-python/python-swiftclient-1.6.0/work/python-swiftclient-1.6.0/.tox/py27
py27 installdeps: -r/mnt/gen2/TmpDir/portage/dev-python/python-swiftclient-1.6.0/work/python-swiftclient-1.6.0/requirements.txt, -r/mnt/gen2/TmpDir/portage/dev-python/python-swiftclient-1.6.0/work/python-swiftclient-1.6.0/test-requirements.txt
py27 inst: /mnt/gen2/TmpDir/portage/dev-python/python-swiftclient-1.6.0/work/python-swiftclient-1.6.0/.tox/dist/python-swiftclient-1.6.0.zip
py27 runtests: commands[0]
running testr
running=${PYTHON:-python} -m subunit.run discover -t ./ ./tests --list
running=${PYTHON:-python} -m subunit.run discover -t ./ ./tests --load-list /tmp/tmpbYu4tl
running=${PYTHON:-python} -m subunit.run discover -t ./ ./tests --load-list /tmp/tmpn6K8gN
running=${PYTHON:-python} -m subunit.run discover -t ./ ./tests --load-list /tmp/tmp30rhT8
running=${PYTHON:-python} -m subunit.run discover -t ./ ./tests --load-list /tmp/tmpQ_1yJU
Ran 75 tests in 0.796s
PASSED (id=0)
___________________________________ summary ____________________________________
  py27: commands succeeded
  congratulations :)
GLOB sdist-make: /mnt/gen2/TmpDir/portage/dev-python/python-swiftclient-1.6.0/work/python-swiftclient-1.6.0/setup.py
pep8 create: /mnt/gen2/TmpDir/portage/dev-python/python-swiftclient-1.6.0/work/python-swiftclient-1.6.0/.tox/pep8
pep8 installdeps: -r/mnt/gen2/TmpDir/portage/dev-python/python-swiftclient-1.6.0/work/python-swiftclient-1.6.0/requirements.txt, -r/mnt/gen2/TmpDir/portage/dev-python/python-swiftclient-1.6.0/work/python-swiftclient-1.6.0/test-requirements.txt
pep8 inst: /mnt/gen2/TmpDir/portage/dev-python/python-swiftclient-1.6.0/work/python-swiftclient-1.6.0/.tox/dist/python-swiftclient-1.6.0.zip
pep8 runtests: commands[0]
pep8 runtests: commands[1]
____________________________________________________________ summary _____________________________________________________________
  pep8: commands succeeded
  congratulations :)

says tox worked and the testsuite passed. Well, nice, but for breaking gentoo policy in running tests at the very first step.

testuser@archtester /mnt/gen2/TmpDir/portage/dev-python/python-swiftclient-1.6.0/work/python-swiftclient-1.6.0 $ PYTHONPATH=. nosetests tests
F..........................................................................
======================================================================
FAIL: tests.test_multithreading.TestMultiThreadingManager.test_instantiation
----------------------------------------------------------------------
_StringException: Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/mock.py", line 1224, in patched
    return func(*args, **keywargs)
  File "/mnt/gen2/TmpDir/portage/dev-python/python-swiftclient-1.6.0/work/python-swiftclient-1.6.0/tests/test_multithreading.py", line 248, in test_instantiation
    self.assertEqual(sys.stdout, thread_manager.print_stream)
  File "/usr/lib64/python2.7/site-packages/testtools/testcase.py", line 322, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/usr/lib64/python2.7/site-packages/testtools/testcase.py", line 417, in assertThat
    raise MismatchError(matchee, matcher, mismatch, verbose)
MismatchError: !=:
reference = <StringIO.StringIO instance at 0x1305f38>
actual = <StringIO.StringIO instance at 0xdcd680>

----------------------------------------------------------------------
Ran 75 tests in 1.699s

FAILED (failures=1)

says we have a problem possibly with system installed mock or testtools and it did in fact run 75 tests.

testuser@archtester /mnt/gen2/TmpDir/portage/dev-python/python-swiftclient-1.6.0/work/python-swiftclient-1.6.0 $ PYTHONPATH=. py.test tests
====================================================== test session starts =======================================================
platform linux2 -- Python 2.7.5 -- pytest-2.3.5
collected 75 items

tests/test_multithreading.py sssssssssss
tests/test_swiftclient.py ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss

says let's take py.test out of the equation.

Now you could take the easy way out here and dismiss this whole entry since your 'virtual environment making' sees the testsuite pass. That leaves something amiss here though.

Now here is a classic conflict.

./.tox/pep8/lib/python2.7/site-packages/python_subunit-0.0.15-py2.7.egg-info/PKG-INFO:

says the virtual env has installed python-subunit. Gentoo's python-subunit is in fact called subunit. We have

/usr/lib64/python2.7/site-packages/subunit/.......

oooops.

I am guessing the use of a virtual env. assumes the instances of the virtual installed deps will perfectly match the system installed packages. Sorry to shatter that one for you.

So, could you consider making a testsuite that runs perfectly under nose or pytest or
$ python tests/test_all.py?
It's really quite a sane request. 'We' can't use tox as a test runner for all the various reasons above. At this point I have the tox run testsuite declaring the tests fine and who knows what will happen on using system installed deps once installed. Only a few days ago I got a different result again where the issue was the version of testtools.

[IAN DELANEY (johneed)]

well after the neutron fiasco, forget it

[Monty Taylor (mordred)]

Hi!

I gave specific instructions in your bug on nova, but I think for the record I'll respond here too. Your request is a valid one, and I totally agree that the testsuite should run outside of tox. However, we use testr to run tests, not nosetests or py.test. I'm happy to go into the details of that decision, but suffice it to say, simply running testr will run the tests against system installed dependencies and neither tox nor pip nor the internet will be affected.

Thanks for your work on gentoo!

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/161043
Committed: https://git.openstack.org/cgit/openstack/python-swiftclient/commit/?id=a4fb70ece189aff85f234ab6b3f275b69e936c03
Submitter: Jenkins
Branch: master

commit a4fb70ece189aff85f234ab6b3f275b69e936c03
Author: Tim Burke <email address hidden>
Date: Tue Mar 3 12:35:03 2015 -0800

    Compare each chunk of large objects when uploading

    Previously, we compared the ETag from Swift against the MD5 of the
    entire large object. However, the ETag for large objects is generally
    the MD5 of the concatenation of the ETags for each segment, unless the
    object is a DLO whose segments span more than one page of a container
    listing. Rather than worry about ETags, just compare each chunk of the
    segmented file. This allows the use of --skip-identical when uploading
    SLOs and DLOs.

    Additionally, there are several test-related improvements:
     * The default arguments for OutputManager are now evaluated on
       construction, rather than on definition, so that
       TestOutputManager.test_instantiation will succeed when using nosetest
       as a test runner. (See also: bug 1251507)
     * An account_username option is now available in the functional tests
       config file for auth systems that do not follow the account:username
       format.
     * CaptureOutput no longer writes to the captured stream, and
       MockHttpTest now captures output. These were polluting test output
       unnecessarily. (See also: bug 1201376)

    Change-Id: Ic484e9a0c186c9283c4012c6a2fa77b96b8edf8a
    Closes-Bug: #1201376
    Closes-Bug: #1379252
    Related-Bug: #1251507