TestwithScenarios incompatible with nose 1.1.2

Bug #872887 reported by John A Meinel on 2011-10-12
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
testscenarios
Low
Unassigned

Bug Description

I tried using TestScenarios with nosetests by inheriting from TestWithScenarios.

According to the pypi entry: http://pypi.python.org/pypi/testscenarios
"(which support these hooks like nose, trial, tribunal) will still run your scenarios. (Of course, if you are using the subclassing approach this is already a surety).".

It at least sounds like subclassing should be supported by all test runners that support the 'unittest' api.

However, nosetests fails with:
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python26\BASE\lib\site-packages\nose\case.py", line 133, in run
    self.runTest(result)
  File "C:\Python26\BASE\lib\site-packages\nose\case.py", line 151, in runTest
    test(result)
AssertionError: ResultProxy for Test(<u1db.tests.test_backends.DatabaseTests testMethod=test_whats_changed_returns_one_id_for_multiple_changes>) (50991056) was called with test <u1db.tests.test_backends.DatabaseTests testMethod=test_whats_changed_returns_one_id_for_multiple_changes> (51092560)

It looks like the real object id() is changing. (not the test.id() but id(test)), and apparently nosetests decided that is not ok. Looking at nose.proxy.ResultProxy it has this function:
    def assertMyTest(self, test):
        # The test I was called with must be my .test or my
        # .test's .test. or my .test.test's .case

        case = getattr(self.test, 'test', None)
        assert (test is self.test
                or test is case
                or test is getattr(case, '_nose_case', None)), (
                "ResultProxy for %r (%s) was called with test %r (%s)"
                % (self.test, id(self.test), test, id(test)))

I don't know why it requires that, but it seems that the way TestScenarios is multiplying its tests during run and then passing the results to the ResultProxy object is forbidden by nose. I tried hacking testtools to set new.test = orig_test or .case or, etc. But it looks like you have to set orig_test.test = new_test. And that doesn't make much sense in the general case, since one test will get farmed out to N scenario tests.

For now, I've just switched from "nosetests" to "python -m testtools.run discover". But it does seem like the two codebases need to find a way to get along :).

Robert Collins (lifeless) wrote :

You need to multiply_scenarios before nose sees the tests, because nose makes untrue assumptions about the unittest protocol.

Changed in testscenarios:
status: New → Invalid

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/12/2011 9:56 PM, Robert Collins wrote:
> You need to multiply_scenarios before nose sees the tests, because
> nose makes untrue assumptions about the unittest protocol.
>
> ** Changed in: testscenarios Status: New => Invalid
>

So I think it is still true that TestWithScenarios is incompatible
with nosetests. Perhaps wontfix is more accurate than invalid.

I also couldn't find a good way to easily do generate_scenarios ahead
of time, because nosetests was treating "load_tests" as another test
to be run, rather than as a test suite loading function (on python 2.6).

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk6X9+QACgkQJdeBCYSNAAMPlACcDoka+mD+HYqMMzeO6REUX7hK
z/0AoMUAbnDtoVatsMnrBh39J70otqHy
=NaQP
-----END PGP SIGNATURE-----

Roberto Lobo (rhlobo+launchpad) wrote :

@Robert Collins (lifeless) - Could you be more specific into how to do that??
@John A Meinel (jameinel) - Where you able to find a way to make it work?

John A Meinel (jameinel) wrote :

I wasn't able to get it working with nose, IIRC. I switched test runners (to testtools.run I believe).

I think I'm using 'load_tests()' which is in the stdlib as of python 2.7 or so. So if nose understood that mechanism, it could probably work.

Roberto Lobo (rhlobo+launchpad) wrote :

Right... Thanks for the quick answer.

Testscenarios also doesn't work when using it's loader:

======================================================================
ERROR: Adapter test runner load hooks to call generate_scenarios.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/akornienko/projects/ceilometer/.tox/py27/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/akornienko/projects/ceilometer/.tox/py27/local/lib/python2.7/site-packages/nose/util.py", line 613, in newfunc
    return func(*arg, **kw)
  File "/home/akornienko/projects/ceilometer/.tox/py27/local/lib/python2.7/site-packages/testscenarios/scenarios.py", line 106, in load_tests_apply_scenarios
    if getattr(params[0], 'suiteClass', None) is not None:
IndexError: tuple index out of range

Changed in testscenarios:
status: Invalid → New
Robert Collins (lifeless) wrote :

Alexei - is that with nose? Nose isn't unittest API compatible...

Changed in testscenarios:
status: New → Triaged
importance: Undecided → Low
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers