TestwithScenarios incompatible with nose 1.1.2

[John A Meinel]

I tried using TestScenarios with nosetests by inheriting from TestWithScenarios.

According to the pypi entry: http://pypi.python.org/pypi/testscenarios
"(which support these hooks like nose, trial, tribunal) will still run your scenarios. (Of course, if you are using the subclassing approach this is already a surety).".

It at least sounds like subclassing should be supported by all test runners that support the 'unittest' api.

However, nosetests fails with:
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python26\BASE\lib\site-packages\nose\case.py", line 133, in run
    self.runTest(result)
  File "C:\Python26\BASE\lib\site-packages\nose\case.py", line 151, in runTest
    test(result)
AssertionError: ResultProxy for Test(<u1db.tests.test_backends.DatabaseTests testMethod=test_whats_changed_returns_one_id_for_multiple_changes>) (50991056) was called with test <u1db.tests.test_backends.DatabaseTests testMethod=test_whats_changed_returns_one_id_for_multiple_changes> (51092560)

It looks like the real object id() is changing. (not the test.id() but id(test)), and apparently nosetests decided that is not ok. Looking at nose.proxy.ResultProxy it has this function:
    def assertMyTest(self, test):
        # The test I was called with must be my .test or my
        # .test's .test. or my .test.test's .case

        case = getattr(self.test, 'test', None)
        assert (test is self.test
                or test is case
                or test is getattr(case, '_nose_case', None)), (
                "ResultProxy for %r (%s) was called with test %r (%s)"
                % (self.test, id(self.test), test, id(test)))

I don't know why it requires that, but it seems that the way TestScenarios is multiplying its tests during run and then passing the results to the ResultProxy object is forbidden by nose. I tried hacking testtools to set new.test = orig_test or .case or, etc. But it looks like you have to set orig_test.test = new_test. And that doesn't make much sense in the general case, since one test will get farmed out to N scenario tests.

For now, I've just switched from "nosetests" to "python -m testtools.run discover". But it does seem like the two codebases need to find a way to get along :).

[Robert Collins (lifeless)]

You need to multiply_scenarios before nose sees the tests, because nose makes untrue assumptions about the unittest protocol.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/12/2011 9:56 PM, Robert Collins wrote:
> You need to multiply_scenarios before nose sees the tests, because
> nose makes untrue assumptions about the unittest protocol.
>
> ** Changed in: testscenarios Status: New => Invalid
>

So I think it is still true that TestWithScenarios is incompatible
with nosetests. Perhaps wontfix is more accurate than invalid.

I also couldn't find a good way to easily do generate_scenarios ahead
of time, because nosetests was treating "load_tests" as another test
to be run, rather than as a test suite loading function (on python 2.6).

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk6X9+QACgkQJdeBCYSNAAMPlACcDoka+mD+HYqMMzeO6REUX7hK
z/0AoMUAbnDtoVatsMnrBh39J70otqHy
=NaQP
-----END PGP SIGNATURE-----

[Roberto Lobo (rhlobo+launchpad)]

@Robert Collins (lifeless) - Could you be more specific into how to do that??
@John A Meinel (jameinel) - Where you able to find a way to make it work?

[John A Meinel (jameinel)]

I wasn't able to get it working with nose, IIRC. I switched test runners (to testtools.run I believe).

I think I'm using 'load_tests()' which is in the stdlib as of python 2.7 or so. So if nose understood that mechanism, it could probably work.

[Roberto Lobo (rhlobo+launchpad)]

Right... Thanks for the quick answer.

[Alexei Kornienko (alexei-kornienko)]

Testscenarios also doesn't work when using it's loader:

======================================================================
ERROR: Adapter test runner load hooks to call generate_scenarios.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/akornienko/projects/ceilometer/.tox/py27/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/akornienko/projects/ceilometer/.tox/py27/local/lib/python2.7/site-packages/nose/util.py", line 613, in newfunc
    return func(*arg, **kw)
  File "/home/akornienko/projects/ceilometer/.tox/py27/local/lib/python2.7/site-packages/testscenarios/scenarios.py", line 106, in load_tests_apply_scenarios
    if getattr(params[0], 'suiteClass', None) is not None:
IndexError: tuple index out of range

[Robert Collins (lifeless)]

Alexei - is that with nose? Nose isn't unittest API compatible...

[Jelmer Vernooij (jelmer)]

Marking this as Won't Fix, with nose being obsolete.