2.9.x doesn't find tests in dirs named ftests, but 3.2.x does

[Paul Winkler]

The zope 3 test runner by default looks only for directories named "tests". The test.py script for zope 3 then overrides this to allow directories named "tests" or "ftests", with this line:

defaults = ['--tests-pattern', '^f?tests$', '--test-path', src]

But the test.py and bin/test.py scripts for zope 2 doesn't find any directories named ftests, because of this line:

defaults = '--tests-pattern ^tests$ -v'.split()

IMO this is a bug because:

- it's an undocumented behavior change between 2.8 and 2.9. There is nothing about this in CHANGES.txt or in the help output of test.py.

- it's inconsistent behavior between zope 2 and 3 which surprises the user given that they share testing infrastructure.

- third-party products such as Plone use the "ftests" naming convention extensively. Under 2.9, these simply won't run and there is no explicit documentation explaining why this changed or how to work around this (use the --tests-pattern argument) or how to move forward (presumably the answer is "use layers").

The simple fix would be to change the line in zope 2's bin/test.py and test.py to read:

defaults = '--tests-pattern ^f?tests$ -v'.split()

After that, it would no longer be necessary to move our functional tests into directories named "tests".

Unfortunately that still doesn't bring full backward compatibility; the "-f" option under 2.9 can't work as expected until third-party functional tests are updated to (as I understand it) run in layers whose names don't contain "unit". Instead, *no* tests would run with -f, and *all* tests would run without -f.
--tests-pattern can be provided as a workaround.

But at least we wouldn't have to rearrange our source trees in a way that contradicts the zope 3 books, which means losing history if we happen to use CVS ...

The docs in zope/testing are great, but all the above could really be better spelled out IMO. I wasted several hours figuring this out.

[Philipp von Weitershausen (philikon)]

We can't really support 'ftests' packages in Zope 2 right now. The reason is that Zope 2 includes many Zope 3 packages that have 'ftests' packages which contain Zope 3 functional tests (e.g. zope.testbrowser or zope.formlib). These tests can't possibly run on Zope 2, at least not as it exists today. For starters, Zope 2 is missing ftesting.zcml and probably a lot more basic infrastructure that is just there in Zope 3.

The 'tests' vs. 'ftests' convention has served us well so far in Zope 2. Zope 2 simply ignores 'ftests' while Zope 3 runs them. The fact that functional tests are contained in 'ftests' is arbitrary by now, but the convention is so that Zope 2 won't be bothered with Zope 3's functional tests. Hence, Plone shouldn't really use the 'ftests' package...

Here are some possible long-term solutions:

1. We leave everything as it is. Zope 3 style functional tests will be put in 'ftests'. Zope 2 can never use 'ftests' because it would break otherwise.

2. We add a way to let the testrunner know that certain test layer's aren't supported. Basically, we could make it so that an exception raised during layer setup will cause the tests in that layer not to be executed (currently the exception raises all the way to the top and will cause the test runner to terminate at that point)

3. We let Zope 2 grow everything it needs to run Zope 3 functional tests. This could quite possibly work. While this is contra-zope.app-should-not-ship-with-Zope2, it is sort of pro-Zope5 (unifying the two platforms a bit more). Of course, we'll at some point want functional tests in Zoep 2/Five as well, so there's good reason for making them work in Zope 2. For that, it'd be good if functional tests for Zope 2 and 3 would be similar enough that test cases could be run on either platform. I presume this could take a while, though.

[Hanno Schlichting (hannosch)]

Zope 2 no longer contains a test runner which imposes any default test pattern. This is up to the test runner of your choosing now.