Pythoscope does not detect conflicts caused by naming convention used for tests.

[Pieter Van Dyck]

environment: windows, python 2.5, pythoscope 1.4.2

test case in attachment

log:
E:\projects\scope_tests\test1>pythoscope --init
INFO: Inspecting module my\cool_class.py.
INFO: Inspecting module my\__init__.py.
INFO: Inspecting module my_cool_class.py.

E:\projects\scope_tests\test1>pythoscope *.py my\*.py
INFO: my\cool_class.py hasn't changed since last inspection, skipping.
INFO: my\__init__.py hasn't changed since last inspection, skipping.
INFO: my_cool_class.py hasn't changed since last inspection, skipping.
INFO: No changes discovered in the source code, skipping dynamic inspection.
INFO: Generating tests for module my_cool_class.py.
INFO: Adding generated TestA to tests\test_my_cool_class.py.
INFO: Generating tests for module my\__init__.py.
INFO: Generating tests for module my\cool_class.py.
INFO: Adding generated TestB to tests\test_my_cool_class.py.
INFO: Test case TestA.test_member3 already exists in tests\test_my_cool_class.py
, skipping.

Both modules my_cool_class and my.cool_class will end up trying to build a test called test_my_cool_class but
Pythoscope does not detect the conflict.
The result is that the test class contains some tests for both modules and misses some of whatever module happens to come 2nd.

The case may seem a bit contrived but will (and has) crop up when refactoring prefixed code to proper modules for instance.

Proper handling of this case may require a bit of work, but flagging the problem would already improve the situation.

[Michal Kwiatkowski (ruby)]

Thank you for pointing this one out! Bug #475504 is related, but it talks about test cases at the test method level, and I totally forgot about conflicts at the package/module level.

OK, so we have a package "my" with a module "cool_class" and a module "my_cool_class". Should their test land in the same test module? Maybe, their names are related. If not, how should they be named? And, the most important question, can pythoscope figure those things out reliably in all situations?

I'm thinking, maybe a bit of configuration would work as a solution to this problem? If there was a file where you could map application modules to test modules (like "my/cool_class => legacy_my_cool_class_test") that would allow pythoscope to use conventions by default and still allow a developer to override those when needed. What do you think, is that a good solution to this problem?

[Pieter Van Dyck (vandyck-pieter)]

2009/11/23 Michal Kwiatkowski <email address hidden>:
> Thank you for pointing this one out! Bug #475504 is related, but it
> talks about test cases at the test method level, and I totally forgot
> about conflicts at the package/module level.
>
> OK, so we have a package "my" with a module "cool_class" and a module
> "my_cool_class". Should their test land in the same test module? Maybe,
> their names are related.

Since pythoscope can't really tell if they're related entities, it
should probably
act as if they are not related. It's the way of least surprises.

> If not, how should they be named?

As long as pythoscope flattens the names, there will always be conflicts.
Is there a fundamental reason why pythoscope should not use packages
or subdirectories for tests?

> And, the most
> important question, can pythoscope figure those things out reliably in
> all situations?

It can, by keeping track of the difference between packages and modules.
This may be related to the work you planned for 1.5
Once projects start to have non-trivial amounts of test cases, you'd
want a way to group them anyway.

>
> I'm thinking, maybe a bit of configuration would work as a solution to
> this problem? If there was a file where you could map application
> modules to test modules (like "my/cool_class =>
> legacy_my_cool_class_test") that would allow pythoscope to use
> conventions by default and still allow a developer to override those
> when needed. What do you think, is that a good solution to this problem?

Not really.
* Configurations may lead to new name conflicts and pythoscope still
doesn't help in detecting them.
* The configuration work-around doesn't scale well to teams doing
parallel development as name conflicts can be (silently) introduced
(by another team for instance) when you thought you were safe.

Also, I don't think it's nice if tools cause trouble and then ask
their users to solve it for them ;-)

cheers,
Pieter

[Michal Kwiatkowski (ruby)]

Very good points! For some reason I haven't thought of the simplest solution - to have test structure correspond *exactly* to code structure. That means creating test modules for application modules (already implemented) and creating test packages for application packages (TBD). Thank you for the input, I would have made things more complicated (and still broken!) without it.