Checkbox should include, in test reports, any variation in test plan

[Jeff Lane ]

Sometimes, testers de-select tests. In some cases, this is fine, in others it is not acceptable. It would be helpful to observe this by seeing, in the results, a diff of the original test plan (as seen on the initial Test Selection screen and the tests actually run (after a user has manually de-selected tests).

So for example, if the PXU indicates:

network/test1
network/test2
network/test3
memory/test1
memory/test2
storage/test1
storage/test2

and the user de-selects the network tests entirely, in the results somewhere would be this:

Tests manually deselected by user:
network/test1
network/test2
network/test3

This would help a lot in finding these things, since the tests are also dynamically executed depending on several factors, it's hard to always know that something has been removed.

For example, If a node has three NICs and the tests look like this:

network/device1-info
network/device2-info
network/device3-info
network/device1-test
network/device2-test
network/device3-test

but he deselects all but the tests for device 1, the actual tests look like this:

network/device1-info
network/device1-test

And it takes a lot more digging to realize "Hey, where are the device2 and device3 network tests?"

[Jeff Lane  (bladernr)]

YAY!!! Thank you! This came up again this week as I have a tester who has not only removed tests from the main test run, but also from subsequent re-tests :/ SO making those cases more obvious will definitely help.