Means to include/exclude categories from test code

[Matthias Kläy]

Hi NUnit,

Use of categories is a great way to include/exclude tests depending on the physical properties of the test machine. In combination with configuration settings this feature becomes even more powerful because it allows to configure the required physical test infrastructure, and the same tests can be executed on different physical machines. In my case for example, settings are:
  'SerialPortA' and 'SerialPortAIsAvailable' (e.g. COM1 and true)
  'SerialPortB' and 'SerialPortBIsAvailable' (e.g. COM2 and false)
  'SerialPortsAreInterconnected' (e.g. false)

When running tests, I can now manually exclude those categories which cannot be executed on the current machine. However, tests should automatically be excluded/ignored if the test machine doesn't provide the required infrastructure. Currently I achive this by:
  if (SerialPortAIsAvailable)
      <Run test case on SerialPortA>
  else
      Assert.Ignore("'SerialPortA' is not available, therefore this test is ignored. Ensure that...

This implementation requires that every test case does perform this check. It would be much more convenient if a whole category was automatically excluded/ignored by the NUnit test runner. I'd prefer 'ignore' because I'd prefer to clearly see that tests were skipped and a yellow instead of a green bar.

Currently using:
> NUnit 2.5.10 (not yet upgraded to 2.6.2 due to the fact that the upgrade lead to issues with .NET 3.5)
> Gui Runner

Best regards
Matthias

YAT - Yet Another Terminal.
Visit YAT at http://sourceforge.net/projects/y-a-terminal/.
Contact YAT by mailto:<email address hidden>.

[Charlie Poole (charlie.poole)]

It sounds like you want a way to extend the existing PlatformAttribute
with your own code, so that you could write something like:

[Platform("SerialPortAIsAvailable")]

and place it on a test or a fixture - or even at a higher level SetUpFixture.

This is a major bit of work, which is planned for NUnit 3.0. In fact,
we probably will replace the PlatformAttribute with something
more general like [Include] and [Exclude] allowing for automatic
detection of Operating System, Runtime, Culture and a few
other things by NUnit, together with a way to extend it with your own
tags and code.

If this is the sort of thing you are after, I'll kick this bug to
NUnit 3.0. If you're looking for something simpler please clarify
before I do that!

As a workaround, I suggest you put an Assume(SerialPortAIsAvailable)
at the head of each test requiring it. This is a bit
less intrusive than the code you gave above and will give an
Inconclusive result when the port is not available. It will not
give you a yellow bar, but you can do that by simply including one
test that uses Ignore as above.

By the way, I hope you have told us about whatever problems prevent
you from upgrading to 2.6.2. Obviously, any fix we
make for this will be in a future release, so you'll eventually want
to upgrade. IME, waiting too long makes it harder.

Charlie

On Fri, Dec 28, 2012 at 4:46 AM, Matthias Kläy <email address hidden> wrote:
> Public bug reported:
>
> Hi NUnit,
>
> Use of categories is a great way to include/exclude tests depending on the physical properties of the test machine. In combination with configuration settings this feature becomes even more powerful because it allows to configure the required physical test infrastructure, and the same tests can be executed on different physical machines. In my case for example, settings are:
> 'SerialPortA' and 'SerialPortAIsAvailable' (e.g. COM1 and true)
> 'SerialPortB' and 'SerialPortBIsAvailable' (e.g. COM2 and false)
> 'SerialPortsAreInterconnected' (e.g. false)
>
> When running tests, I can now manually exclude those categories which cannot be executed on the current machine. However, tests should automatically be excluded/ignored if the test machine doesn't provide the required infrastructure. Currently I achive this by:
> if (SerialPortAIsAvailable)
> <Run test case on SerialPortA>
> else
> Assert.Ignore("'SerialPortA' is not available, therefore this test is ignored. Ensure that...
>
> This implementation requires that every test case does perform this
> check. It would be much more convenient if a whole category was
> automatically excluded/ignored by the NUnit test runner. I'd prefer
> 'ignore' because I'd prefer to clearly see that tests were skipped and a
> yellow instead of a green bar.
>
> Currently using:
>> NUnit 2.5.10 (not yet upgraded to 2.6.2 due to the fact that the upgrade lead to issues with .NET 3.5)
>> Gui Runner
>
> Best regards
> Matthias
>
> YAT - Yet Another Terminal.
> Visit YAT at http://sourceforge.net/projects/y-a-terminal/.
> Contact YAT by mailto:<email address hidden>.
>
> ** Affects: nunitv2
> Importance: Undecided
> Status: New
>
> --
> You recei...

[Matthias Kläy (matthias-klaey)]

Hi Charlie,

Thanks for the detailed response! Aiming for NUnit 3.0 is perfectly fine, I totally understand that it makes little sense to still do significant work in 2.x. Something like [Include] and [Exclude] in 3.0 would fulfill my requirements. First I thought that the category attribute could include/exclude itself based on some condition. But giving it a second thought made me conclude that [Include] and [Exclude] must be something in addition / on top of the current [Category] infrastructure. But thinking about it again, having some logic right in the category attribute would be an obvious and compact solution. I'm sure you'll find the best approach to implement this feature.

Regarding the issues with 2.6.x:
A couple of weeks ago I simply uninstalled 2.5.10 and installed 2.6.2 and ran my tests. But that didn't work with .NET 3.5 and VS2008. I cannot remember the error details, but I guess it's just something minor. I didn't want to bother looking into the issue in more detail since I am about finalizing a version of my software and am trying to focus on that. But some time early in 2013 I will upgrade to 2.6.x. If it's an issue I cannot solve I will of course report it to you.

Best regards,
Matthias