Add a [KnownBug] attribute to tests

[Shay Rojansky]

[Issue now tracked at https://github.com/nunit/nunit-framework/issues/44]

In many cases a bug is reported to a project along with the unit test. The unit test needs to be integrated, but it may take some time before it can be resolved. In this case, it isn't clear what its state should be:

* If it is integrated as is, the unit test fails and causes the whole build to fail. This hinders development on other features and bugs - it's hard to know whether a red unit test represents a regression or just an unrelated open bug.
* [Ignore] seems like the current best solution, but is semantically unclear. Tests can be [Ignore]d for many reasons (temporary breakage) and sometimes remain that way for a long time.

A [KnownBug] mark would semantically handle this, and would effectively create an "open bug list" within the test suite. It would also allow GUIs to visually indicate the state as distinct from [Ignore].

If this seems like a good idea I'll be happy to submit a patch.

[Charlie Poole (charlie.poole)]

This sort have thing has been discussed before and some folks have
even added a similar attribute in their own builds. I guess it's time
has come. I'm moving this to the nunit-3.0 project, however, since we
will not be continuing to add features to NUnit 2.6.2.

In terms of semantics, I gather you would like to have the test
skipped, rather than run. Correct?

Another possibility is a KnownFailure attribute, which would allow the
test to run and expect a failure, perhaps with some specific error
message, but skipping the test seems sensible. We'll need to think
about how this would be used when the programmer was trying to fix the
bug.

On Mon, Aug 26, 2013 at 3:47 AM, Shay Rojansky <email address hidden> wrote:
> Public bug reported:
>
> In many cases a bug is reported to a project along with the unit test.
> The unit test needs to be integrated, but it may take some time before
> it can be resolved. In this case, it isn't clear what its state should
> be:
>
> * If it is integrated as is, the unit test fails and causes the whole build to fail. This hinders development on other features and bugs - it's hard to know whether a red unit test represents a regression or just an unrelated open bug.
> * [Ignore] seems like the current best solution, but is semantically unclear. Tests can be [Ignore]d for many reasons (temporary breakage) and sometimes remain that way for a long time.
>
> A [KnownBug] mark would semantically handle this, and would effectively
> create an "open bug list" within the test suite. It would also allow
> GUIs to visually indicate the state as distinct from [Ignore].
>
> If this seems like a good idea I'll be happy to submit a patch.
>
> ** Affects: nunitv2
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are subscribed to NUnit
> Extended Testing Platform.
> https://bugs.launchpad.net/bugs/1216854
>
> Title:
> Add a [KnownBug] attribute to tests
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nunitv2/+bug/1216854/+subscriptions

[Shay Rojansky (roji)]

OK, thanks for giving this the attention.

Yes, I was thinking it should be skipped - that functionally it would be like the [Explicit] attribute. When the programmer starts working on it, they could run the test explicitly and remove the [KnownBug] afterwards, or remove it before starting work, in which case they have a normal failing test on which they're working.

Running the test and expecting failure is also an option, although I'm not sure what the value of that would be. It raises the question of what to do with a test marked as [KnownBug] or [KnownFailure], but that actually succeeds - this could theoretically be flagged to the programmer, prompting them to remove the attribute, but it seems needlessly complicated...