Test-free assembly given failure icon in tree display

[simon misys]

A test-free assembly is shown with the failure icon, ie, an X with red background, in the GUI runner tree display, as is its parent(s).

However, the progress bar is green, as expected (assuming all other tests pass).

Why would we want to include a test-free assembly? Because we want to create a single NUnit project for an existing large solution (40+ Visual Studio projects) containing all of its assemblies and executables. As we progressively add new tests to our product, we want NUnit to automatically pick them up without anyone having to remember to make sure the corresponding assembly/executable have been added to the NUnit project (and Debug/Release configurations).

It is off-putting (to say the least) and misleading for NUnit label such test-free assemblies as failures in the tree display.
It also contradicts what NUnit reports in the rest of the GUI (and when the NUnit project is run with the console runner).

So, I think NUnit should not show test-free assemblies as failures in the tree display. I think they should count as a success, ie, shown with a check with green background.

Thanks.

[Charlie Poole (charlie.poole)]

Such an assembly is not shown as a failure, but as "NotRunnable". This can be
determined by right-clicking it in the test tree and selecting "Properties." We
are aware that the Properties display is often not discovered by users from
past bug reports on other issues so we are working on some way to make
it more easily discoverable in the next major release.

I agree that there is a problem here, but I don't think that falsely showing the
test as passing is a good solution. We consider putting a non-test assembly
into the project as a user error of the same type as trying to run an
abstract class
or mismatching the arguments of a method.

The problem I see is that there is no other info about the problem. It is not
listed in either the Errors & Failures tab or the Tests not Run tab. It may also
be an issue that we use the same icon for it as for a test failure.
I'd like to re-
target this bug toward improving how we display the problem, rather than
redefining it as not a problem. For most users, an assembly with no tests
is simply a mistake, but if the result is surprising we don't want them to spend
an inordinate amount of time figuring it out.

Feel free to comment further on the issue. I'm interested in knowing what
process you use to generate new project files when you add a new assembly.
Can't you simply filter out those with no tests at that point?

Charlie
On Tue, Feb 1, 2011 at 8:29 AM, simon misys <email address hidden> wrote:
> Public bug reported:
>
> A test-free assembly is shown with the failure icon, ie, an X with red
> background, in the GUI runner tree display, as is its parent(s).
>
> However, the progress bar is green, as expected (assuming all other
> tests pass).
>
> Why would we want to include a test-free assembly?  Because we want to
> create a single NUnit project for an existing large solution (40+ Visual
> Studio projects) containing all of its assemblies and executables.  As
> we progressively add new tests to our product, we want NUnit to
> automatically pick them up without anyone having to remember to make
> sure the corresponding assembly/executable have been added to the NUnit
> project (and Debug/Release configurations).
>
> It is off-putting (to say the least) and misleading for NUnit label such test-free assemblies as failures in the tree display.
> It also contradicts what NUnit reports in the rest of the GUI (and when the NUnit project is run with the console runner).
>
> So, I think NUnit should not show test-free assemblies as failures in
> the tree display.  I think they should count as a success, ie, shown
> with a check with green background.
>
> Thanks.
>
> ** Affects: nunitv2
>     Importance: Undecided
>         Status: New
>
> --
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
> https://bugs.launchpad.net/bugs/711330
>
> Title:
>  Test-free assembly given failure icon in tree display
>

[simon misys (simon-marshall)]

> Such an assembly is not shown as a failure, but as "NotRunnable". [...]
> It may also
> be an issue that we use the same icon for it as for a test failure. [...]

Hi Charlie, that was my point, that it is shown in the tree display with the failure icon (or, if you prefer, with the same icon as used for failures). I did discover from its properties window that it is NotRunnable with no TestFixtures, but it was the use of the failure icon in the tree display itself that I was raising the issue for.

(I also noted that the tree parent is also given the failure icon, though I don't know if you consider it unfortunate that its properties window does not give any indication that the cause is due to a NotRunnable child assembly.)

The user can discover/see that that the node is test-free, since it is not expandable, but I accept that it might be desirable to visually distinguish test-free assemblies from assemblies with tests that succeed in the tree display. But, personally, I don't think it is desirable to make test-free assemblies visually equivalent to assemblies with tests that fail in the tree display. One is "hmmm, we have no tests for this bit, but we should get some" and one is "argh! we can't release this!! fire-fight!!!" I guess choices might be:

- use a new icon specifically for the purpose
- use the existing question-mark icon
- give the user the option to suppress the labelling of a test-free assembly with a non-success icon

> I'm interested in knowing what
> process you use to generate new project files when you add a new assembly.
> Can't you simply filter out those with no tests at that point?

Well, the easiest solution for us is to just manually add all of our existing Visual Studio projects' assemblies to the single new NUnit project. That way, as team members add tests, we don't have to do anything to the NUnit project. We don't expect to add many new Visual Studio projects, but we do expect to add many new tests, in a gradual process. We want to avoid the situation whereby someone adds tests but forgets to make sure the corresponding assembly is part of the NUnit project's configurations (Debug/Release). Does that answer your question? Is that a reasonable use case?

Thanks, Simon.

[Charlie Poole (charlie.poole)]

Hi Simon,

If this turns into a discussion, we should move it to the list. But lets
go another round here on a few points...

On Wed, Feb 2, 2011 at 2:00 AM, simon misys <email address hidden> wrote:
>> Such an assembly is not shown as a failure, but as "NotRunnable". [...]
>> It may also
>> be an issue that we use the same icon for it as for a test failure. [...]
>
> Hi Charlie, that was my point, that it is shown in the tree display with
> the failure icon (or, if you prefer, with the same icon as used for
> failures).  I did discover from its properties window that it is
> NotRunnable with no TestFixtures, but it was the use of the failure icon
> in the tree display itself that I was raising the issue for.
>
> (I also noted that the tree parent is also given the failure icon,
> though I don't know if you consider it unfortunate that its properties
> window does not give any indication that the cause is due to a
> NotRunnable child assembly.)

Yes, we migrate Non-Runnable up the tree because it's pretty severe
in some other cases - bad constructor for example. In fact, this may be
 a different thing, calling for a different category. I'll consider that.

> The user can discover/see that that the node is test-free, since it is
> not expandable, but I accept that it might be desirable to visually
> distinguish test-free assemblies from assemblies with tests that succeed
> in the tree display.  But, personally, I don't think it is desirable to
> make test-free assemblies visually equivalent to assemblies with tests
> that fail in the tree display.  One is "hmmm, we have no tests for this
> bit, but we should get some" and one is "argh!  we can't release this!!
> fire-fight!!!"  I guess choices might be:
>
> - use a new icon specifically for the purpose
> - use the existing question-mark icon
> - give the user the option to suppress the labelling of a test-free assembly with a non-success icon
>
>> I'm interested in knowing what
>> process you use to generate new project files when you add a new assembly.
>> Can't you simply filter out those with no tests at that point?
>
> Well, the easiest solution for us is to just manually add all of our
> existing Visual Studio projects' assemblies to the single new NUnit
> project.  That way, as team members add tests, we don't have to do
> anything to the NUnit project.  We don't expect to add many new Visual
> Studio projects, but we do expect to add many new tests, in a gradual
> process.  We want to avoid the situation whereby someone adds tests but
> forgets to make sure the corresponding assembly is part of the NUnit
> project's configurations (Debug/Release).  Does that answer your
> question?  Is that a reasonable use case?

So are you saying that you might create an assembly with no tests, but
later add tests to it? I guess that means you sometimes put tests into
the same assembly as the code they are testing.

I had been assuming that you would only create a new test assembly
if you had tests to put in it, so this is probably a use case that I wasn't
taking into account.

In any case, it's clear that something needs to be done here. I'll post
a comment when I have a better idea of what it i...

[simon misys (simon-marshall)]

> > (I also noted that the tree parent is also given the failure icon,
> > though I don't know if you consider it unfortunate that its properties
> > window does not give any indication that the cause is due to a
> > NotRunnable child assembly.)

> Yes, we migrate Non-Runnable up the tree because it's pretty severe
> in some other cases - bad constructor for example. In fact, this may be
> a different thing, calling for a different category. I'll consider that.

Sorry if I wasn't clear, I meant that from the parent property you couldn't tell that the failure was due to a NotRunnable with no TestFixtures. It's a minor nit - I mentioned it because you pointed out you could see in the properties of the test-free assembly that the status was NotRunnable with no TestFixtures, so you could understand why there was an issue with the assembly. I was just pointing out that you can't see that sort of information (that the reason is NotRunnable with no TestFixtures) in the parent property, so looking at the parent properties alone would not lead you to understand why the parent icon indicates failure (the parent properties says success but no more).

[simon misys (simon-marshall)]

> So are you saying that you might create an assembly with no tests, but
> later add tests to it? I guess that means you sometimes put tests into
> the same assembly as the code they are testing.

Sort of. We already have many assemblies (40) without tests. There is one assembly with tests, but no one apart from the developer who added it knew about it because he didn't create a top-level NUnit project for all to see. Only if you opened the assembly in NUnit (or looked at the specific code) would you be aware there were any tests. (None of the other developers have made any use of NUnit before.) But we plan to add tests bit by bit, and all developers will do this, as and when appropriate and convenient. Basically, we are not doing anything like enough regression testing and we want to address that with NUnit.

In the meantime, we want to create a top-level NUnit project that contains all (current) assemblies. That way, I think all developers are likely to be reminded that there are tests; all developers are likely to be aware that there are assemblies that aught to have tests but don't; developers don't have to check whether they need to add the assembly to the NUnit project when they do add tests (they do need to add new assemblies of course, but that is rarer than adding new tests to an existing assembly, and I suppose it's also possible that a new assembly might have no tests as you suggest); automated test runs are easier.

If we only added an assembly to the project as and when tests were added to it, I think we run the risk of developers not being encouraged to add tests; run the risk of forgetting to add the assembly to the NUnit project when tests are first added to an assembly. We want to set up the process to make it as likely and easy as possible that things will be done right.

Of course, test-free assemblies don't result in failure _status_ either in the progress bar or the client runner. However, the gripe I had was that test-free assemblies are _shown_ with a failure icon in the tree display, and that won't help our preferred process.

[Charlie Poole (charlie.poole)]

That's almost a separate bug in itself. It's a small problem here since there
is only one level above the assembly, but other NotRunnable tests can be
deeper in the tree. I'll see about adding better information.

Charlie

On Thu, Feb 3, 2011 at 1:50 AM, simon misys <email address hidden> wrote:
>> > (I also noted that the tree parent is also given the failure icon,
>> > though I don't know if you consider it unfortunate that its properties
>> > window does not give any indication that the cause is due to a
>> > NotRunnable child assembly.)
>
>> Yes, we migrate Non-Runnable up the tree because it's pretty severe
>> in some other cases - bad constructor for example. In fact, this may be
>> a different thing, calling for a different category. I'll consider that.
>
> Sorry if I wasn't clear, I meant that from the parent property you
> couldn't tell that the failure was due to a NotRunnable with no
> TestFixtures.  It's a minor nit - I mentioned it because you pointed out
> you could see in the properties of the test-free assembly that the
> status was NotRunnable with no TestFixtures, so you could understand why
> there was an issue with the assembly.  I was just pointing out that you
> can't see that sort of information (that the reason is NotRunnable with
> no TestFixtures) in the parent property, so looking at the parent
> properties alone would not lead you to understand why the parent icon
> indicates failure (the parent properties says success but no more).
>
> --
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
> https://bugs.launchpad.net/bugs/711330
>
> Title:
>  Test-free assembly given failure icon in tree display
>

[Charlie Poole (charlie.poole)]

So, the scenario we need to support is...

1) Legacy code (i.e. no tests)
2) Adding tests gradually in the same assembly

In this case, it seems that you would want some indication of the problem,
so that you know tests need to be written, but that it should have a different
appearance, which tells you immediately what the problem is.

This seems to point toward a different symbol in the tree and listing the
assembly itself in the not-run tree.

Does that make sense?

Charlie

PS: The process you are using isn't the way I've generally gone about
introducing NUnit tests into an environment that has not had them
before. I won't go into it here, but if you'd like to drop me a note offline,
I'll be glad to pass on some ideas about this.

On Thu, Feb 3, 2011 at 1:55 AM, simon misys <email address hidden> wrote:
>> So are you saying that you might create an assembly with no tests, but
>> later add tests to it? I guess that means you sometimes put tests into
>> the same assembly as the code they are testing.
>
> Sort of.  We already have many assemblies (40) without tests.  There is
> one assembly with tests, but no one apart from the developer who added
> it knew about it because he didn't create a top-level NUnit project for
> all to see.  Only if you opened the assembly in NUnit (or looked at the
> specific code) would you be aware there were any tests.  (None of the
> other developers have made any use of NUnit before.)  But we plan to add
> tests bit by bit, and all developers will do this, as and when
> appropriate and convenient.  Basically, we are not doing anything like
> enough regression testing and we want to address that with NUnit.
>
> In the meantime, we want to create a top-level NUnit project that
> contains all (current) assemblies.  That way, I think all developers are
> likely to be reminded that there are tests; all developers are likely to
> be aware that there are assemblies that aught to have tests but don't;
> developers don't have to check whether they need to add the assembly to
> the NUnit project when they do add tests (they do need to add new
> assemblies of course, but that is rarer than adding new tests to an
> existing assembly, and I suppose it's also possible that a new assembly
> might have no tests as you suggest); automated test runs are easier.
>
> If we only added an assembly to the project as and when tests were added
> to it, I think we run the risk of developers not being encouraged to add
> tests; run the risk of forgetting to add the assembly to the NUnit
> project when tests are first added to an assembly.  We want to set up
> the process to make it as likely and easy as possible that things will
> be done right.
>
> Of course, test-free assemblies don't result in failure _status_ either
> in the progress bar or the client runner.  However, the gripe I had was
> that test-free assemblies are _shown_ with a failure icon in the tree
> display, and that won't help our preferred process.
>
> --
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
> https://bugs.launchpad.net/bugs/711330
>
> Title:
>  Test-free assembly given failure icon i...

[simon misys (simon-marshall)]

> This seems to point toward a different symbol in the tree and listing the
> assembly itself in the not-run tree.

> Does that make sense?

Yes, that would be fine. Thanks, Simon.

[Charlie Poole (charlie.poole)]

Looking back at some earlier discussions, I see we decided to simply let test suites without any members to fail or pass on their own merits. Even without tests, such a class might have some sort of setup that could run and result in a failure, so it's best to simply execute it.

This change was implemented with respect to TestFixture classes but not for test assemblies. For consistency, it's best that all types of test suites without child tests be treated the same.