UI won't run tests marked Explicit from the context menu

[David Mason]

NUnit 2.5.10, Gui runner:

A test marked [Explicit] can only be run from the Gui runner context menu if it is the first test run. After that, it is disabled.

[David Mason (david.mason)]

The test method with the ExplicitAttribute is defined in an abstract class, and executed on a concrete subclass.

[Charlie Poole (charlie.poole)]

I'm unable to reproduce this error. I've tested using methods marked
explicit in a
concrete class as well as using a method marked explicit in the abstract base.
If you have a simple example that demonstrates the error, please post
it here and
we can re-open this bug if necessary.

Charlie

On Mon, Sep 5, 2011 at 8:10 AM, David Mason <email address hidden> wrote:
> The test method with the ExplicitAttribute is defined in an abstract
> class, and executed on a concrete subclass.
>
> --
> You received this bug notification because you are the registrant for
> NUnit V2.
> https://bugs.launchpad.net/bugs/841894
>
> Title:
>  UI won't run tests marked Explicit
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nunitv2/+bug/841894/+subscriptions
>

[David Mason (david.mason)]

After receiving notification that you were unable to reproduce this error, I went back to the solution and noticed the following:

The abstract class and concrete class for which there is the problem are in different projects/assemblies.

When I run the test on a subclass of the abstract class that is in the same project/assembly as the abstract class, the problem does not occur.

[Charlie Poole (charlie.poole)]

It's strange that it should behave differently. Does this only occur upon reloading the assemblies or does it happen if you simply rerun without reloading? (Check your settings for automatic reload)

[David Mason (david.mason)]

All of the following are enabled:

Assembly Isolation:
  Test run in NUnit process
  Use seperate AppDomain per Assembly

Assembly Reload:
  Reload before each test run
  Release when test assembly changes

If I right click the assembly and "load fixture", it lets me run the test once more, and then disables it.

[Charlie Poole (charlie.poole)]

Generally, odd things are known to happen when both
  Reload before each test run
  Reload when test assembly changes
are set.

I can't think of any reason to use both of them, since it
will cause an extra reload. However, they are separately
selectable - possibly an error in the design.

Can you try turning off both of them and then try each
separately? You can simulate an automatic reload by
using reload tests.

Note that the menu option reload project is a complete
reload, while reload tests tries to keep around old info
about tests that passed and failed. Reload fixture is
also a complete reload, but limits the load to only one
branch of the tree.

Charlie

On Mon, Sep 5, 2011 at 10:18 AM, David Mason <email address hidden> wrote:
> All of the following are enabled:
>
> Assembly Isolation:
>  Test run in NUnit process
>  Use seperate AppDomain per Assembly
>
> Assembly Reload:
>  Reload before each test run
>  Release when test assembly changes
>
> If I right click the assembly and "load fixture", it lets me run the
> test once more, and then disables it.
>
> --
> You received this bug notification because you are the registrant for
> NUnit V2.
> https://bugs.launchpad.net/bugs/841894
>
> Title:
>  UI won't run tests marked Explicit
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nunitv2/+bug/841894/+subscriptions
>

[David Mason (david.mason)]

_ = nothing, R = before each run, C = change

__: problem doesn't occur (but test doesn't function on second run)
R_: problem occurs (but the test works)
_C: problem doesn't occur (but test doesn't on second run)
RC: problem occurs

[Charlie Poole (charlie.poole)]

OK, I'll set up a test case and see if I can figure it out.

Charlie

On Mon, Sep 5, 2011 at 1:21 PM, David Mason <email address hidden> wrote:
> _ = nothing, R = before each run, C = change
>
> __: problem doesn't occur (but test doesn't function on second run)
> R_: problem occurs (but the test works)
> _C: problem doesn't occur (but test doesn't on second run)
> RC: problem occurs
>
> --
> You received this bug notification because you are the registrant for
> NUnit V2.
> https://bugs.launchpad.net/bugs/841894
>
> Title:
>  UI won't run tests marked Explicit
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nunitv2/+bug/841894/+subscriptions
>

[Charlie Poole (charlie.poole)]

I'm still unable to duplicate this problem. Here's the code I'm using...

Assembly1:

namespace Bug841894
{
    public abstract class AbstractTestClass
    {
        [Test]
        public void SomeTest() { }

        [Test, Explicit]
        public void ExplicitTest() { }
    }
}

Assembly2:

namespace Bug841894
{
    public class DerivedTestClass : AbstractTestClass
    {
        [Test]
        public void AnotherTest() { }
    }
}

If you can modify this and make it reproduce the problem you encountered, I'll try to debug it further.

[David Mason (david.mason)]

Assembly1:
namespace Bug841894
{
    [TestFixture]
    public abstract class AbstractTestClass
    {
        [Test]
        public void SomeTest() { }
        [Test, Explicit]
        public void ExplicitTest() { }
    }
}
Assembly2:
namespace Bug841894
{
    public class DerivedTestClass : AbstractTestClass
    {
        [Test]
        public void AnotherTest() { }
    }
}

------
Just a matter of putting [TestFixture] on the abstract class.

I have now confirmed this problem occurs in 2.6 beta. This is being done as a standard user on a locked down corporate laptop. I'll try again later on a more workable system.

[David Mason (david.mason)]

I find this actually happens for all tests marked [Explicit], not just in this non-trivial arrangement. It just so happens that the only explicit tests I had were in this arrangement.

I have downloaded the code for 2.6 and debugged this myself. I am not finished, and I can get by (as a user) by removing the call to TestSuiteTreeNode.Include in the code that sets the Enabled property on the Run item in the context menu.

This Include property is set to false by the EmptyFilter that is applied to the tree when the tests are run, and never restored afterwards.

Later, I will investigate further to determine what needs to be done to fix this, unless someone else realizes the problem from what I've typed here and beats me to it.

[Charlie Poole (charlie.poole)]

test comment: I'm posting this on the website rather than by mail to see what the latency is.

Charlie

[Charlie Poole (charlie.poole)]

It's been 22 hours since I posted to this bug via email and it's still not appearing, so I'm re-posting via the internet here:

1) I'm still not able to duplicate this problem under any circumstances. One possibility is that we are using the Gui in slightly different ways. Please describe step by step what you are doing to select and run the explicit test case.

2) In particular, does it only happen when you use the context menu and select 'Run'? What happens if you select the test case and then click the main 'Run' button instead? What happens if you double-click the test case?

3) I can review the code involved in setting the Include property, but seeing the bug first would make me more confident of a fix.

Charlie

[David Mason (david.mason)]

1) Step by step follows
2) It happens on the context menu but not the run button. Double-clicking does not work either.
3) Hopefully the step-by-step will reproduce the result for you.

Step-by -step
---------------

NUnit GUI config:
  Assembly Isolation: Run tests directly in NUnit process, Use seperate AppDomain per assembly
  Assembly Reload: Reload before each test run, Reload when test assembly changes
  Runtime selection: Select default...
  Advanced: Shadow copy enabled.

Create a new Visual Studio C# library project.
Add a reference to the nunit.framework assembly (I'm using 2.6.0.11240)
Add a single testfixture class:

using System;
using System.Diagnostics;
using NUnit.Framework;

namespace Test1
{
    [TestFixture]
    public class TestFixtureClass
    {
        [Test]
        [Explicit]
        public void TestExplicit()
        {
            Trace.WriteLine("TestExplicit "+DateTime.Now);
            Assert.Fail();
        }

        [Test]
        [Ignore]
        public void TestIgnore()
        {
            Trace.WriteLine("TestIgnore " + DateTime.Now);
            Assert.Fail();
        }

        [Test]
        public void TestTest()
        {
            Trace.WriteLine("TestTest " + DateTime.Now);
        }
    }
}

Build this project, then start the NUnit GUI, create a project for it and add the output of this C# project to the NUnit project. Save the project and exit the NUnit GUI.

Restart the NUnit GUI. The project you've just created should be reloaded automatically.

At this point look at the treeview. All the test names are black, and all the bubbles are gray. Right-click each of the three tests and look at the availability of the Run option. They're all enabled.

Right-click TestExplicit and run it. It works.
Select another test. TestExplicit is now blue.
Right click TestExplicit. Run is greyed out.

Right click the assembly name in the tree and load the fixture. Original situation restored.

Run TestTest. It runs. Select it again to move the highlight. TestExplicit is now blue, and its Run option is disabled.

Also note in all these cases that the Run option is never disabled for TestIgnore.

[Charlie Poole (charlie.poole)]

I have now observed the bug and will work on it!

A few minor observations...

1) Your use of Load Fixture for the side effect of reloading the assembly is a bit odd but does not contribute to the problem. In fact, since there is only one assembly, it has no other effect than if you used the Reload Project menu item.

2) Run is enabled by design for ignored fixtures, because doing anything else could lead the user to wonder why Run is not available. However, in the next beta, tests with the Ignore attribute will show up as yellow and it may make sense to disable run for them in that case.

I think the best solution may be to eliminate marking of Explicit tests by changing the font color. The blue color was originally intended to show tests filtered out by use of categories. I'll have to research how and why we made this change.

Charlie