SimpleTestRunner miss handles point of exception

[Lucas Caballero]

[This bug is now tracked at https://github.com/nunit/nunit-framework/issues/4]

The SimpleTestRunner performs a an odd skip of Exception processing when faced with a fully un-implemented EventListener. What happens is that my RunStarted throws an Exception and this gets caught end then RunFinished is called which then throws an exception -- the stack trace makes it look like the RunFinished was the culprit when in fact the RunStart began the exceptional behavior. This occurs mainly because the catch( Exception exception ) catches everything. Perhaps what it should do is catch NUnit exceptions and throw all other exceptions -- I'm not familiar enough with the architecture around Exception testing, Assert Exceptions, etc.

So here is the culprit code in SimpleTestRunner, Line 141, of release 2.5.5.101112.

public virtual TestResult Run( EventListener listener, ITestFilter filter )
{
 try
 {
        log.Debug("Starting test run");

  // Take note of the fact that we are running
  this.runThread = Thread.CurrentThread;

  listener.RunStarted( this.Test.TestName.FullName, test.CountTestCases( filter ) );

  testResult = test.Run( listener, filter );

  // Signal that we are done
  listener.RunFinished( testResult );
        log.Debug("Test run complete");

  // Return result array
  return testResult;
 }
 catch( Exception exception )
 {
                // RunStart actually threw the exception. so RunFinish doesn't make sense.

                // RunFinish then throws an exception when really the first exception should be handled first.
  // Signal that we finished with an exception
  listener.RunFinished( exception );
  // Rethrow - should we do this?
  throw;
 }
 finally
 {
  runThread = null;
 }
}

Cheers,
L-

[Charlie Poole (charlie.poole)]

The problem is subtle. If one of the RunStart event handlers throws an exception, some handlers may have already successully processed the event while others will never receive it at all. It's not possible to discriminate among the handlers without taking over the event routing from .NET and we don't want to do that.

Safest thing seems to be that we should not send RunFinished to any handlers unless all RunStarted events processed successfully.

[Charlie Poole (charlie.poole)]

OK, I tried suppressing RunFinished and now I see why I did it the way I did.

If no RunFinished is issued, then the GUI never realiizes that the run is finished and
NUnit just appears to hang.

Back to the drawing board!

[Charlie Poole (charlie.poole)]

Postponing this to another release. It's a true can of worms.

Essentially, we should probably have some sort of AddinException that wraps exceptions like this and treat them specially. We may simply have to remove addins once they throw an exception.

[Charlie Poole (charlie.poole)]

Moving entirely to the 3.0 release

[Charlie Poole (charlie.poole)]

I removed this from the 2.9.6 milestone since it's not clear whether custom EventListeners will be used in NUnit 3.0. We may end up providing this functionality without use of addins, in which case the problem will be resolved in a different way.