Wrong failure message for parameterized tests that expect exceptions

[Adam Churvis]

This problem exists in both 2.5.10 and 2.6 Beta 2. This problem happens regardless of how it is run, though we have been using the GUI runner.

There are two classes, MyFixture and MySource. MyFixture looks like this:

[TestFixture]
public class MyFixture
{
 [TestCaseSource(typeof(MySource), "Cases")]
 public string BlahTest(string signifier)
 {
  switch (signifier)
  {
   case "A":
    return "BLAH";

   case "B":
    throw new Exception("HERE");

   case "C":
    return "C";

   default:
    return null;
  }
 }
}

MySource looks like this:

public class MySource
{
 public static IEnumerable<TestCaseData> Cases
 {
  get
  {
   yield return new TestCaseData("A").Returns("BLAH");
   yield return new TestCaseData("B").Throws("System.Exception");
   yield return new TestCaseData("C").Throws("System.Exception");
   yield return new TestCaseData("D").Throws("System.Exception");
  }
 }
}

We expect that the first two tests will succeed and that the second two will fail, and that is the case. However, the failure message on the third case is not the correct message. We are supposed to see "System.Exception was expected". However, we see 'Expected: null But was: "C"'. The reason is that in the NUnit source, the ParameterSet class that retrieves test parameters from TestCaseData instances has the following on line 277:

parms.Result = GetParm(source, PropertyNames.Result);

This line is executed for every test case regardless of whether an exception is expected. The setter for the Result property (lines 146-7) has the following:

result = value;
hasExpectedResult = true;

When this is consumed by the test runner, the runner thinks the test case has an expected result, when in fact it does not. To fix this, the following line should be inserted immediately before line 277:

if(parms.ExpectedExceptionName != null)

This will make sure the Result is only set if an exception is not expected, thus preserving the behavior the developer intended.

(Credit for discovering the solution to this problem goes to my son, David Churvis.)

[Charlie Poole (charlie.poole)]

The problem is as described, but the proposed fix causes further issues. TestCaseData needs a separate property to indicate it actually has a result, since null is a valid argument for Returns.

[Charlie Poole (charlie.poole)]

The resolution of this was to add HasExpectedException property to ITestCaseData. Additionally, the Result property was changed to ExpectedResult. The same changes are needed in nunit 3.0.