TestCaseSource does not use Arguments, Categories etc as described in documentation for 2.5.7

[Hamish Gunn]

What happens:

Created a class X that is returned as an IEnumerable<X> in method M of class Z.
This is used in [TestCaseSource(typeof(Z), "M"]

Class X has been given

  public string TestName { get; private set; }

  public IList<string> Categories { get; private set; }

  public bool Ignored;

In the constructor for X, these have been initialised
   this.Categories = new List<string> { "foo" };
   this.TestName = note;
   this.Ignored = true;

The test is executed(contrary to Ignored==true).
There are no categories in the NUinit GUI.
The test is not renamed.

What I expect:
The test is flagged as ignore.
There is a category "foo".
The test is named after the string value in TestName.

NUnit 2.5.7.
Nunit.exe (GUI)

[Charlie Poole (charlie.poole)]

I'm not sure why you are expecting this to work.
What's the signature of the test method to which
you apply this attribute?

Charlie

On Thu, Sep 9, 2010 at 11:59 AM, Hamish Gunn <email address hidden> wrote:
> Public bug reported:
>
> What happens:
>
> Created a class X that is returned as an IEnumerable<X> in method M of class Z.
> This is used in [TestCaseSource(typeof(Z), "M"]
>
> Class X has been given
>
>                public string TestName { get; private set; }
>
>                public IList<string> Categories { get; private set; }
>
>                public bool Ignored;
>
> In the constructor for X,  these have been initialised
>                        this.Categories = new List<string> { "foo" };
>                        this.TestName = note;
>                        this.Ignored = true;
>
> The test is executed(contrary to Ignored==true).
> There are no categories in the NUinit GUI.
> The test is not renamed.
>
> What I expect:
> The test is flagged as ignore.
> There is a category "foo".
> The test is named after the string value in TestName.
>
> NUnit 2.5.7.
> Nunit.exe (GUI)
>
> ** Affects: nunitv2
>     Importance: Undecided
>         Status: New
>
> --
> TestCaseSource does not use Arguments, Categories etc as described in documentation for 2.5.7
> https://bugs.launchpad.net/bugs/633884
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
>

[Hamish Gunn (hamish-gunn)]

It is a logical deduction from the description at http://nunit.org/index.php?p=testCaseSource&r=2.5.7

From there I read:
"If sourceType is not specified, the class containing the test method is used. NUnit will construct it using either the default constructor or - if arguments are provided - the appropriate constructor for those arguments.

The sourceName argument represents the name of the source used to provide test cases. It has the following characteristics:

...
    * It must return an IEnumerable or a type that implements IEnumerable.
    * The individual items returned by the enumerator must be compatible with the signature of the method on which the attribute appears. The rules for this are described in the next section. "

and

"Constructing Test Cases

In constructing tests, NUnit uses each item test case returned by the enumerator as follows:
...
If it is any other type of object, it is examined using reflection and any public fields or properties with the following names are used:
...
Categories
    An IList of categories to be applied to the test case.
...
TestName
    Provides a name for the test. If not specified, a name is generated based on the method name and the arguments provided
Ignored
    If true, the test case is ignored.

.... "

Here is the key reason I expect it to work - you say:
***" it is examined using reflection and any public fields or properties with the following names are used:"***.

"It" is "each item test case returned by the enumerator". In my case (see below for actual code), "it" is an instance of class SimTestDescriptor. You'll see it has public TestName amongst others. Since I expose that, I expect it to be used according to your explanation; ditto Categories and Ignored.

Here are the signatures (comments etc. excluded):

[TestCaseSource(typeof(SimTestDataGenerator), "TestArguments")]
public void Test1(SimTestDescriptor descriptor)

public class SimTestDataGenerator
{
  public SimTestDataGenerator()
...
  public IEnumerable<SimTestDescriptor> TestArguments()
...
}

public class SimTestDescriptor
{
        public SimTestDescriptor(string category, string note, string simFile, bool update, Regression.RunLocation runLocation, ThreadMode threadMode, ShockFileAction shockFileAction, string baseLineCulture, string versionUnderTestCulture)
...
        public string TestName { get; private set; }
        public IList<string> Categories { get; private set; }
        public string Note;
        public string SimFile;
        public bool Update;
        public Regression.RunLocation RunLocation;
        public ThreadMode ThreadMode;
        public ShockFileAction ShockFileAction;
        public string BaselineCulture;
        public string VersionUnderTestCulture;
        public bool Ignored;
}

[Charlie Poole (charlie.poole)]

Indeed! I see the problem. I'm afraid it's a documentation bug.

There should be an additional element in the list under "Constructing
Test Cases"

0. If the test has a single argument and the returned value matches the type of
that argument it is used directly.

This is a case where the author (me) thought that was "obvious" - but clearly
it is not.

If you want your test to process the contents of your own struct or class, then
use it as an argument and return that type. If you want NUnit to perform
processing described, then the actual arguments should be stored in an
Arguments member. In fact, in that case, you may as well use TestCaseData,
which contains all the functionality you are looking for and will also continue
to work under NUnit 3.0, where we no longer use reflection in this way.

Sorry for the confusion.

Charlie

On Fri, Sep 10, 2010 at 1:51 AM, Hamish Gunn <email address hidden> wrote:
> It is a logical deduction from the description at
> http://nunit.org/index.php?p=testCaseSource&r=2.5.7
>
> >From there I read:
> "If sourceType is not specified, the class containing the test method is used. NUnit will construct it using either the default constructor or - if arguments are provided - the appropriate constructor for those arguments.
>
> The sourceName argument represents the name of the source used to
> provide test cases. It has the following characteristics:
>
> ...
>    * It must return an IEnumerable or a type that implements IEnumerable.
>    * The individual items returned by the enumerator must be compatible with the signature of the method on which the attribute appears. The rules for this are described in the next section. "
>
> and
>
> "Constructing Test Cases
>
> In constructing tests, NUnit uses each item test case returned by the enumerator as follows:
> ...
> If it is any other type of object, it is examined using reflection and any public fields or properties with the following names are used:
> ...
> Categories
>    An IList of categories to be applied to the test case.
> ...
> TestName
>    Provides a name for the test. If not specified, a name is generated based on the method name and the arguments provided
> Ignored
>    If true, the test case is ignored.
>
> .... "
>
> Here is the key reason I expect it to work - you say:
> ***" it is examined using reflection and any public fields or properties with the following names are used:"***.
>
> "It" is "each item test case returned by the enumerator".  In my case
> (see below for actual code), "it" is an instance of class
> SimTestDescriptor.  You'll see it has public TestName amongst others.
> Since I expose that, I expect it to be used according to your
> explanation; ditto Categories and Ignored.
>
> Here are the signatures (comments etc. excluded):
>
> [TestCaseSource(typeof(SimTestDataGenerator), "TestArguments")]
> public void Test1(SimTestDescriptor descriptor)
>
> public class SimTestDataGenerator
> {
>                public SimTestDataGenerator()
> ...
>                public IEnumerable<SimTestDescriptor> TestArguments()
> ...
> }
>
>
> public class SimTestDescriptor
> {
>        public SimTestDescriptor(string category, string note, string simF...

[Hamish Gunn (hamish-gunn)]

Thanks. I'm afraid I don't understand your recommendation.

This is the first bit I don't follow:
"If you want your test to process the contents of your own struct or class, then use it as an argument and return that type."

Do you mean
  [TestCaseSource(typeof(SimTestDataGenerator), "TestArguments")]
  public void IndividualTestCasesShouldBeGenerated(Type simTestDescriptorClass) ?

What do I return "that type" from?

Sorry but I'm feeling pretty foolish here. Can you give me an example?

I don't follow this either:
 "If you want NUnit to perform processing described, then the actual arguments should be stored in an Arguments member."
A member of which class?

You also say
"In fact, in that case, you may as well use TestCaseData, which contains all the functionality you are looking for and will also continue to work under NUnit 3.0, where we no longer use reflection in this way."

I don't want to write code that will be deprecated in the next version of NUnit so does that mean TestCaseData will be the only way forward?

I don't think I'm stupid but I'm coming at this for the first time and I suspect the gulf between your familiarity and mine is the problem.

Thanks for your time.

[Hamish Gunn (hamish-gunn)]

I've got something working using TestCaseData. It does seem to cut out a step. However it means a long signature for the test.

  [TestCaseSource(typeof(SimTestDataGenerator), "TestArguments")]
  public void IndividualTestCasesShouldBeGenerated(bool ignored, IList<string> categories, string note,
   string simFile, bool update, RunLocation runLocation, ThreadMode threadMode, ShockFileAction shockFileAction,
   string baselineCulture, string versionUnderTestCulture)

I think I may see your point now if I use TestCaseData to return an instance of a class that describes the data. Perhaps something along the lines of:

  [TestCaseSource(typeof(SimTestDataGenerator), "TestArguments")]
  public void IndividualTestCasesShouldBeGenerated(SimTestDescriptor descriptor)
  {
...
and

  public IEnumerable TestArguments()
  {
   foreach (SimTestFileDescription line in this.Descriptions)
   {
    if (!line.Ignore)
    {
     yield return new TestCaseData(new SimTestDescriptor(line.Category, line.Note, line.SimFile, line.Update,
                                     GetEnum<RunLocation>("RunLocation", line.RunLocation),
         GetEnum<ThreadMode>("ThreadMode", line.ThreadMode),
         GetEnum<ShockFileAction>("ShockFileAction", line.ShockFileAction),
         line.BaselineCulture, line.VersionUnderTestCulture))
                .SetCategory(line.Category)
                .SetName(line.Note)
                .SetDescription(line.Note);
    }
   }
  }

[Charlie Poole (charlie.poole)]

Hi Hamish,

I think I'll start again. :-)

Let's say I have a test that I want to run multiple times with
differing arguments. I decide
to use TestCaseSourceAttribute to specify them. The source may be a
field, a method
or a property and should provide an IEnumerable of some type. There
are three choices
with regard to that type:

1. It may be an object array that gives the arguments needed by the
test method, e.g.:

   object[] TestArguments = new object[] {
            new object[] { 5, "five" },
            new object[] { 42, "answer" }
   };

In this case the outer object[] is providing the IEnumerable and each
inner object[]
represents a test case. The method signature would have to be M(int, string);

This approach does not allow you to do things like ignoring an individual case.

2. It may be the exact type required by the method, but only when the method
takes just one argument, e.g.:

   int[] TestArguments = new int[] { 5, 42 };

The method signature for this case would be M(int);

This approach does not allow ignoring a test case, etc.

As a special case of this approach, you could pass in a custom struct
or class, e.g.:

   object[] TestArguments = new object[] {
            new MyStruct( 5, "five"),
            new MyStruct( 42, "answer")
   };

This last is what you have done and it does not allow ignoring tests
or any of the
other special things suggested in the docs.

3. You may use TestCaseData or any type of your own which has some of the
same properties which are supported by TestCaseData. One of those properties
is Arguments and the items provided there must match the method parameters.
So, for example...

   TestCaseData[] TestArguments = new TestCaseData[] {
      new TestCaseData(5, "five").Ignore("Still under development),
      new TestCaseData(42, "answer")
   };

This is the only approach that allows use of Ignore, SetCategory, etc.

So...

The obvious question is "If I define a custom struct or class, how does
NUnit decide whether I'm using approach #2 or #3?"

I had to look at the code to figure it out. It checks for the presence of the
interface NUnit.Framework.ITestCaseData. Note that the interface is not
actually used by NUnit. In order to support multiple framework versions,
reflection is used. But the interface helps identify the class and helps
you to make sure you are implementing the correct properties.

In fact, the entire approach of defining your own replacement for TestCaseData
is probably not worth supporting any longer. The only reason I can see for
defining your own replacement is if you want to use a different fluent syntax
from that supported by TestCaseData.

However, so long as you use the ITestCaseData interface, your code will be
pretty compatible with version 3.0 - the interface could change, but
will likely
only add new properties.

I hope this clears things up. The entire area is a bit too complex for my taste,
so the basic docs will probably just say to use TestCaseData in the future,
and the stuff about implementing your own class will only be discussed
under the heading of extending NUnit.

Charlie

> Thanks.  I'm afraid I don't understand your recommendation.
>
> This is the first bit I...

[Hamish Gunn (hamish-gunn)]

Thanks Charlie. I'm running (almost) happily with TestCaseData now.

I say "almost" because I've a use case that doesn't seem to be covered. I want to be able to pass data _in_ to the test case data generator. E.g. I'm generating data from a file. I don't want to build the file name into the data generator.

I can see a possible solution - call the [TextFixtureSetUp] way up front, before any generation of dynamic tests. At the moment this gets called _after_ the tests are generated - too late! (I've also tried [SetUpFixture]/[SetUp] but they seem to get called after the test generation too.)

Calling it first is not ideal but it allows the name of the file I want to be used to be used by the generator class. I could do it from a global class's static property (yuch!).

e.g. Let the generator "know" about the TestFixture class but it could be any global class.

[TestFixture]
public class TestHarness ....

   static string Filename { get; set; } // This holds the name of the file to be used in generating test data

   [TextFixtureSetUp]
   public void SetUp()
   {
      Filename = ... // doesn't work at the moment because the order of execution means this is called too late.
   }

   [TestCaseData("Generator", "GetData"]
   public void ...
...
}

and in the Generator class:

public class Generator
{

   public Generator()
   {
      Filename = TestHarness.Filename; // now it can be used in this class
      ...
   }

   public IEnumerable<..> GetData...
   {

---------------------------------------------------------------

Thanks again - dynamic tests are really important.

[Charlie Poole (charlie.poole)]

Yes... NUnit currently generates all tests at the time the assembly is loaded,
which is of course before any of them are run. I sometimes call these "static"
tests - although I'm not sure that's the best name.

We also expect to have "dynamic" tests in NUnit 3.0, generated immediately
before the tests are executed. Some existing attributes will be redefined to
produce tests dynamically and new attributes will be added.

This is actually a much more extensive change than you might imagine,
which is why it isn't happening in 2.x.

Charlie

On Tue, Sep 14, 2010 at 8:08 AM, Hamish Gunn <email address hidden> wrote:
> Thanks Charlie.  I'm running (almost) happily with TestCaseData now.
>
> I say "almost" because I've a use case that doesn't seem to be covered.
> I want to be able to pass data _in_ to the test case data generator.
> E.g. I'm generating data from a file.  I don't want to build the file
> name into the data generator.
>
> I can see a possible solution - call the [TextFixtureSetUp] way up
> front, before any generation of dynamic tests.  At the moment this gets
> called _after_ the tests are generated - too late! (I've also tried
> [SetUpFixture]/[SetUp] but they seem to get called after the test
> generation too.)
>
> Calling it first is not ideal but it allows the name of the file I want
> to be  used to be used by the generator class.  I could do it from a
> global class's static property (yuch!).
>
> e.g. Let the generator "know" about the TestFixture class but it could
> be any global class.
>
> [TestFixture]
> public class TestHarness ....
>
>   static string Filename { get; set; } // This holds the name of the
> file to be used in generating test data
>
>   [TextFixtureSetUp]
>   public void SetUp()
>   {
>      Filename = ... // doesn't work at the moment because the order of execution means this is called too late.
>   }
>
>   [TestCaseData("Generator", "GetData"]
>   public void ...
> ...
> }
>
> and in the Generator class:
>
> public class Generator
> {
>
>   public Generator()
>   {
>      Filename = TestHarness.Filename;  // now it can be used in this class
>      ...
>   }
>
>   public IEnumerable<..> GetData...
>   {
>
> ---------------------------------------------------------------
>
> Thanks again - dynamic tests are really important.
>
> --
> TestCaseSource does not use Arguments, Categories etc as described in documentation for 2.5.7
> https://bugs.launchpad.net/bugs/633884
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
>

[Charlie Poole (charlie.poole)]

As it turns out, both the documentation and my answer above wasn't correct. See the updated
entry on the web site.