Need ability to turn off combinatorial

[Wayne Brantley]

[Issue now tracked at https://github.com/nunit/nunit-framework/issues/15]

When you have a combinatorial attribute such as the following:

[Test, Combinatorial]
public void MyTest(
    [Values(1,2,3)] int x,
    [Values("A","B")] string s)
{
    ...
}

Sometimes there is an issue with a certain set of values and you need to debug it. Would be nice to just turn off the combinatorial feature (by not having the attribute!), so that this could work for debugging:

[Test, TestCase(3,"A")]
public void MyTest(
    [Values(1,2,3)] int x,
    [Values("A","B")] string s)
{
    ...
}

As it is now, you have to completely redo the signature (and put it back after fixing bug/test) or setup a complex break point, etc....

[Test, TestCase(3,"A")]
public void MyTest(
    int x,
    string s)
{
    ...
}

[Charlie Poole (charlie.poole)]

Requiring the presence of the Combinatorial attribute in order to generate cases from the attributes on the parameters would be a significant breaking change since the missing cases would simply disappear from the run, without any failure. I'm re-assigning this bug to NUnit 3.0 with the idea that we should figure out some way to make it easy to resolve the problem, possibly by requiring the attribute, possibly in some other way.

Note that in the Gui, you can simply right-click on the test you want to execute.

[Wayne Brantley (wayne-brantley)]

Ok, or even an attribute that would just turn it off for now...combinatorial is a great feature...if you do not have to debug it!

[Charlie Poole (charlie.poole)]

Don't you use the Gui?

On Tue, Jul 10, 2012 at 9:30 AM, Wayne Brantley
<email address hidden> wrote:
> Ok, or even an attribute that would just turn it off for
> now...combinatorial is a great feature...if you do not have to debug it!
>
> --
> You received this bug notification because you are subscribed to NUnit
> Framework.
> https://bugs.launchpad.net/bugs/1022810
>
> Title:
> Need ability to turn off combinatorial
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nunit-3.0/+bug/1022810/+subscriptions

[Wayne Brantley (wayne-brantley)]

Don't use the GUI for debugging.....directly from the IDE in Visual studio
we can just right click the code in a test and debug or run it...or click
the entire fixture and run it...so never need/use gui...

On Tue, Jul 10, 2012 at 12:49 PM, Charlie Poole <email address hidden> wrote:

> Don't you use the Gui?
>
> On Tue, Jul 10, 2012 at 9:30 AM, Wayne Brantley
> <email address hidden> wrote:
> > Ok, or even an attribute that would just turn it off for
> > now...combinatorial is a great feature...if you do not have to debug it!
> >
> > --
> > You received this bug notification because you are subscribed to NUnit
> > Framework.
> > https://bugs.launchpad.net/bugs/1022810
> >
> > Title:
> > Need ability to turn off combinatorial
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/nunit-3.0/+bug/1022810/+subscriptions
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1022810
>
> Title:
> Need ability to turn off combinatorial
>
> Status in NUnit Test Framework:
> New
>
> Bug description:
> When you have a combinatorial attribute such as the following:
>
> [Test, Combinatorial]
> public void MyTest(
> [Values(1,2,3)] int x,
> [Values("A","B")] string s)
> {
> ...
> }
>
>
> Sometimes there is an issue with a certain set of values and you need to
> debug it. Would be nice to just turn off the combinatorial feature (by not
> having the attribute!), so that this could work for debugging:
>
>
> [Test, TestCase(3,"A")]
> public void MyTest(
> [Values(1,2,3)] int x,
> [Values("A","B")] string s)
> {
> ...
> }
>
>
> As it is now, you have to completely redo the signature (and put it back
> after fixing bug/test) or setup a complex break point, etc....
>
> [Test, TestCase(3,"A")]
> public void MyTest(
> int x,
> string s)
> {
> ...
> }
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nunit-3.0/+bug/1022810/+subscriptions
>

[Charlie Poole (charlie.poole)]

Ah! That explains it. I assume you're using TestDriven or something similar.

The reason this has not been a consideration before is that the NUnit Gui allows running individual testcases. I'll try to get some sort of fix for the problem into 3.0 then.

[Wayne Brantley (wayne-brantley)]

Yes....
I used to use testdriven....now I use CodeRush from devexpress has direct
support for unit tests much like testdriven. Also Re-sharper has support
for it....and VS2012 has this kind of support built right in (as you know -
since you built the provider for it)....

- Like I said, even a attribute to turn it off would be helpful...

Thanks!

On Wed, Jul 11, 2012 at 9:26 AM, Charlie Poole <email address hidden> wrote:

> Ah! That explains it. I assume you're using TestDriven or something
> similar.
>
> The reason this has not been a consideration before is that the NUnit
> Gui allows running individual testcases. I'll try to get some sort of
> fix for the problem into 3.0 then.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1022810
>
> Title:
> Need ability to turn off combinatorial
>
> Status in NUnit Test Framework:
> New
>
> Bug description:
> When you have a combinatorial attribute such as the following:
>
> [Test, Combinatorial]
> public void MyTest(
> [Values(1,2,3)] int x,
> [Values("A","B")] string s)
> {
> ...
> }
>
>
> Sometimes there is an issue with a certain set of values and you need to
> debug it. Would be nice to just turn off the combinatorial feature (by not
> having the attribute!), so that this could work for debugging:
>
>
> [Test, TestCase(3,"A")]
> public void MyTest(
> [Values(1,2,3)] int x,
> [Values("A","B")] string s)
> {
> ...
> }
>
>
> As it is now, you have to completely redo the signature (and put it back
> after fixing bug/test) or setup a complex break point, etc....
>
> [Test, TestCase(3,"A")]
> public void MyTest(
> int x,
> string s)
> {
> ...
> }
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nunit-3.0/+bug/1022810/+subscriptions
>

[Charlie Poole (charlie.poole)]

Another possible approach would be to introduce a setting that allowed
you to control the automatic use of Combinatorial when the attribute
is not present. I'll think about all these options for 3.0.

For the 2.6.1 release, it depends on whether I have to change a lot of
code (probably creating new bugs) or just a little. I am being
extremely conservative in changing 2.6 at this point, because I want
to have time to focus on 3.0.

Charlie

On Wed, Jul 11, 2012 at 7:01 AM, Wayne Brantley
<email address hidden> wrote:
> Yes....
> I used to use testdriven....now I use CodeRush from devexpress has direct
> support for unit tests much like testdriven. Also Re-sharper has support
> for it....and VS2012 has this kind of support built right in (as you know -
> since you built the provider for it)....
>
> - Like I said, even a attribute to turn it off would be helpful...
>
> Thanks!
>
> On Wed, Jul 11, 2012 at 9:26 AM, Charlie Poole <email address hidden>
> wrote:
>
>> Ah! That explains it. I assume you're using TestDriven or something
>> similar.
>>
>> The reason this has not been a consideration before is that the NUnit
>> Gui allows running individual testcases. I'll try to get some sort of
>> fix for the problem into 3.0 then.
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1022810
>>
>> Title:
>> Need ability to turn off combinatorial
>>
>> Status in NUnit Test Framework:
>> New
>>
>> Bug description:
>> When you have a combinatorial attribute such as the following:
>>
>> [Test, Combinatorial]
>> public void MyTest(
>> [Values(1,2,3)] int x,
>> [Values("A","B")] string s)
>> {
>> ...
>> }
>>
>>
>> Sometimes there is an issue with a certain set of values and you need to
>> debug it. Would be nice to just turn off the combinatorial feature (by not
>> having the attribute!), so that this could work for debugging:
>>
>>
>> [Test, TestCase(3,"A")]
>> public void MyTest(
>> [Values(1,2,3)] int x,
>> [Values("A","B")] string s)
>> {
>> ...
>> }
>>
>>
>> As it is now, you have to completely redo the signature (and put it back
>> after fixing bug/test) or setup a complex break point, etc....
>>
>> [Test, TestCase(3,"A")]
>> public void MyTest(
>> int x,
>> string s)
>> {
>> ...
>> }
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/nunit-3.0/+bug/1022810/+subscriptions
>>
>
> --
> You received this bug notification because you are subscribed to NUnit
> Framework.
> https://bugs.launchpad.net/bugs/1022810
>
> Title:
> Need ability to turn off combinatorial
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nunit-3.0/+bug/1022810/+subscriptions

[Charlie Poole (charlie.poole)]

This will stay on the list for 3.0 for now, but won't go into 2.6.1.

In general, I think this is a problem for the runner to resolve. All the nunit runners allow running an individual test case and 3rd party runners could easily supply the same capability. Nevertheless, we can keep this need in mind when designing the data attributes for 3.0.