Need a way to generate data when test is executed

[Garth Kidd]

[Bug now tracked at https://github.com/nunit/nunit-framework/issues/3]

NUnit 2.5.2 and 2.5.3 run every method targeted by TestCaseSource, even if the test targeting it is Explicit or the test fixture is Explicit.

The behaviour I expect is: NUnit shouldn't run a test's TestCaseSource if the test itself won't be run.

Detail:

I have an expensive TestCaseSource generating a few hundred thousand permutations and combinations. I've applied Explicit to both the test targeting the source and the fixture surrounding both test and source.

NUnit console spends ten minutes needlessly constructing test cases from the TestCaseSource before ignoring them and proceeding with the non-Explicit tests.

If I throw NotImplementedException from the first line of the expensive TestCaseSource. NUnit runs the non-Explicit tests immediately. If I put a MessageBox.Show call in the first line, I see the message box.

[Cliff Burger (cliff-burger)]

We use TestCaseSource translate QUnit javascript tests into Nunit results (reading the HTML page loaded via Watin). The consequence of the NUnit 2-0 behavior is that all QUnit tests run as soon as you load the NUnit framework. There are other scenarios where this would be useful if I leverage categories to run on specific data driven tests.

[Charlie Poole (charlie.poole)]

If I understand correctly, you use a call to a method listed as the source
of test data. That method actually runs the test, rather than providing the
data for the test. Consequently, the test itself runs when the method
is called at the time of loading the NUnit tests.

Have I understood correctly? I want to be sure before replying further.

Charlie

On Fri, Jan 21, 2011 at 12:27 PM, Cliff Burger
<email address hidden> wrote:
> We use TestCaseSource translate QUnit javascript tests into Nunit
> results (reading the HTML page loaded via Watin). The consequence of the
> NUnit 2-0 behavior is that all QUnit tests run as soon as you load the
> NUnit framework. There are other scenarios where this would be useful if
> I leverage categories to run on specific data driven tests.
>
> --
> You received this bug notification because you are subscribed to NUnit
> Framework.
> https://bugs.launchpad.net/bugs/538070
>
> Title:
>  Need a way to generate data when test is executed
>

[Stein Jakob Nordbø (steinjak)]

Hi, hope you don't mind that I jump in on this one,

We're also running into the same problem; our current work-around is to run all QUnit tests within a single NUnit test. The advantage of running QUnit tests in a source is to be able to translate 1:1 to NUnit tests so that our build system reports QUnit test failures individually rather than in summary, and that our test counts include both NUnit and QUnit tests. Naturally, running browser-based tests is quite time-consuming in the normal flow of development, and since we don't want our back-end developers to be waiting for the full suite of javascript tests when working in another area of the system, we prefer to defer running of these until the build on the build server (hence Explicit+Category); UI devs run the JS tests more regularily, but manually.

Stein Jakob

[Cliff Burger (cliff-burger)]

Usually, our developers will run the QUnit tests directly through the
browser. This applies to our Continuous Integration build

The test case source does the following:
- Using Watin, navigate to the Url with the QUnit tests.
- Execute each test and parse the results in to a data structure
- There is a single test in this suite that looks like this:

[Test, TestCaseSource("RunTests")]
        public void Run(QUnitTestResult result)
        {
            if (!result.IsSuccess)
            {
                // ReSharper runner clears the output window between each
call to Run()
                _allFailedTestsOutput.AppendLine(result.Description);
            }
            Assert.IsTrue(result.IsSuccess, result.Description);
        }

If TestCaseSource executed only when it required (like the "delayed
evaluation" in LINQ), it would open up many other possible uses.

2011/1/21 Stein Jakob Nordbø <email address hidden>

> Hi, hope you don't mind that I jump in on this one,
>
> We're also running into the same problem; our current work-around is to
> run all QUnit tests within a single NUnit test. The advantage of running
> QUnit tests in a source is to be able to translate 1:1 to NUnit tests so
> that our build system reports QUnit test failures individually rather
> than in summary, and that our test counts include both NUnit and QUnit
> tests. Naturally, running browser-based tests is quite time-consuming in
> the normal flow of development, and since we don't want our back-end
> developers to be waiting for the full suite of javascript tests when
> working in another area of the system, we prefer to defer running of
> these until the build on the build server (hence Explicit+Category); UI
> devs run the JS tests more regularily, but manually.
>
>
> Stein Jakob
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/538070
>
> Title:
> Need a way to generate data when test is executed
>
> Status in NUnit Test Framework:
> Confirmed
>
> Bug description:
> NUnit 2.5.2 and 2.5.3 run every method targeted by TestCaseSource,
> even if the test targeting it is Explicit or the test fixture is
> Explicit.
>
> The behaviour I expect is: NUnit shouldn't run a test's TestCaseSource
> if the test itself won't be run.
>
> Detail:
>
> I have an expensive TestCaseSource generating a few hundred thousand
> permutations and combinations. I've applied Explicit to both the test
> targeting the source and the fixture surrounding both test and source.
>
> NUnit console spends ten minutes needlessly constructing test cases
> from the TestCaseSource before ignoring them and proceeding with the
> non-Explicit tests.
>
> If I throw NotImplementedException from the first line of the
> expensive TestCaseSource. NUnit runs the non-Explicit tests
> immediately. If I put a MessageBox.Show call in the first line, I see
> the message box.
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/nunit-3.0/+bug/538070/+subscribe
>

[Charlie Poole (charlie.poole)]

OK, I understand the use case now and will keep it in mind when this is implemented.

It's possible that we may be able to avoid the odd use of TestCaseSource and allow you to put your
iterator directly in the test method, which would then report multiple results just as it now dows.

Charlie