Feature request: attrbiute for controlling of test order

[Robert Sosnowski]

[This issue is now tracked at https://github.com/nunit/nunit-framework/issues/51]

I'm aware that in the past similar request were rejected (for example ID: 878856). However I'll try once again giving some rationale for this.
Here: http://www.nunit.org/wiki/doku.php?id=dev:vision it is clearly stated: "support use of NUnit for integration and acceptance testing, but only to the extent that this support didn't interfere with our primary goal"
Even if this would not be written all we know that NUnit is not only used for unit testing but for all kinds of automatic testing, for example running web pages (Selenium), WinForms (project White), and many other purposes.
In general creating full setup in integration test scenario is costly. Other reason: http://stackoverflow.com/questions/1078658/nunit-test-run-order/2889524#2889524. Also good justification here: http://aspadvice.com/blogs/ssmith/archive/2005/04/12/1858.aspx
In my particular test I have an integration with database: test if account creates, then if address for this account creates, then test if payment for this account suceeds, then ... (very long user case chain). While it is possible to create independent tests it is difficult to do, causes code repetition and slows down testing.
Adding additional ordering attribute should not break primary goal: pure unit testing can be still possible when not using this special attribute.
It seems for me that such an attribute would not be costly in implementation effort. Especially in comparison with other sophisticated NUnit functionality.

[Charlie Poole (charlie.poole)]

Hi Robert,

Actually, we have not rejected the idea of ordering. Rather - as shown
by the new vision statement you quote -
we are embracing it for the new NUnit. For that reason, I'm moving
your report to that project.

It may not be quite as simple as adding a single attribute, since we
don't want to use the existing unit test-focused
TestFixture to represent ordered sets of tests. However, we can
discuss details of how it will be implemented on the
nunit-discuss list.

(BTW, I'm not able to find a bug with ID 878856)

Charlie

On Tue, Mar 22, 2011 at 3:22 PM, Robert Sosnowski
<email address hidden> wrote:
> Public bug reported:
>
> I'm aware that in the past similar request were rejected (for example ID: 878856). However I'll try once again giving some rationale for this.
> Here: http://www.nunit.org/wiki/doku.php?id=dev:vision it is clearly stated: "support use of NUnit for integration and acceptance testing, but only to the extent that this support didn't interfere with our primary goal"
> Even if this would not be written all we know that NUnit is not only used for unit testing but for all kinds of automatic testing, for example running web pages (Selenium), WinForms (project White), and many other purposes.
> In general creating full setup in integration test scenario is costly. Other reason: http://stackoverflow.com/questions/1078658/nunit-test-run-order/2889524#2889524. Also good justification here: http://aspadvice.com/blogs/ssmith/archive/2005/04/12/1858.aspx
> In my particular test I have an integration with database: test if account creates, then if address for this account creates, then test if payment for this account suceeds, then ... (very long user case chain). While it is possible to create independent tests it is difficult to do, causes code repetition and slows down testing.
> Adding additional ordering attribute should not break primary goal: pure unit testing can be still possible when not using this special attribute.
> It seems for me that such an attribute would not be costly in implementation effort. Especially in comparison with other sophisticated NUnit functionality.
>
> ** Affects: nunitv2
>     Importance: Undecided
>         Status: New
>
> --
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
> https://bugs.launchpad.net/bugs/740539
>
> Title:
>  Feature request: attrbiute for controlling of test order
>

[Robert Sosnowski (robsosno)]

First I've reviewed existing feature requests. Link here: http://www.nunit.org/index.php?p=contactUs shows http://sourceforge.net/tracker/?atid=360749&group_id=10749&func=browse as appropriate for this.
Under this last link I've found similar rejected requests for attributes showing order or dependency between tests: 878856, 1800615, 685680.
However I've just found in the documentation that order of executing tests is documented and supported (alphabetical order). After upgrading NUnit I've got many failing tests because of incorrect order of tests.
While this is not as good as it could be with attributes I see that database integration tests can be done using the alphabetical order.
With this workaround I will wait for attributes.

[Charlie Poole (charlie.poole)]

The requests you are looking at are for older versions of NUnit, where
they were not appropriate. All feature requests are now tracked using
the Launchpad bug tracker. I'll correct the invalid link on the
contact page.

Relying on alphabetical order is a workaround that you can use but it
is _not_ documented and supported beyond the visual order of the
display. In theory it could change at any time. In practice it won't
change until NUnit 3.0, so you're pretty safe using it as a
workaround. The basic problem is that the order of test execution
should not be coupled to the display order as it is in version 2.x.
Further problems arise from the fact that the display must be in some
particular order, while the test execution can take place in several
threads, but that also won't apply till 3.0.

On Wed, Mar 23, 2011 at 12:16 PM, Robert Sosnowski
<email address hidden> wrote:
> First I've reviewed existing feature requests. Link here: http://www.nunit.org/index.php?p=contactUs shows http://sourceforge.net/tracker/?atid=360749&group_id=10749&func=browse as appropriate for this.
> Under this last link I've found similar rejected requests for attributes showing order or dependency between tests: 878856, 1800615, 685680.
> However I've just found in the documentation that order of executing tests is documented and supported (alphabetical order). After upgrading NUnit I've got many failing tests because of incorrect order of tests.
> While this is not as good as it could be with attributes I see that database integration tests can be done using the alphabetical order.
> With this workaround I will wait for attributes.
>
> --
> You received this bug notification because you are subscribed to NUnit
> Extended Testing Platform.
> https://bugs.launchpad.net/bugs/740539
>
> Title:
>  Feature request: attrbiute for controlling of test order
>