Test time includes execution time of Setup and Teardown methods

[Dave Kipfer]

The calculated time of a test includes its SetUp and TearDown methods.

This includes:

The "time" attribute in the "test-case" element in an XML report is not limited to the execution time of the decorated test method.

The "MaxTime" attribute fails on this same value (time of Setup + time of teardown + time of test). This is first of all misleading, and secondly problematic.

A work around would appear to be to avoid Setup and Teardown methods (as has been suggested for other reasons by some) and do setup/teardown in the scope of the fixture or the test class constructor/destructor. In many cases, inlining setup and teardown within the test case itself is not a feasible workaround.

[Simone Busoli (simone.busoli)]

Hi Dave,

it would be useful to know the reasons why you think that this behavior is misleading and problematic.

[Charlie Poole (charlie.poole)]

Hi Dave,

This has been the historical behavior of NUnit for a long time. SetUp
and TearDown are considered an integral part of the test and we have
generally felt that refactoring by moving some setup from SetUp to
the test itself - or vice versa - should not change the time reported.

As Simone suggests, can you give us some example of why this
presents a problem for you?

Charlie

On Fri, Oct 26, 2012 at 10:44 AM, Simone Busoli
<email address hidden> wrote:
> Hi Dave,
>
> it would be useful to know the reasons why you think that this behavior
> is misleading and problematic.
>
> --
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
> https://bugs.launchpad.net/bugs/1071819
>
> Title:
> Test time includes execution time of Setup and Teardown methods
>
> Status in NUnit V2 Test Framework:
> New
>
> Bug description:
> The calculated time of a test includes its SetUp and TearDown methods.
>
> This includes:
>
> The "time" attribute in the "test-case" element in an XML report is
> not limited to the execution time of the decorated test method.
>
> The "MaxTime" attribute fails on this same value (time of Setup + time
> of teardown + time of test). This is first of all misleading, and
> secondly problematic.
>
> A work around would appear to be to avoid Setup and Teardown methods
> (as has been suggested for other reasons by some) and do
> setup/teardown in the scope of the fixture or the test class
> constructor/destructor. In many cases, inlining setup and teardown
> within the test case itself is not a feasible workaround.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nunitv2/+bug/1071819/+subscriptions
>
> _______________________________________________
> Mailing list: https://launchpad.net/~nunit-core
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~nunit-core
> More help : https://help.launchpad.net/ListHelp

[Dave Kipfer (david-kipfer)]

Hi Charlie, Simone,

I find it misleading as the "test-step" XML element, by all appearances, maps to the "Test" attribute in the class, yet the time elapsed includes "SetUp" and "TearDown". Similarly, "Maxtime" is an attribute against the "Test", not "SetUp" and "TearDown", yet they can affect those results as well. Take a look at the documentation for Maxtime. The language used indicates the timing is against the "Test" method and there are no notes stating otherwise.

I find it problematic as the only way I can see to hide or forgive the performance of "SetUp" and "TearDown" would be to structure the test such that those activities are placed in additional test steps in order to get a timing for the actual stuff I want to time (i.e. what's in "Test"). This results in superfluous test steps in the report which really aren't important in the overall scheme of the test procedure.

If there is some way of suppressing the reporting of test steps, that may alleviate that problem. Otherwise, the time should be a little more specific:

1) In addition to "time", there could be "test-time", "setup-time", and "teardown-time" attributes in the XML report.
2) Attributes such as "Maxtime" could be further parameterized to give direction as to which part's timing is critical, since it applies to more than the test step.

Thanks,
Dave

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Charlie Poole
Sent: Friday, October 26, 2012 2:20 PM
To: Kipfer, David
Subject: Re: [Nunit-core] [Bug 1071819] Re: Test time includes execution timeof Setup and Teardown methods

Hi Dave,

This has been the historical behavior of NUnit for a long time. SetUp and TearDown are considered an integral part of the test and we have generally felt that refactoring by moving some setup from SetUp to the test itself - or vice versa - should not change the time reported.

As Simone suggests, can you give us some example of why this presents a problem for you?

Charlie

On Fri, Oct 26, 2012 at 10:44 AM, Simone Busoli <email address hidden> wrote:
> Hi Dave,
>
> it would be useful to know the reasons why you think that this
> behavior is misleading and problematic.
>
> --
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
> https://bugs.launchpad.net/bugs/1071819
>
> Title:
> Test time includes execution time of Setup and Teardown methods
>
> Status in NUnit V2 Test Framework:
> New
>
> Bug description:
> The calculated time of a test includes its SetUp and TearDown methods.
>
> This includes:
>
> The "time" attribute in the "test-case" element in an XML report is
> not limited to the execution time of the decorated test method.
>
> The "MaxTime" attribute fails on this same value (time of Setup + time
> of teardown + time of test). This is first of all misleading, and
> secondly problematic.
>
> A work around would appear to be to avoid Setup and Teardown methods
> (as has been suggested for other reasons by some) and do
> setup/teardown in the scope of the fixture or t...

[Charlie Poole (charlie.poole)]

Hi Dave,

First I'll address NUnit 2.6.x, then NUnit 3.0....

NUnit 2.6.x
SetUp plus Test plus TearDown have been treated as a single conceptual unit for 12 years and it seems like a very big surprise for users for us to change it. However, now that you have explained why it seems misleading, I think we could explain it a lot better in the docs. We'll do that for 2.6.3.

Regarding the xml output, I'm a bit confused. There is no "test-step" element in the NUnit xml results. Are you looking at something created by another app from the NUnit output?

NUnit 3.0

For 3.0, the door is open to making more significant changes. In fact, some greater level of detail in the xml is a direction I'd like to go. I'm adding this issue to the NUnit 3.0 project as well, as a reminder to re-consider the issues you raise there.

Charlie