Comment 3 for bug 1072379

There was one with Jiri, but I'm not sure if it was on the list or
offline now. Anyway, subsequent discussion seems to show that his
interest is in measuring parts of the test - for example, test case
generation versus setup versus the actual test versus teardown. That's
an entirely different matter and I'll wait for him to write something
about it.

My interest is in the resolution of what we are already measuring. The
issue came up because of how NUnit reports results to VS2012. MSTest
tests show up frequently with timings displayed as '< 1ms' but NUnit
shows a blank or some multiple of 10 to 15ms. Seems to me that if we
are measuring elapsed time, we want it to be accurate for normal
ranges of execution. These days, micro tests run very fast so we need
better resolution and precision of timing than we did in the early
days of NUnit.

I just finished running some benchmarks using two Stopwatches. The
inner Stopwatch measured the time to execute a test for a given number
of repetitions, while the outer one measured that plus the overhead of
the inner stopwatch. The test itself was very fast - it simply
returned Math.Sqrt(d), where d was the iteration count.

Over repetitions from 100 to 100,000 I found that the results were
fairly consistent. The other stopwatch time is a bit less than double
the inner stopwatch. My test itself ran in 10 to 12 nanoseconds on my
laptop, while the stopwatch overhead averaged between 7 and 10
nanoseconds.

In a second set of tests, I measured the time it takes to run my test
method plus two calls to DateTime.Now, one before and one after, which
is our current approach. Using DateTime.Now, my test now ran in a
range from 150 to 200 nanoseconds, that is 15 to 20 times the overhead
of Stopwatch. There doesn't seem to be a downside here.

I think the next step is to actually spike an implementation to see
what the impact is with a set of real tests. Since NUnitLite has
pretty fast tests, that seems to be a good place to start.

Charlie

On Sun, Oct 28, 2012 at 9:17 AM, Simone Busoli
<email address hidden> wrote:
> I probably missed the discussion, but was there a specific requirement
> behind the need for a lower resolution?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1072379
>
> Title:
> Report test execution time at a higher resolution
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nunit-3.0/+bug/1072379/+subscriptions