running large tests via nunit-console-x86.exe results in OutOfMemoryException

[SoloSK]

[This issue is now being handled under Github.
See https://github.com/nunit/nunit-console/issues/27]

Hi,

when we're running large tests with many assemblies we get an OutOfMemoryException. Maybe there is an issue with the dispose of the runners. We currently avoid this problem with a temporarily fix. As you can see below, we dispose every runner after its AddResult is complete.

src/ClientUtilities/util/AggregatingTestRunner.cs
...
else
{
    foreach (TestRunner runner in runners)
        if (filter.Pass(runner.Test))
        {
            try { result.AddResult(runner.Run(this, filter, tracing, logLevel)); }
            finally { runner.Dispose(); }
        }
}
...
public void Dispose()
{
    foreach (TestRunner runner in runners)
        if (runner != null)
        {
            //runner.Dispose();
        }
}

We're not sure if this could cause any side effects, but for now it helps us through. We can reconstruct this problem with a simple dummy test where memory is allocated by a big static array. Also get the OutOfMemoryException with a heap of 1.457.708K, with the fix the test is running through with a heap of 450.944K.

We're running NUnit with the pre 2.6.2 version, but the problem also occurs in release 2.6.1.
[Runtime Environment]
[OS Version: Microsoft Windows NT 5.1.2600 Service Pack 3]
[CLR Version: 4.0.30319.269 ( Net 4.0 )]

Tried also some different parameters, but running with default settings for now.

Thanks in advance.

[Charlie Poole (charlie.poole)]

The problem I envision with your fix is that under the Gui, the runners may be needed to re-run the tests. I would expect NUnit to crash in this situation when it tried to use a runner that was already disposed.

I'm about to issue 2.6.2, so it's a bit late to spend time on this now. If you can attach your dummy program that shows the problem in a single domain, we'll try to fix it for 2.6.3.

Charlie

[SoloSK (sebastian-krusch)]

Thanks for your quick answer!

I've attached some dummy assemblies which I'am running via console command line. The call looks like this:
..\nunit-console-x86.exe "C:\..\..\ClassLibrary1.dll" "C:\..\..\ClassLibrary2.dll" "C:\..\..\ClassLibrary3.dll" and so on.

I've also attached one source file of the assemblies, they're all looking similar. The static array is used to simulate the memory allocation of our live tests.

In our live system we're running up to 50 assemblies. Due to reasons of metrics evaluation like code coverage we run these tests in one call, to avoid the necessary of result merging and some other big efforts.

Sebastian

[Darin King (darin-king)]

We are experience the exact same issue when running 750 tests. Is this going to be addressed any time soon?

[Charlie Poole (charlie.poole)]

We no longer track bugs on Launchpad and this one has never been evaluated and moved. I'll do that now and add a pointer to the new bug on GitHub. We'll check it out and report the result there.