nunit-agent hangs after tests complete

[Andy Lindeman]

If a test fixture is being run in a different process (using nunit-agent), every now and then nunit-console will print out the results of the tests and exit, but the nunit-agent process will hang around indefinitely.

I believe the issue is with .NET Remoting, specifically the TestAgent/RemoteTestAgent.Stop operations. Stop is marked as a one-way call, which tells the .NET Framework that the call can be completed asynchronously. However, this also means that when nunit-console exits immediately after calling Stop, the message may never actually be sent to the nunit-agent.

Recommend removing the OneWay attribute so that the call is completed synchronously and thereby guaranteed to be delivered.

[Ace Olszowka (aolszowka)]

I believe we are seeing the same issue every so often, I'm grabbing the source to try the fix suggested by Andy Lindeman. Because the issue is transient it maybe hard to tell if the suggested fix actually worked. Here's our environment encase this helps:

NUnit Release: 2.5.7.10213
OS: Windows Server 2008R2
CLR Version (As reported by Help->About NUnit in the GUI): 2.0.50727.4952

All of our Test Assemblies are built against .NET 4.0.30319 and forced x86. We're using the nunit-console-x86 application to run our unit tests with the following arguments:

"C:\Program Files (x86)\NUnit 2.5.7\bin\net-2.0\nunit-console-x86.exe" /nologo C:\BuildOutput\Release\SomeUnitTest.dll /include=Smoke /xml=C:\BuildOutput\Release\SomeUnitTest.dll.TestResult.xml /framework=4.0.30319

This seems to have become more prominate since our swtich to .NET 4.0, although this could be a concidence.

[Andy Lindeman (alindeman)]

It could definitely be related to the upgrade to .NET 4.0 because
nunit *has* to use the agent in that case .. it can run .NET 1.0-3.5
in process without the agent.

Let me know if I can help more.

[Ace Olszowka (aolszowka)]

Thanks Charlie for getting that fix committed, is there any place where I might find a nightly development build to test the fix? or is my best bet to try and build locally? I'll admit I'm pretty new to Launchpad and I'm not quite sure how to grab the latest version of the source.

[Charlie Poole (charlie.poole)]

You'd have to build it from the latest source. It's fairly trivial to
install bazaar
and get the latest source. See the developer FAQ on the wiki.

Charlie

On Mon, Sep 27, 2010 at 10:23 AM, Ace Olszowka
<email address hidden> wrote:
> Thanks Charlie for getting that fix committed, is there any place where
> I might find a nightly development build to test the fix? or is my best
> bet to try and build locally? I'll admit I'm pretty new to Launchpad and
> I'm not quite sure how to grab the latest version of the source.
>
> --
> nunit-agent hangs after tests complete
> https://bugs.launchpad.net/bugs/602761
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
>

[Rob Burns (masterob8)]

Hey Charlie, I just got latest and compiled the code (revision 3236) and it looks like this issue is still present. I am running Nunit-console as part of a Nant build and now every test fixture is leaving an nunit-agent process running. I am using .net 4.0 framework and running on a Windows server R2 machine.

My NAnt lines are all in this format :
<exec program="${NunitEXE}" failonerror="false" timeout="3000000">
   <arg line="/run:TestClasses.BTests ${TestDLL} /framework=4.0.30319 /noshadow /nologo /labels /out:${TextOutputPath}\BTest.txt /xml:${LogFilePath}\TestBTest.xml" />
</exec>

Any help would be greatly appreciated. Is there another way to run nunit to work around this issue?

[Charlie Poole (charlie.poole)]

If your version supports it, you can just tell NAnt to use net-4.0. If
not, exec a bat
file that set's COMPLUS_Version before running nunit-console to get the same
effect. That way, nunit-agent won't be used at all.

Charlie

On Fri, Oct 8, 2010 at 9:08 AM, Rob Burns <email address hidden> wrote:
> Hey Charlie, I just got latest and compiled the code (revision  3236)
> and it looks like this issue is still present. I am running Nunit-
> console as part of a Nant build and now every test fixture is leaving an
> nunit-agent process running. I am using .net 4.0 framework and running
> on a Windows server R2 machine.
>
> My NAnt lines are all in this format :
> <exec program="${NunitEXE}" failonerror="false" timeout="3000000">
>                        <arg line="/run:TestClasses.BTests ${TestDLL} /framework=4.0.30319 /noshadow /nologo /labels /out:${TextOutputPath}\BTest.txt /xml:${LogFilePath}\TestBTest.xml" />
> </exec>
>
> Any help would be greatly appreciated. Is there another way to run nunit
> to work around this issue?
>
> --
> nunit-agent hangs after tests complete
> https://bugs.launchpad.net/bugs/602761
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
>

[Sloan (sloan-ipass)]

Sorry, I reported this was happening to me. It turns out I was using the x386 version of the console app, on a 64bit machine.

I was using nunit-console-x86.exe instead of nunit-console.exe.

My apologies.

[Charlie Poole (charlie.poole)]

See Bug #612052, which has just been reactivated. Since the fix for that bug released in 2.5.8 causes the agent
process to _always_ hang around, it masks the symptoms of the current bug. We'll have to fix that one first.

[Charlie Poole (charlie.poole)]

Still a problem with nunit-console, even after fixing Bug #612052

[Michael Priemaux (michael-primeaux)]

I just tried the development snapshot build 2.5.9.10304 and the hang is still present. Specifically, the nunit-agent.exe process remains resident. Terminating it causes the build to continue as expected.

[Andy Lindeman (alindeman)]

Is there any way to get NUnit to run .NET 4.0 assemblies without the
agent? While the agent would still be broken, in the short term, it
would at least be a workaround for most of us, it seems.

On 2010-11-01, <email address hidden> <email address hidden> wrote:
> I just tried the development snapshot build 2.5.9.10304 and the hang is
> still present. Specifically, the nunit-agent.exe process remains
> resident. Terminating it causes the build to continue as expected.
>
> --
> nunit-agent hangs after tests complete
> https://bugs.launchpad.net/bugs/602761
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in NUnit V2 Test Framework: Fix Committed
>
> Bug description:
> If a test fixture is being run in a different process (using nunit-agent),
> every now and then nunit-console will print out the results of the tests and
> exit, but the nunit-agent process will hang around indefinitely.
>
> I believe the issue is with .NET Remoting, specifically the
> TestAgent/RemoteTestAgent.Stop operations. Stop is marked as a one-way
> call, which tells the .NET Framework that the call can be completed
> asynchronously. However, this also means that when nunit-console exits
> immediately after calling Stop, the message may never actually be sent to
> the nunit-agent.
>
> Recommend removing the OneWay attribute so that the call is completed
> synchronously and thereby guaranteed to be delivered.
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/nunitv2/+bug/602761/+subscribe
>

--
Sent from my mobile device

Andy Lindeman
http://www.andylindeman.com/

[Charlie Poole (charlie.poole)]

Yes, you can simply run NUnit under .NET 4.0. This is done by any of
the following
means:

1) Modifying the config file to require .NET 4.0
2) Setting the COMPLUS_Version environment variable
3) Not having any other .NET version on your machine. :-)

The details for all of these - well the first two - have already been covered n
the mail list.

However, the fix is already committed if you build from source and today's
snapshot will have it.

Charlie

On Mon, Nov 1, 2010 at 2:52 PM, Andy Lindeman <email address hidden> wrote:
> Is there any way to get NUnit to run .NET 4.0 assemblies without the
> agent?  While the agent would still be broken, in the short term, it
> would at least be a workaround for most of us, it seems.
>
> On 2010-11-01, <email address hidden> <email address hidden> wrote:
>> I just tried the development snapshot build 2.5.9.10304 and the hang is
>> still present. Specifically, the nunit-agent.exe process remains
>> resident. Terminating it causes the build to continue as expected.
>>
>> --
>> nunit-agent hangs after tests complete
>> https://bugs.launchpad.net/bugs/602761
>> You received this bug notification because you are a direct subscriber
>> of the bug.
>>
>> Status in NUnit V2 Test Framework: Fix Committed
>>
>> Bug description:
>> If a test fixture is being run in a different process (using nunit-agent),
>> every now and then nunit-console will print out the results of the tests and
>> exit, but the nunit-agent process will hang around indefinitely.
>>
>> I believe the issue is with .NET Remoting, specifically the
>> TestAgent/RemoteTestAgent.Stop operations.  Stop is marked as a one-way
>> call, which tells the .NET Framework that the call can be completed
>> asynchronously.  However, this also means that when nunit-console exits
>> immediately after calling Stop, the message may never actually be sent to
>> the nunit-agent.
>>
>> Recommend removing the OneWay attribute so that the call is completed
>> synchronously and thereby guaranteed to be delivered.
>>
>> To unsubscribe from this bug, go to:
>> https://bugs.launchpad.net/nunitv2/+bug/602761/+subscribe
>>
>
> --
> Sent from my mobile device
>
> Andy Lindeman
> http://www.andylindeman.com/
>
> --
> nunit-agent hangs after tests complete
> https://bugs.launchpad.net/bugs/602761
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
>

[Charlie Poole (charlie.poole)]

The fix was committed after the release of that snapshot, so you can only
get it from the source repository right now. The next snap will have it.

Charlie

On Mon, Nov 1, 2010 at 2:38 PM, <email address hidden>
<email address hidden> wrote:
> I just tried the development snapshot build 2.5.9.10304 and the hang is
> still present. Specifically, the nunit-agent.exe process remains
> resident. Terminating it causes the build to continue as expected.
>
> --
> nunit-agent hangs after tests complete
> https://bugs.launchpad.net/bugs/602761
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
>

[Michael Priemaux (michael-primeaux)]

Charlie,

I can confirm the latest development snapshot (2.5.9.10305) does indeed correct the hang in our environment.

Regards,
Michael

[Charlie Poole (charlie.poole)]

Great. Thanks for the info.

Charlie

On Tue, Nov 2, 2010 at 11:49 AM, Michael Priemaux
<email address hidden> wrote:
> Charlie,
>
> I can confirm the latest development snapshot (2.5.9.10305) does indeed
> correct the hang in our environment.
>
> Regards,
> Michael
>
> --
> nunit-agent hangs after tests complete
> https://bugs.launchpad.net/bugs/602761
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
>

[Anders Juul (andersjuulsfirma)]

Hi all,

I had same issue, fell over this discussion and upgraded my nunit from 2.5.8 to NUnit-2.5.9.10344 (first non-official release of any kind I ever install on a work machine!) - and it works like a charm.

It just struck me how I take it for granted that nunit works - and I now appreciate your effort more.

Thanks for providing the best test framework!

Anders,
Denmark

[Dean Meyer (dean-meyer)]

We have downloaded 2.5.9.10348 and continue to see the nunit-agent-x86 process hanging around when using nunit-console-x86. However, this appears to occur only when we kill the nunit-console-x86 process during the test. If the test completes successfully, then all is well.

Cheers,
Dean

[Hawkeye Parker (hawkeye-parker)]

We're still seeing the nunit-agent process hang (indefinitely) in 2.5.9.10348, even with successful runs:

.\NUnit\nunit-console.exe /fixture=Autodesk.Buzzsaw.Test.UnitTest.TestCases.UnitTest Autodesk.Buzzsaw.UnitTest.dll
NUnit version 2.5.9.10348

Tests run: 1, Errors: 0, Failures: 0, Inconclusive: 0, Time: 9.6719988 seconds
  Not run: 0, Invalid: 0, Ignored: 0, Skipped: 0

A few notes:
-Each time I run, a new nunit-agent spawns and hangs. I have to manually kill the processes.
-The dll which hosts these "unit tests" is big. Not a lot of tests, per se, but a lot of code: the tests are really high-level, automated integration tests.
-Running through the GUI is different: after running the tests, if I close the GUI, the nunit-agent process stays open for ~10 secs. and then exits. If I load a smaller project into the GUI, then run and exit, nunit-agent exits immediately.

[Aaron Jensen (splatteredbits)]

What's the status on this bug. It is causing our organization a great deal of pain. We have to manually log into our build servers at least once a day to clear out all the nunit-agent.exe processes that didn't close properly. It is very frustrating.

[Russell Myers (3-launchpad-net-russellmyers-com)]

We are seeing what might be a related issue in that the agent hangs and displays a Segmentation Fault error message. We are able to reproduce this on a single Windows 2003 R2 machine; however, other machines with the same operating system do not hang and/or yield a segmentation fault. Like the others, we are running the x86 version and are using 2.5.9.10348. Also, the failures inconsistently happen for the same set of tests.

If there is any more information I can provide, please let me know!

[Charlie Poole (charlie.poole)]

Do you also see this error when running under TeamCity? Does it appear
when running the console independently?

On Mon, Mar 21, 2011 at 11:55 AM, Russell Myers
<email address hidden> wrote:
> We are seeing what might be a related issue in that the agent hangs and
> displays a Segmentation Fault error message. We are able to reproduce
> this on a single Windows 2003 R2 machine; however, other machines with
> the same operating system do not hang and/or yield a segmentation fault.
> Like the others, we are running the x86 version and are using
> 2.5.9.10348. Also, the failures inconsistently happen for the same set
> of tests.
>
> If there is any more information I can provide, please let me know!
>
> --
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
> https://bugs.launchpad.net/bugs/602761
>
> Title:
>  nunit-agent hangs after tests complete
>

[Charlie Poole (charlie.poole)]

The bug was fixed for many people but then re-opened because others
were still seeing a problem. I don't believe you have reported it before,
so can you provide info on your environment and how you are running NUnit?

Charlie

On Wed, Mar 16, 2011 at 10:50 AM, Aaron Jensen <email address hidden> wrote:
> What's the status on this bug.  It is causing our organization a great
> deal of pain.  We have to manually log into our build servers at least
> once a day to clear out all the nunit-agent.exe processes that didn't
> close properly.  It is very frustrating.
>
> --
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
> https://bugs.launchpad.net/bugs/602761
>
> Title:
>  nunit-agent hangs after tests complete
>

[Russell Myers (3-launchpad-net-russellmyers-com)]

Charlie,

Yes, it is true that I have not reported this before. We just recently upgraded to the newest version of NUnit from a much older version due to the need to support .NET 4.0.

I get a popup and the event log gives me the following description:

"Application popup: nunit-agent-x86.exe - Application Error : The instruction at "0x7c82a82f" referenced memory at "0x4ba1dff4". The memory could not be "written".

Click on OK to terminate the program"

The environment is a Windows 2003 R2 OS. I'm executing this command by command console, invoking nant which then calls nunit-console-x86.exe. What's strange about this is that it comes up inconsistently... so much so that running it up to 10 times may not produce the issue for me. Further, I attempted to run the x86 console directly and was unable to reproduce the issue after several attempts, but I can't be sure that I simply didn't try it enough.

It seems plausible that the issue is environmental, but it seems rather similar to the problem here. Is there any further way that I might be able to diagnose the issue? Logs? Anything I've missed that may be of value?

[Russell Myers (3-launchpad-net-russellmyers-com)]

An update: When we revert to 2.5.7 the issue appears to disappear. We've ran the same sequence that was failing before 20-30 times to confirm. If it reappears with this older version I'll add a comment here.

[Ace Olszowka (aolszowka)]

Just a post for anyone still following this: We ended up just forcing the nunit-console and nunit-gui to run on the .NET Framework 4.0 as suggested by Charlie above. This can be done by the following (since it didn't seem to already be posted in this thread):

1. Open each app.config file in the text editor of your choice (we did nunit.exe, nunit-x86.exe, nunit-console.exe nunit-console-x86.exe)
2. Right below the configuration append the following:
  <startup>
     <supportedRuntime version="v4.0.30319"/>
  </startup>
3. Save the file and restart Nunit

Note that this will force nunit to run with the .net framework v4.0.30319 all the time, for us this isn't a big deal as we have separate build environment for each version of the .net framework on which we deploy against.

[Aaron Jensen (splatteredbits)]

We run on Windows 2008 R2. Our continuous integration server is CruiseControl 1.6. We run NUnit directly from our MSBuild scripts and under NCover 3.4. Here is how we invoke NUnit outside of NCover:

   nunit-console.exe Assembly.dll Assembly2.dll /noshadow /framework=4.0 /domain=Single /include="Unit Test" /xml=NUnitResults.xml

Executing through NCover is a little more complicated. Our build script creates a temporary NUnit project which is then fed into NCover using the MSBuild task's that ship with NCover.

We get zombie/orphaned nunit-agent.exe processes regardless of how NUnit gets run. We are working around the problem by creating a pre-build target that kills any nunit-agent.exe processes that are misbehaving.

[Svein Ackenhausen (ackenpacken)]

Hey, I have the same problem only in a more controlled environment. In the AutoTest.Net project we are hosting NUnit inside our own runner. Right now inside our runner we are issuing a GetCurrentProcess().Kill() to be able to shut down our process. If I remove this line I get the hang issue and our runner never exits. It happens consequently when running tests in the AutoTest.Net project with our own runner. If you guys want to investigate the source is at https://github.com/acken/AutoTest.Net. To get and use just pull, run build.bat and then deploy.bat. All you need is in the ReleaseBinaries folder. I'm running on a 32bit system.

If I can do anything to help you guys out just let me know.

[Mohammed farwana (mferwana)]

Hi
I am having the same problem in our environment.
msbuild hangs forever until I kill nunit-console-(86x).exe manually.

I have nunit (2.5.9) in source control. we are uisng VS2010 on windows 2008 R2 as a TFS.

and we use Windows 7 64bit as build agents.

[Gregory Bok (greg-bok)]

We're having similar problems in our environment. I am wondering if the untimed wait on the RemoteTestAgent stopping in nunit-agent has anything to do with it? Could this be turned into a timed wait?

[Charlie Poole (charlie.poole)]

Making this a timed wait would certainly guarantee stopping within a certain
time, but it might stop prematurely.

When the RemoteTestAgent is started, it immediately registeres with the
TestAgency (part of the runner) and is then given a set of tests to run.
In the case of the Gui runner, it may be asked to load and run tests
multiple times - up to the entire length of the user session.

When the runner no longer needs the agent, it signals it to stop and the
agent process exits.

Even though we can't change that wait, you've given me an idea to look
into: what happens if the runner terminates prematurely? Does that leave
the agent process executing?

Charlie

On Tue, Aug 9, 2011 at 4:51 AM, Gregory Bok <email address hidden> wrote:
> We're having similar problems in our environment.  I am wondering if the
> untimed wait on the RemoteTestAgent stopping in nunit-agent has anything
> to do with it?  Could this be turned into a timed wait?
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/602761
>
> Title:
>  nunit-agent hangs after tests complete
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nunitv2/+bug/602761/+subscriptions
>

[Gregory Bok (greg-bok)]

I now see the issue with making that an untimed wait. The scenario you mentioned is an interesting one however. Is that something that would be apparent from the nunit agent log file?

[Charlie Poole (charlie.poole)]

You would have to compare entries in the agent log file and the
gui or console runner file, looking at the timestamps.

Charlie

On Thu, Aug 11, 2011 at 8:04 AM, Gregory Bok <email address hidden> wrote:
> I now see the issue with making that an untimed wait.  The scenario you
> mentioned is an interesting one however.  Is that something that would
> be apparent from the nunit agent log file?
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/602761
>
> Title:
>  nunit-agent hangs after tests complete
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nunitv2/+bug/602761/+subscriptions
>

[JC (savard-jean-christophe)]

We have the same problem in our organization. We use a single test list for all our different test levels, so a given fixture sometime will called with a test level for which there is no test case. I've found that the problem of unit-agent.exe hanging happen most of the time on fixtures like those that contains 0 test for the current test level.

Would it be possible for the runner to start and exit too fast for the unit-agent.exe to receive the 'end' signal ?

[Charlie Poole (charlie.poole)]

This particular bug is fixed, but "hanging nunit-agent" is an extremely general symptom. If anyone sees the problem again, please start a new bug that includes the precise details of the situation where it happens: 32- versus 64-bit, .NET 4.0 versus 2.0, etc.