"Assembly not supported" for .NET 3.5 test project when extension directory contains .NET 4+ assembly

[Mattias Nordvall]

We are having problem with the test adapter in our TFS build environment. Here are the versions used:

* NUnit version 2.6.2
* NUnit test adapter version 0.96.0.4
* Visual Studio 2012 update 3
* TFS 2012 update 1

Here are the steps to reproduce:
* Create a .NET 3.5 class library project with at lease one NUNit test. In this example it is called "aaa.bbb.Tests.dll"
* Configure your build environment according to this guide: http://www.mytechfinds.com/articles/software-testing/6-test-automation/72-running-nunit-tests-from-team-foundation-server-2012-continuous-integration-build

Basically you copy the test adapter assemblies to a folder reachable by the build controller and edit the "test assembly file specification" in the build definition to match the name of your NUNit test project.

* Queue a build, and wait for it to finish.

Result: You should see correct test results in the build report.

However, the folder specified in "version control path to custom assemblies", where the test adapter assemblies are copied, is used for more than just test adapters. For example you place assemblies that contains workflow activities for customized build workflows. And here comes the strange part:

* Create a new .NET 4.0 class library project in visual studio. Name it something that comes before "nunit" in the alphabet, for example "abc.dll". In our case it contains workflow activities, but we can reproduce the problem with just the the standard "Class1.cs".
* Build the .NET 4.0 project and check in the resulting "abc.dll" to the "version control path to custom assemblies" folder. That is the same folder as the NUNit test adapter assemblies were copied earlier.
* Queue another build.

Result: No tests are run and the build report says:

Assembly not supported: C:\Builds\5\aaa\bbb\bin\aaa.bbb.Tests.dll
No test is available in C:\Builds\5\aaa\bbb\bin\aaa.bbb.Tests.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.

We have found the following ways to make it work again:
* Remove the "abc.dll" from the custom assemblies folder.
* Rename "abc.dll" to "xyz.dll" (!)
* Change the target framework version of "abc.dll" to .NET 3.5.

We can also reproduce the problem locally in Visual Studio, using the same project setup as above. This is a strange thing to do, but much easier for troubleshooting.
* Install the latest NUnit test adapter for Visual Studio.
* Open the .NET 3.5 class library project containing NUnit tests.
* Open the Test Explorer window and build the project. Your tests should appear.
* Right click some of the tests and select "Run selected tests".

Result: Your tests should run successfully.

Now look up the Visual Studio extensions folder, where the test adapter is installed. On my computer it is C:\Users\username\AppData\Local\Microsoft\VisualStudio\11.0\Extensions\do0tbn4e.opv.

* Copy the "abc.dll" to the test adapter folder you found above.
* Restart Visual Studio, open and build the NUnit project

Result: The tests are discovered in the Test Explorer window as before, but when you right click a test and select "Run selected tests" you get the following error message in Output/Tests:

NUnit 0.96.0.4 executing tests is started
Assembly not supported: c:\Projects\aaa\bbb\bin\Debug\aaa.bbb.Tests.dll
NUnit 0.96.0.4 executing tests is finished

We find it very strange that an unreferenced .NET 4.0 assembly in the same folder as the test adapter can disturb the test execution. And we have really no choice to alter our .NET versions. The NUnit test projects need to be 3.5 for legacy reasons and the workflow activity assemblies needs to be .NET 4.0, or actually .NET 4.5, to reference the TFS 2012 assemblies.

[Mattias Nordvall (mattias-nordvall)]

We have found a workaround for this bug:

In "version control path to custom assemblies", the test adapters must be placed in the root. But workflow activities can actually exist in a sub folder. Because of that we can leave the test adapter in the root and move the .NET4 stuff into a subfolder. That makes both the test adapter and the workflow activities work.

Thats OK for now, but we would also like to use the Chutzpah test adapter for JavaScript. It will need to be in the root folder, and if that one is .NET4+ we might the problems back. I'll get back when we tried.

[Terje Sandstrom (b-tfrje-j)]

I've forwarded this to the PG in Microsoft, as this is probably something with the VS Test runner.

[Charlie Poole (charlie.poole)]

Moved to https://github.com/nunit/nunit-vs-adapter/issues/11 and marked Won't Fix here.