Failure on tests parameterized with long strings

[Sergey S.]

[This issue is now tracked on github, separately for each project affected:
  https://github.com/nunit/nunit-framework/issues/59 (framework)
  https://github.com/nunit/nunit-gui/issues/20 (gui)
  https://github.com/nunit/nunit-vs-adapter/issues/3 (adapter)]

Having the following test

  [TestCaseSource("SignBufferSamples")]
  public void GetSignBuffer(KeyValuePair<int, string> sample) {}

the adapter fails with the following log:

------ Discover test started ------
Contents of string 'GetSignBuffer([2075, MjAxMjA5MTg0MDAuMDA3MUVVUlBlbnRlZ3l0ZXN0IGFkZHJlc3MxMzQ5MDk5NzgwNGNsYXNzaWZpZWRZVDI2NjEyMzkwNDEwMjAzMjE0M0JZVDI2NjEyMzkwNDEwMjAzMjE0M0JBQVJCREU1VzEwODExMTEyMjIyNTg4ODQ4LUVVUi0zMDMxLTAxLURPTjU4ODg0OC1VQUgtMzAzMS0wMS1ET05FVVJVU0Q1NTUxMDBDb3JudWNvcGlhIEZyZXNoIEdyb2Nlcmllc0FBUkVBTCBCQU5LIEFHLDE1LCBQQVVMSU5FTlNUUkFTU0UsV0lFU0JBREVOLEdFUk1BTllBQVJCREU1VzEwOEFBUkVBTCBCQU5LIEFHLDE1LCBQQVVMSU5FTlNUUkFTU0UsV0lFU0JBREVOLEdFUk1BTll0ZXN0IGRldGFpbHMgb2YgcGF5bWVudEZhbHNlRmFsc2UyNjY0ODQwNDQzMURvbmV0c2tUcnVlRmFsc2VUcnVlRmFsc2VUcnVl])' exceeds max of '449', string has been truncated.

Exception has been thrown by the target of an invocation.
========== Discover test finished: 3203 found (0:00:05.3175317) ==========

My test assembly is built with NUnit 2.6.0.12051. The issue is reproduced no matter which .NET framework I target (3.5, 4.0, 4.5).

[Sergey S. (frogbot)]

Updated to NUnit 2.6.1.12217 - same issue.

[Charlie Poole (charlie.poole)]

Is this happening only in the adapter, or directly under NUnit as well?

Charlie

On Thu, Oct 11, 2012 at 4:06 AM, Sergey S. <email address hidden> wrote:
> Updated to NUnit 2.6.1.12217 - same issue.
>
> --
> You received this bug notification because you are subscribed to NUnit
> Extended Testing Platform.
> https://bugs.launchpad.net/bugs/1065462
>
> Title:
> Failure on tests parameterized with long strings
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nunit-vs-adapter/+bug/1065462/+subscriptions

[Sergey S. (frogbot)]

In the adapter only, both NUnit GUI and command line runner work fine.

[Charlie Poole (charlie.poole)]

I'm guessing that the IDE has a size limit for test display names that we have to adhere to. I've sent a note to the MS dev folks to be sure.

A workaround is for you to alias the test case name, using TestCaseData.SetName().

Charlie

[Charlie Poole (charlie.poole)]

It turns out that VS requires the test name to be no more than 449 characters, because it is used as an index into a database by some of the data collectors. It's probably better if NUnit truncates the name itself, by using ellipses in embedded strings.

[Charlie Poole (charlie.poole)]

This bug would require some change in NUnit itself. I'm removing it from the 0.93 milestone and adding NUnit to the list of projects affected.

[Mark Heath (mark-heath)]

We're running into the same issue with some large XML strings as test case sources:

        public string[] LegacyXmlTestCases = new string[]
        {
            Resources.FakeEmergencyAudioItem202,
            Resources.FakeEmergencyAudioItem303,
            Resources.EmergencyEventItem202,
            Resources.EmergencyEventItem303,
        };

        [TestCaseSource("LegacyXmlTestCases")]
        public void CanImportLegacyEmergencyItems(string xml)
        {
            var imported = RecordedItemImportExportHelpers.TestImportXml(xml);
            Assert.IsTrue(RecordedItemUtils.IsEmergency(imported));
        }

[Charlie Poole (charlie.poole)]

It turns out that sometimes the truncation of the name results in a crash, with subsequent tests not being run.

To resolve this, release 0.94 of the adapter will skip any tests that exceed the limit. You will have to use an alternate name in order for the test to be run.

The final fix has to wait for NUnit 2.6.3, where I propose to truncate each string longer than 20 chars, adding ... at the end. I'd appreciate comments from the folks experiencing this problem.

[Mark Heath (mark-heath)]

Sounds like a reasonable solution to me. Personally I think that the VS Test Explorer should have special support for parameterised tests, and only populate the nodes for individual test cases when the user expands the node or runs the test (I think the NUnit GUI already does this). Sometimes we use TestCaseSource to probe various network locations for data files, and I only want the testcasesource function running if I actually choose to run that test.

Thanks for your work on NUnit. The VS test adapter has potential to be very useful for us once VS supports filtering by test category.

[Charlie Poole (charlie.poole)]

Moved to https://github.com/nunit/nunit-vs-adapter/issues/3 and marked Won't Fix here.