Report should display all result states

[Charlie Poole]

Only success, failure and not-run test results are reported currently. Since nunit-results was originally written, NUnit has added a number of other result states... for example: Inconclusive, Canceled, NotRunnable, Skipped and Ignored. The purpose of this feature request is to introduce some or all of those states as colums in the report.

Because nunit-results can only report on information that is included in the xml file from NUnit, this feature depends on NUnit adding new state information to that file. The implementation needs to deal with both old and new formats of the xml file, doing the best it can to determine the result state when input is in the older file format.

[Milan (milan-vukoslavcevic)]

Hi Uniters,

as mention already in this bug report by Charli (#491484) nunit-results-1.1 should list all test cases existing in the NUnit-GUI Tree meaning all PASS Unit-tests function names should be listed under link of respective TestClass.
Once more on Index.html should be list of all test classes and when clicked on it on the second class_name.html page should be list of all PASS and FAILED Unit test function names.

Current state is so that if everythings goes well there is just name of the test-class on index.html page and the number od unit tests cases but no list/information what test cases/functions are executed. If unit test gives FAIL than name and reason is displayed and that is good so.

Having no overview what(unit test function names) went OK is like having no overview :(
I believe that list of PASS unit function names would make test report much more User friendly.

Br,
Milan.

[Charlie Poole (charlie.poole)]

Hi Milan...

Your comment extends the bug significantly from what I had intended. I was thinking of adding columns to the report for Non-Runnable, Inconclusive and other "advanced" run states.

Going to another level of display as an option is a good idea, but will probably end up being split off into a separate feature at the point where we are implementing it.

Charlie

[Milan (milan-vukoslavcevic)]

Hi Charlie,

I was not sure what exactly you thought so I did not want to make
additional Feature/Bug entry, but I believe when you have already
hands on adding such advanced states that you can fast add
successfully/PASS states.
In this moment I think having list of PASS Unit tests names would be
more important than those advanced states. Do you think the same ?

Best regards,
Milan.

On Thu, Dec 10, 2009 at 7:32 PM, Charlie Poole <email address hidden> wrote:
> Hi Milan...
>
> Your comment extends the bug significantly from what I had intended. I
> was thinking of adding columns to the report for Non-Runnable,
> Inconclusive and other "advanced" run states.
>
> Going to another level of display as an option is a good idea, but will
> probably end up being split  off into a separate feature at the point
> where we are implementing it.
>
> Charlie
>
> --
> Report should display all result states
> https://bugs.launchpad.net/bugs/491484
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in NUnit Test Result Reporting: New
>
> Bug description:
> Only success, failure and not run are reported. We should handle all result states.
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/nunit-results/+bug/491484/+subscribe
>

[Milan (milan-vukoslavcevic)]

Additionally should I make the new FEATURE reqest related to
displaying PASS states too ?

regards,
Milan.

On Fri, Dec 11, 2009 at 12:02 PM, Milan Vukoslavcevic
<email address hidden> wrote:
> Hi Charlie,
>
> I was not sure what exactly you thought so I did not want to make
> additional Feature/Bug entry, but I believe when you have already
> hands on adding such advanced states that you can fast add
> successfully/PASS states.
> In this moment I think having list of PASS Unit tests names would be
> more important than those advanced states. Do you think the same ?
>
> Best regards,
> Milan.
>
>
> On Thu, Dec 10, 2009 at 7:32 PM, Charlie Poole <email address hidden> wrote:
>> Hi Milan...
>>
>> Your comment extends the bug significantly from what I had intended. I
>> was thinking of adding columns to the report for Non-Runnable,
>> Inconclusive and other "advanced" run states.
>>
>> Going to another level of display as an option is a good idea, but will
>> probably end up being split  off into a separate feature at the point
>> where we are implementing it.
>>
>> Charlie
>>
>> --
>> Report should display all result states
>> https://bugs.launchpad.net/bugs/491484
>> You received this bug notification because you are a direct subscriber
>> of the bug.
>>
>> Status in NUnit Test Result Reporting: New
>>
>> Bug description:
>> Only success, failure and not run are reported. We should handle all result states.
>>
>> To unsubscribe from this bug, go to:
>> https://bugs.launchpad.net/nunit-results/+bug/491484/+subscribe
>>
>

[Charlie Poole (charlie.poole)]

Bugs need to be separated in order that different people can work on them and they can be assingned diffferent priorities. Sometimes it's hard to judge but this one seems very clear:..

Adding new states that are not known in NUnt 2.2 to nunit-results requires one set of changes. Adding a new reporting level that lists all methods run requires a different set of changes.

Please keep this in mind in the future - for this report, it's already done.

The report content would seem like a very good topic for community discussion - i.e. on the nunit-discuss list.

[Milan (milan-vukoslavcevic)]

Hi Charlie,

my proposals inside Bug #495049 and this Bug #491484 can be handled as one new Feature request so considering me my work inside those two bugs are done.

Meaning Bug #495049 can be closed and this Bug #491484 processed as before my influence.

regards,
Milan.

[Charlie Poole (charlie.poole)]

Regarding Bug #495049, see comments on that bug.

A new request, Bug #495734, has been added to cover your request for details by test case. Adding elapsed time seems
like a separate request, since it applies to all levels, not just methods.

I'm updating the description of this bug to remove any confusion about what it addresses.

[Milan (milan-vukoslavcevic)]

Hi Charlie,

I agree that in the:

"Bug 491484 Report should display all result states"
shoud be handle advanced test states like: Inconclusive, Canceled,
NotRunnable, Skipped and Ignored (for this is needed testresult.xml
changes-not quite easy to do) and in the:

Bug #495734 Add an option to report all test cases for each class to the reports
should produce a more detailed report that lists the results (all PASS
and FAILED test methods) of each test class (for this change
testresult.xml does not have to be changed because it already have
that information inside xml file).

Additionally for every test method there should be additional column
with test method duration. I would put this feature under Bug #495734
and not under new feature request because one that writes code for
reading all test method names from xml file can also read out test
duration (time elapsed) for each of the test methods because it is
written next to the method name in the xml file.

Br,
Milan.

[Charlie Poole (charlie.poole)]

On the NUnit side, this bug requires adding a result attribute to all tests, giving extended result state information...

For example, result="inconclusive".

New reports can use this attribute instead of the older methods of deciding if a test passed, failed or was ignored.

[Charlie Poole (charlie.poole)]

Moved to GitHub: https://github.com/CharliePoole/nunit-results/issues/5