Feature: Extraction of unit tests from NUnit test asembly and calling appropriate one

[Milan]

Hi all,

in order to integrate NUnit better in the existing solution it would be good that
by using NUnit(GUI or command line version) over a given Test.exe (where all
necessery test tags/unit tests are allready compiled) to retreive list
of all Unit test names together with its argumens.

In particular case one have Test.exe which is written in order to test some
library Library.dll and now by using command line one call
NUNit over the Test.exe and to get back list of all Unit tests which are
ready to be tested. Also one could as command line argument submit which
test class or which cathegory is imjortant so that list of the returned tests is filtered.

Once the list of all Unit test cases is present (it can be displayed in some other
industry integrated tool) tester can decide what test case will be executed by calling
again using command line specific Unit test. It would be also very good that every test
has its testresult file, having everything in one file makes problem in accessing only
important test and it is not in accordance with other HW/SW test tool where every test
has its testresult file.

Br,
Milan.

[Charlie Poole (charlie.poole)]

You appear to be asking for what NUnit Gui already does - that is, you can point it at an assembly and it provides a display of all the tests.

Please explain what you want that is not available from NUnit.exe.

The issue of the test result file is separate, of course.

Charlie

[Milan (milan-vukoslavcevic)]

Hi,
I am not experienced with console runner so what I wanted is to see first the list of all test cases from one assembly like it is showed in GUI runner.
With the help from console runner I have following info found but there is no option List All test cases or similar. I find option of listing all test cases important for me because I could take that list (could be thousends of test cases-with combinotoric input parameter combinations) so I can export that list to some other seqeuncer(in company I work for, we use company specific sequencer):

NUNIT-CONSOLE [inputfiles] [options]

Runs a set of NUnit tests from the console.

You may specify one or more assemblies or a single
project file of type .nunit.

Options:
/fixture=STR Test fixture or namespace to be loaded (Deprecated) (Short format: /load=STR)
/run=STR Name of the test case, fixture or namespace to run
/config=STR Project configuration (e.g.: Debug) to load
/xml=STR Name of XML output file (Default: TestResult.xml)
/xmlConsole Display XML to the console (Deprecated)
/output=STR File to receive test output (Short format: /out=STR)
/err=STR File to receive test error output
/labels Label each test in stdOut
/include=STR List of categories to include
/exclude=STR List of categories to exclude
/process=X Process model for tests: Single, Separate, Multiple
/domain=X AppDomain Usage for tests: None, Single, Multiple
/framework=STR Framework version to be used for tests
/noshadow Disable shadow copy when running in separate domain
/nothread Disable use of a separate thread for tests
/timeout=X Set timeout for each test case in milliseconds
/wait Wait for input before closing console window
/nologo Do not display the logo
/nodots Do not display progress
/help Display help (Short format: /?)

br,
Milan.

[Charlie Poole (charlie.poole)]

Hi Milan,

What would you expect with regard to invalid, ignored or skipped
tests? Note that tests may be skipped on a certain platform or using a
certain culture, but executed in a different environment.
Will you always create the list under the same environment as it is
eventually used?

Charlie

On Thu, Jul 15, 2010 at 8:49 AM, Milan
<email address hidden> wrote:
> Hi,
> I am not experienced with console runner so what I wanted is to see first the list of all test cases from one assembly like it is showed in GUI runner.
> With the help from console runner I have following info found but there is no option List All test cases or similar. I find option of listing all test cases important for me because I could take that list (could be thousends of test cases-with combinotoric input parameter combinations) so I can export that list to some other seqeuncer(in company I work for, we use company specific sequencer):
>
> NUNIT-CONSOLE [inputfiles] [options]
>
> Runs a set of NUnit tests from the console.
>
> You may specify one or more assemblies or a single
> project file of type .nunit.
>
> Options:
> /fixture=STR            Test fixture or namespace to be loaded (Deprecated) (Short format: /load=STR)
> /run=STR                Name of the test case, fixture or namespace to run
> /config=STR             Project configuration (e.g.: Debug) to load
> /xml=STR                Name of XML output file (Default: TestResult.xml)
> /xmlConsole             Display XML to the console (Deprecated)
> /output=STR             File to receive test output (Short format: /out=STR)
> /err=STR                File to receive test error output
> /labels                 Label each test in stdOut
> /include=STR            List of categories to include
> /exclude=STR            List of categories to exclude
> /process=X              Process model for tests: Single, Separate, Multiple
> /domain=X               AppDomain Usage for tests: None, Single, Multiple
> /framework=STR          Framework version to be used for tests
> /noshadow               Disable shadow copy when running in separate domain
> /nothread               Disable use of a separate thread for tests
> /timeout=X              Set timeout for each test case in milliseconds
> /wait                   Wait for input before closing console window
> /nologo                 Do not display the logo
> /nodots                 Do not display progress
> /help                   Display help (Short format: /?)
>
> br,
> Milan.
>
> --
> Feature: Extraction of unit tests from NUnit test asembly and calling appropriate one
> https://bugs.launchpad.net/bugs/531873
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
>

[Milan (milan-vukoslavcevic)]

Hi Charlie,

I did not go so deep in consideration but I was wanted just "simple" option of nunit-console to list all test cases with its input parameters as they would be showed in GUI runner. That means before unit tests even run! - just to read all test cases from some test assembly, filtered by selected cathegory(like in GUI runner). So for example it could be: nunit-console.exe milans_test.dll /Category: Testable.
If the cathegory is set to "Testable", than just unit cases under such cathegory should be listed. I usually have 2 cathegories "Testable" and "Untestable".
Listing could be also usefull after test cases have run, but for that case I do not have clear "requirements".

Br,
Milan.

[Charlie Poole (charlie.poole)]

You misunderstand me. The questions I am asking all pertain to test
cases "before they are run"
I have to decide whether to include each status category I mentioned.
I left out the categories
like success, failure and error, which can only appear after the test is run.

Please think about what would be most useful for each of these
categories (NotRunnable,
Ignored, Skipped). What happens if your external sequencer is given a
test in each of these
categories?

Of course, if you are not sure what's best, we can do a simple
implementation that includes
all of these and you can file bugs if it doesn't work well for your case.

Charlie

On Fri, Jul 16, 2010 at 1:35 AM, Milan
<email address hidden> wrote:
> Hi Charlie,
>
> I did not go so deep in consideration but I was wanted just "simple" option of nunit-console to list all test cases with its input parameters as they would be showed in GUI runner. That means before unit tests even run! - just to read all test cases from some test assembly, filtered by selected cathegory(like in GUI runner). So for example it could be: nunit-console.exe milans_test.dll /Category: Testable.
> If the cathegory is set to "Testable", than just unit cases under such cathegory should be listed. I usually have 2 cathegories "Testable" and "Untestable".
> Listing could be also usefull after test cases have run, but for that case I do not have clear "requirements".
>
> Br,
> Milan.
>
> --
> Feature: Extraction of unit tests from NUnit test asembly and calling appropriate one
> https://bugs.launchpad.net/bugs/531873
> You received this bug notification because you are a member of NUnit
> Developers, which is subscribed to NUnit V2.
>

[Milan (milan-vukoslavcevic)]

Hi Charlie,

I believe we have slight misunderstanding here :(
I am talking from the beginning about nunit-console runner and you
about GUI runner. I am talking about just having a list of all test
cases with all input parameters combinations (like in GUI runner) and
you are talking about result states.

Since you are concentrated in your last email on Categories like (NotRunnable,
Ignored, Skipped), ...sorry but I do not understand that part for me
cathegories are just labels for grouping nunit tests so any name used
there is OK to me.

sorry I could not contribute more,

br,
Milan

On Fri, Jul 16, 2010 at 11:29 PM, Charlie Poole <email address hidden> wrote:
> You misunderstand me. The questions I am asking all pertain to test
> cases "before they are run"
> I have to decide whether to include each status category I mentioned.
> I left out the categories
> like success, failure and error, which can only appear after the test is run.
>
> Please think about what would be most useful for each of these
> categories (NotRunnable,
> Ignored, Skipped). What happens if your external sequencer is given a
> test in each of these
> categories?
>
> Of course, if you are not sure what's best, we can do a simple
> implementation that includes
> all of these and you can file bugs if it doesn't work well for your case.
>
> Charlie
>
> On Fri, Jul 16, 2010 at 1:35 AM, Milan
> <email address hidden> wrote:
>> Hi Charlie,
>>
>> I did not go so deep in consideration but I was wanted just "simple" option of nunit-console to list all test cases with its input parameters as they would be showed in GUI runner. That means before unit tests even run! - just to read all test cases from some test assembly, filtered by selected cathegory(like in GUI runner). So for example it could be: nunit-console.exe milans_test.dll /Category: Testable.
>> If the cathegory is set to "Testable", than just unit cases under such cathegory should be listed. I usually have 2 cathegories "Testable" and "Untestable".
>> Listing could be also usefull after test cases have run, but for that case I do not have clear "requirements".
>>
>> Br,
>> Milan.
>>
>> --
>> Feature: Extraction of unit tests from NUnit test asembly and calling appropriate one
>> https://bugs.launchpad.net/bugs/531873
>> You received this bug notification because you are a member of NUnit
>> Developers, which is subscribed to NUnit V2.
>>
>
> --
> Feature: Extraction of unit tests from NUnit test asembly and calling appropriate one
> https://bugs.launchpad.net/bugs/531873
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in NUnit V2 Test Framework: Incomplete
>
> Bug description:
> Hi all,
>
> in order to integrate NUnit better in the existing solution it would be good that
> by using NUnit(GUI or command line version) over a given Test.exe (where all
> necessery test tags/unit tests are allready compiled) to retreive list
> of all Unit test names together with its argumens.
>
> In particular case one have Test.exe which is written in order to test some
> library Library.dll and now by using command line one call
> NUNit over the Test.exe and to get back list of ...

[Milan (milan-vukoslavcevic)]

Hi Charlie,

I believe we have slight misunderstanding here :(
I am talking from the beginning about nunit-console runner and you
about GUI runner. I am talking about just having a list of all test
cases with all input parameters combinations (like in GUI runner) and
you are talking about result states.

Since you are concentrated in your last email on Categories like (NotRunnable,
Ignored, Skipped), ...sorry but I do not understand that part for me
cathegories are just labels for grouping nunit tests so any name used
there is OK to me.

sorry I could not contribute more,

br,
Milan

[Charlie Poole (charlie.poole)]

Hi Milan,

No, I am talking about the console runner - or even about an independent program
we might write to list tests. The status values are intrinsic to the tests, once loaded,
so any program has to start with a spec that indicates which status values are needed.

Imagine, how can I write a test if I don't know what you want?

I probably confused you by using the word "categories" (with small c) in my last note.
The status we are concerned with is called RunState and it is a property of each test.
The possible values are Runnable, NotRunnable, Explicit, Skipped and Ignored.

You should decide which of these you want. The simplest, of course, is to include
all of them, but that means youir runner, which uses the list, must know how to
deal with all the types.

Considerations for you:

Does your runner recognize [Ignore] [Explicit]?

Can your runner handle non-runnable (invalid) tests or will it crash?

Will your runner handle [Platform], [Culture] and other attributes that
cause skipping a test?

Will your runner always execute on the same machine that is used
to generate the list? Under the same culture? On the same OS?

What do you want done with regard to /other/ options on the command
line?

Charlie

[Charlie Poole (charlie.poole)]

We'll reconsider this in 3.0

[Charlie Poole (charlie.poole)]

This is now provided by the --explore option of the console runner