ExpectedException specification

[Petr Kuzel]

[Issue now tracked at https://github.com/nunit/nunit-framework/issues/18]

I subclassed ExpectedException attribute passing to super constructor AssertionException type.

Now if I attibute my test method with both my subclass and ExpectedException(AssertionException) I get runner and attribute order dependent behaviour:

  - R# runner interpretes all attributes in sequence so a test method body that passes is toggled to failure by the first ExpectedException attribute then the failure is toggled back to pass by the next ExpectedException attribute.

  - Nunit 2.5.3 gui-runner seems to interpret just the first one

for:

    <Test()> <ObservedBehaviour("Code generator produces duplicates.")> <ExpectedException(GetType(AssertionException))> _
    Public Sub ObservedBehaviourAfterChangeTest()

yielding:

     Observed behaviour has been changed. Please balance the value of the change with compatibility breach costs.
     Originally observed behaviour: Code generator produces duplicates.
     NUnit.Framework.AssertionException was expected

while for

    <Test()> <ExpectedException(GetType(AssertionException))> <ObservedBehaviour("Code generator produces duplicates.")> _
    Public Sub ObservedBehaviourAfterChangeTest()

returning:

    NUnit.Framework.AssertionException was expected

The documentation deserves a clarification.

[Charlie Poole (charlie.poole)]

It is not possible to universally determine textual order of attributes based on the metadata alone. Since NUnit tests are driven by the metadata, and not the source code, there is no way to accomplish what you want. (Note: it could be done in a framework-dependent, version-dependent manner, but not across all frameworks and versions)

Further, NUnit expects and allows only _one_ ExpectedExceptionAttribute on a method. The approach of deriving a new attribute from ExpectedException does not cause the new attribute to recognized by NUnit at all.

Charlie

PS: Sorry for the delay, this was inadvertently marked Incomplete some time ago and I only now re-examined it.

[Petr Kuzel (petrkuzel)]

OK, thank you!

Note that the observation is not aligned with your
expectation (ExEx subclass not recognized at all).

  Cc.

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Charlie Poole
Sent: Saturday, October 30, 2010 20:47
To: Petr Kužel
Subject: [Bug 532536] Re: ExpectedException specification

It is not possible to universally determine textual order of attributes based on the metadata alone. Since NUnit tests are driven by the metadata, and not the source code, there is no way to accomplish what you want. (Note: it could be done in a framework-dependent, version- dependent manner, but not across all frameworks and versions)

Further, NUnit expects and allows only _one_ ExpectedExceptionAttribute on a method. The approach of deriving a new attribute from ExpectedException does not cause the new attribute to recognized by NUnit at all.

Charlie

PS: Sorry for the delay, this was inadvertently marked Incomplete some time ago and I only now re-examined it.

** Changed in: nunitv2
       Status: Incomplete => Won't Fix

--
ExpectedException specification
https://bugs.launchpad.net/bugs/532536
You received this bug notification because you are a direct subscriber of the bug.

Status in NUnit V2 Test Framework: Won't Fix

Bug description:
I subclassed ExpectedException attribute passing to super constructor AssertionException type.

Now if I attibute my test method with both my subclass and ExpectedException(AssertionException) I get runner and attribute order dependent behaviour:

  - R# runner interpretes all attributes in sequence so a test method body that passes is toggled to failure by the first ExpectedException attribute then the failure is toggled back to pass by the next ExpectedException attribute.

  - Nunit 2.5.3 gui-runner seems to interpret just the first one

for:

    <Test()> <ObservedBehaviour("Code generator produces duplicates.")> <ExpectedException(GetType(AssertionException))> _
    Public Sub ObservedBehaviourAfterChangeTest()

yielding:

     Observed behaviour has been changed. Please balance the value of the change with compatibility breach costs.
     Originally observed behaviour: Code generator produces duplicates.
     NUnit.Framework.AssertionException was expected

while for

    <Test()> <ExpectedException(GetType(AssertionException))> <ObservedBehaviour("Code generator produces duplicates.")> _
    Public Sub ObservedBehaviourAfterChangeTest()

returning:

    NUnit.Framework.AssertionException was expected

The documentation deserves a clarification.

To unsubscribe from this bug, go to:
https://bugs.launchpad.net/nunitv2/+bug/532536/+subscribe

ScanedByWebsenseForEurofins

This message has been scanned for malware by SurfControl plc. www.surfcontrol.com

[Charlie Poole (charlie.poole)]

Yes, you are correct. NUnit didn't always recognized derivatives of
its attributes because we reflect on the attribute name. However,
we started descending the inheritance chain for selected attributes
in 2.5 and after a recent bug fix to ExplicitAttribute, I believe all
of them are recognized.

However, where we expect a single attribute, we use the first one
found. AFAIK, there is no way for us to actually enforce use of
a single attribute at compile time when inheritance is used, which
is clearly a source of confusion.

We could introduce a runtime check for those situations where only
one such attribute is used. Or we could simply document the problem.

Charlie

On Tue, Nov 2, 2010 at 1:30 AM, Petr Kuzel <email address hidden> wrote:
> OK, thank you!
>
> Note that the observation is not aligned with your
> expectation (ExEx subclass not recognized at all).
>
>  Cc.
>
> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Charlie Poole
> Sent: Saturday, October 30, 2010 20:47
> To: Petr Kužel
> Subject: [Bug 532536] Re: ExpectedException specification
>
> It is not possible to universally determine textual order of attributes
> based on the metadata alone. Since NUnit tests are driven by the
> metadata, and not the source code, there is no way to accomplish what
> you want. (Note: it could be done in a framework-dependent, version-
> dependent manner, but not across all frameworks and versions)
>
> Further, NUnit expects and allows only _one_ ExpectedExceptionAttribute
> on a method. The approach of deriving a new attribute from
> ExpectedException does not cause the new attribute to recognized by
> NUnit at all.
>
> Charlie
>
> PS: Sorry for the delay, this was inadvertently marked Incomplete some
> time ago and I only now re-examined it.
>
>
> ** Changed in: nunitv2
>       Status: Incomplete => Won't Fix
>
> --
> ExpectedException specification
> https://bugs.launchpad.net/bugs/532536
> You received this bug notification because you are a direct subscriber of the bug.
>
> Status in NUnit V2 Test Framework: Won't Fix
>
> Bug description:
> I subclassed ExpectedException attribute passing to super constructor AssertionException type.
>
> Now if I attibute my test method with both my subclass and
> ExpectedException(AssertionException) I get runner and attribute order
> dependent behaviour:
>
>  - R# runner interpretes all attributes in sequence so a test method
> body that passes is toggled to failure by the first ExpectedException
> attribute then the failure is toggled back to pass by the next
> ExpectedException attribute.
>
>  - Nunit 2.5.3 gui-runner seems to interpret just the first one
>
> for:
>
>    <Test()> <ObservedBehaviour("Code generator produces duplicates.")> <ExpectedException(GetType(AssertionException))> _
>    Public Sub ObservedBehaviourAfterChangeTest()
>
> yielding:
>
>     Observed behaviour has been changed. Please balance the value of the change with compatibility breach costs.
>     Originally observed behaviour: Code generator produces duplicates.
>     NUnit.Framework.AssertionException was expected
>
> while for
>
>    <Test()> <ExpectedException(GetType(As...

[Petr Kuzel (petrkuzel)]

I accepted that you can stay silent - just leave a trace in
the issue tracking system.

My rationale is: in future this could get unlocked somehow.
I can for example image that if attribute order is not in meta
one could add another ordering attribute, or use an implicit order
(first attributes with exact type match then subclasses, ...)

It all depends on future requirements... Currently just
few people would value semantic I targeted by the
ObservedBehaviour attribute, it's not a TDD one...

  Thank you
  Cc.

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Charlie Poole
Sent: Tuesday, November 02, 2010 15:40
To: Petr Kužel
Subject: Re: [Bug 532536] Re: ExpectedException specification

Yes, you are correct. NUnit didn't always recognized derivatives of its attributes because we reflect on the attribute name. However, we started descending the inheritance chain for selected attributes in 2.5 and after a recent bug fix to ExplicitAttribute, I believe all of them are recognized.

However, where we expect a single attribute, we use the first one found. AFAIK, there is no way for us to actually enforce use of a single attribute at compile time when inheritance is used, which is clearly a source of confusion.

We could introduce a runtime check for those situations where only one such attribute is used. Or we could simply document the problem.

Charlie

On Tue, Nov 2, 2010 at 1:30 AM, Petr Kuzel <email address hidden> wrote:
> OK, thank you!
>
> Note that the observation is not aligned with your expectation (ExEx
> subclass not recognized at all).
>
>  Cc.
>
> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf
> Of Charlie Poole
> Sent: Saturday, October 30, 2010 20:47
> To: Petr Kužel
> Subject: [Bug 532536] Re: ExpectedException specification
>
> It is not possible to universally determine textual order of
> attributes based on the metadata alone. Since NUnit tests are driven
> by the metadata, and not the source code, there is no way to
> accomplish what you want. (Note: it could be done in a
> framework-dependent, version- dependent manner, but not across all
> frameworks and versions)
>
> Further, NUnit expects and allows only _one_
> ExpectedExceptionAttribute on a method. The approach of deriving a new
> attribute from ExpectedException does not cause the new attribute to
> recognized by NUnit at all.
>
> Charlie
>
> PS: Sorry for the delay, this was inadvertently marked Incomplete some
> time ago and I only now re-examined it.
>
>
> ** Changed in: nunitv2
>       Status: Incomplete => Won't Fix
>
> --
> ExpectedException specification
> https://bugs.launchpad.net/bugs/532536
> You received this bug notification because you are a direct subscriber of the bug.
>
> Status in NUnit V2 Test Framework: Won't Fix
>
> Bug description:
> I subclassed ExpectedException attribute passing to super constructor AssertionException type.
>
> Now if I attibute my test method with both my subclass and
> ExpectedException(AssertionException) I get runner and attribute or...

[Charlie Poole (charlie.poole)]

For NUnit 3.0 we should re-address this. Options...

1) Allow multiple ExpectedExceptionAttributes on a method, where any of the given exceptions satisfies the test
2) Verify that only one ExpectedException is present and give an error if there are more

This may apply to other attributes as well.