skip_by_date shows MTU tests as passing when we believe they should be failing

Bug #1855148 reported by Dan Watkins on 2019-12-04
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
curtin
Medium
Unassigned

Bug Description

e.g. in https://jenkins.ubuntu.com/server/job/curtin-vmtest-devel-amd64-proposed/84/

09:16:36 ======================================================================
09:16:36 ERROR: test suite for <class 'vmtests.test_network_mtu.EoanTestNetworkMtu'>
09:16:36 ----------------------------------------------------------------------
09:16:36 Traceback (most recent call last):
09:16:36 File "/usr/lib/python3/dist-packages/nose/plugins/multiprocess.py", line 788, in run
09:16:36 self.setUp()
09:16:36 File "/usr/lib/python3/dist-packages/nose/suite.py", line 292, in setUp
09:16:36 self.setupContext(ancestor)
09:16:36 File "/usr/lib/python3/dist-packages/nose/plugins/multiprocess.py", line 770, in setupContext
09:16:36 super(NoSharedFixtureContextSuite, self).setupContext(context)
09:16:36 File "/usr/lib/python3/dist-packages/nose/suite.py", line 315, in setupContext
09:16:36 try_run(context, names)
09:16:36 File "/usr/lib/python3/dist-packages/nose/util.py", line 471, in try_run
09:16:36 return func()
09:16:36 File "/var/lib/jenkins/servers/server/workspace/curtin-vmtest-devel-amd64-proposed/curtin-84/tests/vmtests/__init__.py", line 495, in wrapper
09:16:36 raise RuntimeError(msg)
09:16:36 RuntimeError: skip_by_date(EoanTestNetworkMtu.setUpClass) LP: #1671951 fixby=2019-10-02 removeby=2019-10-23: (REMOVE_WORKAROUND) Passed.

Related branches

Ryan Harper (raharper) on 2019-12-10
Changed in curtin:
importance: Undecided → Medium
status: New → Triaged
Revision history for this message
Ryan Harper (raharper) wrote :

I've sorted this out. Once the skip_by_date fixby date has passed, we invoke the test; in this case the test looked like this:

def test_ip_output(self):
   pass

Which passes; but without actually running the test.

What we need to do instead is:

def test_ip_output(self):
    super().test_ip_output()

Revision history for this message
Dan Watkins (oddbloke) wrote : Re: [Bug 1855148] Re: skip_by_date shows MTU tests as passing when we believe they should be failing

On Tue, Feb 04, 2020 at 09:04:05PM -0000, Ryan Harper wrote:
> I've sorted this out. Once the skip_by_date fixby date has passed, we
> invoke the test; in this case the test looked like this:
>
> def test_ip_output(self):
> pass
>
> Which passes; but without actually running the test.
>
> What we need to do instead is:
>
> def test_ip_output(self):
> super().test_ip_output()

Could we achieve the same by entirely omitting the definition of
test_ip_output in the subclass?

Revision history for this message
Ryan Harper (raharper) wrote :

On Tue, Feb 4, 2020 at 3:20 PM Dan Watkins <email address hidden>
wrote:

> On Tue, Feb 04, 2020 at 09:04:05PM -0000, Ryan Harper wrote:
> > I've sorted this out. Once the skip_by_date fixby date has passed, we
> > invoke the test; in this case the test looked like this:
> >
> > def test_ip_output(self):
> > pass
> >
> > Which passes; but without actually running the test.
> >
> > What we need to do instead is:
> >
> > def test_ip_output(self):
> > super().test_ip_output()
>
> Could we achieve the same by entirely omitting the definition of
> test_ip_output in the subclass?
>

I only wanted to skip this one test, so I need to override it to decorate
it.

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1855148
>
> Title:
> skip_by_date shows MTU tests as passing when we believe they should be
> failing
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1855148/+subscriptions
>

Revision history for this message
Dan Watkins (oddbloke) wrote :

On Tue, Feb 04, 2020 at 09:26:20PM -0000, Ryan Harper wrote:
> I only wanted to skip this one test, so I need to override it to decorate
> it.

Aha, makes perfect sense. Thanks!

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

This bug is fixed with commit 2a2bcf03 to curtin on branch master.
To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=2a2bcf03

Changed in curtin:
status: Triaged → Fix Committed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers