Functional tests failing with FAIL: process-returncode

[Assaf Muller]

After https://review.openstack.org/#/c/277813/, we started seeing failures in the functional job. The root cause is that the patch is using self.addOnException, and it looks like if the method that is invoked on exception itself raises an exception, testr freaks out and fails the test. I think that in this particular case, the method (collect_debug_info) may be executed out of order, after test clean ups already occur so namespaces fixtures are already cleaned up.

Example trace:
http://paste.openstack.org/show/487818/

Failure rate seems to be hovering around 80% in the last couple of days.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/283228
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=544753b2112c927c418de5c13b330d4769138cd9
Submitter: Jenkins
Branch: master

commit 544753b2112c927c418de5c13b330d4769138cd9
Author: Assaf Muller <email address hidden>
Date: Mon Feb 22 19:58:19 2016 +0000

    Revert "tests: Collect info on failure of conn_tester"

    More info in bug report. I suggest we first revert, then
    re-introduce the collect_debug_info patch with a different
    approach. I suspect the fix is not trivial if indeed ordering
    is an issue and the namespaces are cleaned up before
    collect_debug_info is fired.

    Related-Bug: #1548547
    Change-Id: Ice93abbc6e143cdbb90e7d41d1be86dc9eb05006

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/283340

[Jakub Libosvar (libosvar)]

@Assaf: so maybe the metaclass wasn't that bad idea :)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/283340
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7a80caff5975afea757534e767b2d0182b7c9098
Submitter: Jenkins
Branch: master

commit 7a80caff5975afea757534e767b2d0182b7c9098
Author: Kevin Benton <email address hidden>
Date: Mon Feb 22 08:40:59 2016 -0800

    Override addOnException to catch exceptions

    addOnException in testtools will leak exceptions in addOnException
    handlers.[1] Rather than trying to be diligent about making
    bulletproof exception handlers, this patch just overrides
    addOnexception to catch exceptions and add them as details to the
    test failure info.

    1. http://testtools.readthedocs.org/en/latest/for-test-authors.html#addonexception

    Related-Bug: #1548547
    Change-Id: Iee0a4203b8e81cbc0a07d8ea119f04b53f5f3126

[Assaf Muller (amuller)]

This bug was resolved with https://review.openstack.org/283228.