Neutron functional job is failing without output that would aid in debugging

[Maru Newby]

The Neutron functional job has been failing periodically since tests for the l3 agent merged on Aug 29:

https://review.openstack.org/#/c/109860/

The failures leave no useful test output:

http://logs.openstack.org/60/115360/4/check/check-neutron-dsvm-functional/366b616/console.html

http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkludm9jYXRpb25FcnJvcjogJy9vcHQvc3RhY2svbmV3L25ldXRyb24vLnRveC9kc3ZtLWZ1bmN0aW9uYWwvYmluL3B5dGhvblwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI4NjQwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MDk2MTMzMDA3NTJ9

[Robert Collins (lifeless)]

So the root issue here is that though we don't catch BaseException (and shouldn't) sys.exit(0) results in a pass.

[Robert Collins (lifeless)]

See https://bugs.launchpad.net/testtools/+bug/1364188 for the issue in testtools permitting this to happen.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/118233

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/118225
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=486937907a5dec6f1f227a1bc3330cad061ff437
Submitter: Jenkins
Branch: master

commit 486937907a5dec6f1f227a1bc3330cad061ff437
Author: Kevin Benton <email address hidden>
Date: Mon Sep 1 13:03:27 2014 -0700

    Prevent SystemExits when running tests

    Adds a check for SystemExit exceptions during tests
    to convert them into a test failure rather than an exit.
    Includes the traceback in the failure so the source of
    the exit can be located.

    Related-Bug: #1364171
    Change-Id: I2c7d9010ebf935d39ed58fe7a6fc4a1a867e2548

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/118233
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0918ce91e5af5100ee3a4fcd6861741c4f00ac41
Submitter: Jenkins
Branch: master

commit 0918ce91e5af5100ee3a4fcd6861741c4f00ac41
Author: Maru Newby <email address hidden>
Date: Tue Sep 2 03:02:23 2014 +0200

    Skip functional l3 agent test

    Since this test merged there have been periodic failures that
    seem to be related to SystemExit being raised. Skipping for now
    to give us time to fix.

    Change-Id: Ia857835102a9c4c74820cf5eb3c529a3c2fb421e
    Related-bug: 1364171

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/119001

[Maru Newby (maru)]

I'm marking this bug as complete since Kevin's patch did fix the problem of test output not being saved on failure. John's fix will be retargeted at a more specific bug that this bug was masking.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/119034

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/119001
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fdf0fa8b43537bbd5ab6e1a2e55f694b77209a00
Submitter: Jenkins
Branch: master

commit fdf0fa8b43537bbd5ab6e1a2e55f694b77209a00
Author: John Schwarz <email address hidden>
Date: Thu Sep 4 11:34:29 2014 +0300

    Fix leftover Timeout effecting most eventlet calls

    When registering a new eventlet.timeout.Timeout object, eventlet
    automatically starts a timer for most (if not all) future eventlet
    calls. Normally, eventlet codes do not hold a timeout unless such a
    timeout is used or a specific timeout length is specified through
    the API, but once a Timeout object is initialized, it is left there
    unless canceled.

    This change fixes an un-canceled Timeout which causes some
    functional tests to fail, reintroduces a fix for bug #1358206,
    which was written prior to discovering the uncanceled timeout, and
    increases the timeout of a test that depended on this timeout.

    Closes-bug: #1358206
    Related-bug: #1364171
    Change-Id: I1bfc5af6917c525894eecd8b477d787763edbd02

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/119034
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=85aa417d77bd3402690e26f105465f0338897cfd
Submitter: Jenkins
Branch: master

commit 85aa417d77bd3402690e26f105465f0338897cfd
Author: John Schwarz <email address hidden>
Date: Thu Sep 4 14:11:26 2014 +0300

    Revert "Skip functional l3 agent test"

    This reverts commit 0918ce91e5af5100ee3a4fcd6861741c4f00ac41. The
    specified commit skipped a test which caused non-deterministic gate
    failures, but the actual reason for the failures was found and fixed
    so the skip is no longer needed.

    Related-bug: #1358206
    Related-bug: #1364171
    Change-Id: I9b0d9ad9ad8b62ee92877fc0a6db0323604433ac