Use testr instead of nose as the unittest runner.

[Clark Boylan]

We want to start using testr as our test runner instead of nose so that we can start running tests in parallel. For the projects that have switched we have seen improvements to test speed and quality.

As part of getting set for that, we need to start using testtools and fixtures so provide the plumbing and test isolation needed for automatic parallelization. The work can be done piecemeal - with testtools and fixtures being added first, and then tox/run_tests being ported to us testr/subunit instead of nose.

This work was semi tracked during Grizzly with this https://blueprints.launchpad.net/openstack-ci/+spec/grizzly-testtools blueprint. I am opening this bug so that we can track migration to testr on a per project basis.

[Angus Salkeld (asalkeld)]

https://review.openstack.org/#/q/status:open+project:openstack/python-ceilometerclient+branch:master+topic:testr,n,z

[Clark Boylan (cboylan)]

https://review.openstack.org/#/c/18637/ implements the use of testr for python-keystoneclient. It is currently in need of reviews.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/29043

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/29044

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/29045

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/29046

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/29047

[Gabriel Hurley (gabriel-hurley)]

I'll be interested to see how this works out since Horizon has fairly different testing infrastructure than most projects since it relies on Django's test harnass as well as selenium...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/29043
Committed: http://github.com/openstack/ceilometer/commit/06093adfafa330a0087cb834ada16fd33510e11b
Submitter: Jenkins
Branch: master

commit 06093adfafa330a0087cb834ada16fd33510e11b
Author: Angus Salkeld <email address hidden>
Date: Thu May 9 14:46:36 2013 +1000

    Get all tests to use tests.base.TestCase

    part of bug 1177924

    Change-Id: If8124e2ca21bf9766adf770e30a630437c6fcd95

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/29045
Committed: http://github.com/openstack/ceilometer/commit/8da9ac25fe95c24eeae9a5654ea05cae504ef56d
Submitter: Jenkins
Branch: master

commit 8da9ac25fe95c24eeae9a5654ea05cae504ef56d
Author: Angus Salkeld <email address hidden>
Date: Tue May 14 14:20:53 2013 +1000

    remove unused tests/skip module

    part of bug 1177924
    Change-Id: I9760d81b1bb52a3847cd2c6cc595dad29042a701

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/29044
Committed: http://github.com/openstack/ceilometer/commit/e9a30f61fe1f51a8758c4587e8822723fefc3c4e
Submitter: Jenkins
Branch: master

commit e9a30f61fe1f51a8758c4587e8822723fefc3c4e
Author: Angus Salkeld <email address hidden>
Date: Fri May 17 09:52:34 2013 +1000

    Change from unittest to testtools

    part of bug 1177924

    Change-Id: I99be2c48dc7143015883b89130083c8ea11abeec
    Signed-off-by: Doug Hellmann <email address hidden>

[Angus Salkeld (asalkeld)]

https://github.com/openstack/python-ceilometerclient/commit/7091db40a9486ebdc8a19699826b171465e40384

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/29046
Committed: http://github.com/openstack/ceilometer/commit/6693c9fb04722de9b911fd51a708d70158760178
Submitter: Jenkins
Branch: master

commit 6693c9fb04722de9b911fd51a708d70158760178
Author: Angus Salkeld <email address hidden>
Date: Sat May 18 17:26:38 2013 +1000

    Use fixtures in the tests

    TempDir() makes sure all files created under it are deleted
    at the end of the tests.
    FakeLogger() hides logs when the test passes and displays logs
    for failed tests.

    part of bug 1177924
    Change-Id: I07acb66daa1932d7864a5431f1b64570b747ce5a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/29047
Committed: http://github.com/openstack/ceilometer/commit/1a723a05c74361d8fe52220061b0697de8003221
Submitter: Jenkins
Branch: master

commit 1a723a05c74361d8fe52220061b0697de8003221
Author: Angus Salkeld <email address hidden>
Date: Sat May 18 18:06:38 2013 +1000

    Replace nose with testr

    - add .testr.conf (in base dir and nova_tests)
    - remove all references to nose
    - adjust paths: A couple of tests would assume the current directory
      was ./tests/ but it is now ./
    - don't run the tests in parallel as the db tests have one connection
      per test class.

    nova_tests hackery:
    It seems testtools/testr imports everything and I just don't see a way of
    running things in tests/ and nova_tests/ seperately but in one .test.conf.
    So if you want to use testr directly you will need to:
     testr run
     cd nova_tests
     testr run

    part of bug 1177924
    Change-Id: I41875dcf94463fa5f9c07a7840c37089226c59ad

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/30291

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/30291
Committed: http://github.com/openstack/cinder/commit/930f5891b0815e1b49b9b2cc840e0c24b2796e84
Submitter: Jenkins
Branch: master

commit 930f5891b0815e1b49b9b2cc840e0c24b2796e84
Author: Michael Kerrin <email address hidden>
Date: Thu May 16 08:10:27 2013 +0000

    Fix up the test framework

    When trying to test a bug the cinder test framework was
    running all the tests when I just wanted one to run.

    Instead of fixing this problem in nose I tried to move cinder to
    testr. I copied a fair bit of code from nova to get this to work.
    This way at least if some one makes an improvement to the nova
    test framework cinder can improve here also.

    Note that now all the tests a thread safe. Added a concurrency argument
    to run_tests.sh that defaults to 1. You can specify your own level then
    and take the risk of intermittent failures. Tell jenkins to use a
    concurrency level of 1 to avoid failing gate jobs.

    Note One of these tests is broken and I have disabled for now. There
    is a bug for this here: https://bugs.launchpad.net/cinder/+bug/1173266

    Note: cinder.api.openstack.FaultWrapper is deprecated and is causing
    a circular import when I try and import cinder.tests.test_wsgi. This
    is fixed by only importing the openstack.FaultWrapper during the
    __init__ phase of the wsgi middleware.

    Fixes: bug 1183434
    Fixes: bug 1177924

    Change-Id: I5e10b55c5b236eb81a6a3e0e9ea56af8ca4ef8e1

[Matt Rutkowski (mrutkows)]

I am out of the office until 07/01/2013.

I will be on vacation until Monday, July 1st and will not be able to
respond to emails for the majority of my time away. Please contact Johanna
Koester, my manager, in case of emergencies.

Note: This is an automated response to your message "[Bug 1177924] Re: Use
testr instead of nose as the unittest runner." sent on 06/26/2013 7:51:29
AM.

This is the only notification you will receive while this person is away.

[Dolph Mathews (dolph)]

I think we should escalate this bug Johanna Koester ^

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/51594

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/51594
Committed: http://github.com/openstack/keystone/commit/59adb86b26394d95dca4e16ee6e099f8012310bf
Submitter: Jenkins
Branch: master

commit 59adb86b26394d95dca4e16ee6e099f8012310bf
Author: David Stanek <email address hidden>
Date: Thu Oct 10 13:36:03 2013 +0000

    Changes to testr as the test runner

    run_tests.sh has also been deprecated.

    Fixes-Bug: #1177924
    Change-Id: I15c1707eb6a62c74a0021a48b89ae15f03fcfea8

[Erno Kuvaja (jokke)]

This has been implemented in Glance change https://review.openstack.org/#/c/59699/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/214206

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/214206
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=9d7f71d5754c8b45f8e7c6ab80202de09933afb8
Submitter: Jenkins
Branch: master

commit 9d7f71d5754c8b45f8e7c6ab80202de09933afb8
Author: Richard Hawkins <email address hidden>
Date: Fri Aug 7 18:14:13 2015 -0500

    Modify functional tests to use ostestr/testr

    Defcore uses Tempest, which uses Test Repository.
    This change makes it easier for Defcore to pull functional
    tests from Swift and run them. Additionally, using testr
    allows tests to be run in parallel.

    Concurrency set to 1 for now, >1 causes failures for
    reasons that are still TBD.

    With switch to ostestr all the server logs are being sent to stdout
    which makes it completely unreadable. Suppressing the logs by default
    now with a flag to enable it if desired.

    Co-Authored-By: John Dickinson <email address hidden>
    Co-Authored-By: Robert Collins <email address hidden>
    Co-Authored-By: Matthew Oliver <email address hidden>
    Co-Authored-By: Ganesh Maharaj Mahalingam <email address hidden>

    Change-Id: I53ef4a116996a772cf1f3abc2eb0ad60047322d5
    Related-Bug: 1177924

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: feature/hummingbird
Review: https://review.openstack.org/264517

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/264517
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=93ddaffaebb620bc712d52d0267194562dce02be
Submitter: Jenkins
Branch: feature/hummingbird

commit f53cf1043d078451c4b9957027bf3af378aa0166
Author: Ondřej Nový <email address hidden>
Date: Tue Jan 5 20:20:15 2016 +0100

    Fixed few misspellings in comments

    Change-Id: I8479c85cb8821c48b5da197cac37c80e5c1c7f05

commit 79222e327f9df6335b58e17a6c8dd0dc44b86c17
Author: ChangBo Guo(gcb) <email address hidden>
Date: Sat Dec 26 13:13:37 2015 +0800

    Fix AttributeError for LogAdapter

    LogAdapter object has no attribute 'warn' but has attribute
    'warning'.

    Closes-Bug: #1529321
    Change-Id: I0e0bd0a3dbc4bb5c1f0b343a8809e53491a1da5f

commit 684c4c04592278a280032002b5313b171ee7a4c0
Author: janonymous <email address hidden>
Date: Sun Aug 2 22:47:42 2015 +0530

    Python 3 deprecated the logger.warn method in favor of warning

    DeprecationWarning: The 'warn' method is deprecated, use 'warning'
    instead

    Change-Id: I35df44374c4521b1f06be7a96c0b873e8c3674d8

commit d0a026fcb8e8a9f5475699cc56e1998bdc4cd5ca
Author: Hisashi Osanai <email address hidden>
Date: Wed Dec 16 18:50:37 2015 +0900

    Fix duplication for headers in Access-Control-Expose-Headers

    There are following problems with Access-Control-Expose-Headers.

    * If headers in X-Container-Meta-Access-Control-Expose-Headers are
      configured, the headers are kept with case-sensitive string.
      Then a CORS request comes, the headers are merged into
      Access-Control-Expose-Headers as case-sensitive string even if
      there is a same header which is not case-sensitive string.

    * Access-Control-Expose-Headers is handled by a list.
      If X-Container/Object-Meta-XXX is configured in container/object
      and X-Container-Meta-Access-Control-Expose-Headers, same header
      is listed in Access-Control-Expose-Headers.

    This patch provides a fix for the problems.

    Change-Id: Ifc1c14eb3833ec6a851631cfc23008648463bd81

commit 0bcd7fd50ec0763dcb366dbf43a9696ca3806f15
Author: Bill Huber <email address hidden>
Date: Fri Nov 20 12:09:26 2015 -0600

    Update Erasure Coding Overview doc to remove Beta version

    The major functionality of EC has been released for Liberty and
    the beta version of the code has been removed since it is now
    in production.

    Change-Id: If60712045fb1af803093d6753fcd60434e637772

commit 84ba24a75640be4212e0f984c284faf4c894e7c6
Author: Alistair Coles <email address hidden>
Date: Fri Dec 18 11:24:34 2015 +0000

    Fix rst errors so that html docs are complete

    rst table format errors don't break the gate job
    but do cause sections of the documents to go missing
    from the html output.

    Change-Id: Ic8c9953c93d03dcdafd8f47b271d276c7b356dc3

commit 87f7e907ee412f5847f1f9ffca7a566fb148c6b1
Author: Matthew Oliver <email address hidden>
Date: Wed Dec 16 17:19:24 2015 +1100

    Pass HTTP_REFERER down to subrequests

    Currently a HTTP_REFERER (Referer) header isn't passed down to
    subreq...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: feature/crypto
Review: https://review.openstack.org/272201

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/272201
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=f9b7fd3074b5b0e5d6ea879d4144f7bfeec5d46b
Submitter: Jenkins
Branch: feature/crypto

commit e13a03c379273ee10e678818078b9c40a96a7dc9
Author: Tim Burke <email address hidden>
Date: Wed Jan 20 16:06:26 2016 -0800

    Stop overriding builtin range

    Change-Id: I315f8b554bb9e96659b455f4158f074961bd6498

commit 0a404def7d54d1ef1c85c11a378052260c4fda4c
Author: John Dickinson <email address hidden>
Date: Wed Jan 20 15:19:35 2016 -0800

    remove unneeded duplicate dict keys

    Change-Id: I926d7aaa9df093418aaae54fe26e8f7bc8210645

commit 221f94fdd39fd2dcd9a2e5565adceab615d55913
Author: John Dickinson <email address hidden>
Date: Tue Jan 19 14:50:24 2016 -0800

    authors and changelog updates for 2.6.0

    Change-Id: Idd0ff9e70abc0773be183c37cd6125fe852da7c0

commit 58359269b0e971e52f0eb7f97221566ca2148014
Author: Samuel Merritt <email address hidden>
Date: Tue Dec 8 16:36:05 2015 -0800

    Fix memory/socket leak in proxy on truncated SLO/DLO GET

    When a client disconnected while consuming an SLO or DLO GET response,
    the proxy would leak a socket. This could be observed via strace as a
    socket that had shutdown() called on it, but was never closed. It
    could also be observed by counting entries in /proc/<pid>/fd, where
    <pid> is the pid of a proxy server worker process.

    This is due to a memory leak in SegmentedIterable. A SegmentedIterable
    has an 'app_iter' attribute, which is a generator. That generator
    references 'self' (the SegmentedIterable object). This creates a
    cyclic reference: the generator refers to the SegmentedIterable, and
    the SegmentedIterable refers to the generator.

    Python can normally handle cyclic garbage; reference counting won't
    reclaim it, but the garbage collector will. However, objects with
    finalizers will stop the garbage collector from collecting them* and
    the cycle of which they are part.

    For most objects, "has finalizer" is synonymous with "has a __del__
    method". However, a generator has a finalizer once it's started
    running and before it finishes: basically, while it has stack frames
    associated with it**.

    When a client disconnects mid-stream, we get a memory leak. We have
    our SegmentedIterable object (call it "si"), and its associated
    generator. si.app_iter is the generator, and the generator closes over
    si, so we have a cycle; and the generator has started but not yet
    finished, so the generator needs finalization; hence, the garbage
    collector won't ever clean it up.

    The socket leak comes in because the generator *also* refers to the
    request's WSGI environment, which contains wsgi.input, which
    ultimately refers to a _socket object from the standard
    library. Python's _socket objects only close their underlying file
    descriptor when their reference counts fall to 0***.

    This commit makes SegmentedIterable.close() call
    self.app_iter.close(), thereby unwinding its generator's stack and
    making it eligible for garbage collection.

    * in Python < 3...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: feature/crypto
Review: https://review.openstack.org/277950

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alistair Coles (<email address hidden>) on branch: feature/crypto
Review: https://review.openstack.org/277950

[Brian Cline (briancline)]

Seems this is implemented as of 2.6.0 right?
https://github.com/openstack/swift/commit/9d7f71d5754c8b45f8e7c6ab80202de09933afb8

[John Dickinson (notmyname)]

Brian, in Swift that's only the functests. Unit tests haven't been ported yet.

[clayg (clay-gerrard)]

fixed I53ef4a116996a772cf1f3abc2eb0ad60047322d5

[Akihiro Motoki (amotoki)]

Horizon is one of projects which do not use testtools for testing. This is because we use Django test framework which depends on python standard unittest. testtools support in Django is out of scope of the horizon project and replacing horizon test from Django to testtools is not productive. we mark this bug as Won't Fix.

On the other hand, Django supports several test runners. Looking at the bug description, the motivation of switching to testtools is to run tests in parallel (and use subunit for output serialization though it is not explicitly mentioned). It is better to be explored separately.

[Jason Grosso (jgrosso)]

https://bugs.launchpad.net/manila-ui/+bug/1177924/comments/34