Unit tests are super slow

[Brant Knudson]

The unit tests used to take about 5 minutes on my computer. Now they take 15 minutes.

The unit tests got much slower when https://review.openstack.org/#/c/214720/ merged. Initializing the paste pipeline must take a lot longer with this change.

The tests shouldn't be initializing the paste pipeline for tests that only need to validate driver / backend / controller behavior.

This affects developer productivity since it takes too long to validate changes locally.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/218605

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/218610

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/218530
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=d7fd29ceebd451c86b26dcc43e1a9c94f44a4472
Submitter: Jenkins
Branch: master

commit d7fd29ceebd451c86b26dcc43e1a9c94f44a4472
Author: Brant Knudson <email address hidden>
Date: Fri Aug 28 19:21:59 2015 -0500

    Refactor mapping rule engine tests to not create servers

    The MappingRuleEngineTests were proper unit tests in that none
    of them simulated REST calls. These tests should not have been
    initializing servers and should also not have been in a
    test_v3_* file since that indicates that the tests go through
    the v3 REST interface.

    The tests now run faster.

    Partial-Bug: 1490160
    Change-Id: I52f6e1f8f750cdbebcd8364cda406e3464c42355

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/218605
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=02dbacbb4b78b1f64d4974cb2b7c1f146fe104cf
Submitter: Jenkins
Branch: master

commit 02dbacbb4b78b1f64d4974cb2b7c1f146fe104cf
Author: Brant Knudson <email address hidden>
Date: Sat Aug 29 14:45:20 2015 -0500

    Merge v2 client sql tests

    The v2 client-based REST tests were run with both sql and non-sql
    configurations. We don't need to run the v2 REST tests with both
    since the REST response doesn't depend on the backend in use.

    The tests now all run with sql since the sql tests had more
    coverage. The coverage report shows that no code coverage is lost.

    There are fewer tests run so the tests take less time to run.

    Change-Id: I8f30966cf5a68c186bdbaaec8f5e54f7a636ce27
    Partial-Bug: 1490160

[Lance Bragstad (lbragstad)]

From the Keystone meeting yesterday [0], Brant wants to set up a bug bash specifically to improve performance of unit tests.

All of the patches as a result of that work should be tied to this bug report in some way.

[0] http://eavesdrop.openstack.org/irclogs/%23openstack-meeting/%23openstack-meeting.2015-09-01.log.html#t2015-09-01T18:02:42

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/219323
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=828beb09f7bca6e8e97f0751dc6e7717a85fae11
Submitter: Jenkins
Branch: master

commit 828beb09f7bca6e8e97f0751dc6e7717a85fae11
Author: David Stanek <email address hidden>
Date: Tue Sep 1 15:36:46 2015 +0000

    Adds caching to paste deploy's egg lookup

    We are constantly reloading the application object during our tests.
    This caused a slow down when the paste.ini was changed to use entry
    points instead of class paths.

    Closes-Bug: 1490160
    Change-Id: I2f3a1872fccbfb2841907d55630d0eba8c8f73ba

[OpenStack Infra (hudson-openstack)]

Change abandoned by Brant Knudson (<email address hidden>) on branch: master
Review: https://review.openstack.org/218038
Reason: Tests were sped up by some fancy caching.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Brant Knudson (<email address hidden>) on branch: master
Review: https://review.openstack.org/218610
Reason: I don't have time to work on this.