Refactor scenario tests to address gate performance

Bug #1571092 reported by Peter Stachowski on 2016-04-15
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
Medium
Peter Stachowski

Bug Description

When the scenario tests were originally written, the main goal was to have a series of tests that could be run on multiple datastores while maintaining the same code base (for the most part). This is in the last stages of being realized so the focus is now on having these tests run in the check/gate on each patchset. Before these been accomplished, each individual test needs to be refactored (i.e. broken into smaller parts) to let it to be 'interleaved' with other tests. This will allow long-running single tests (such as those that create an instance) to be run in parallel, thus reducing the amount of time the entire test suite takes.

It is proposed that this work be broken down into smaller parts. First is a general refactor that will remove the creation of the 'runner' from the group. This will allow the same runner to be passed in to each of the newly refactored 'subgroups' so that state can be preserved between them. Once this is done, each existing group can then be split apart into sections. Using the module tests as an example, the following new groups will probably be needed:

module_create: This handles things like creating/deleting modules, etc. No instances are required for this group
module_instance: This handles applying modules to an instance and also creating a new instance with a module. It *won't* wait for the instance to go active however
module_instance_verify: This waits for the instance to go active and runs any other required tests
module_delete: This handles delete related calls, and will delete the instance created by module_instance. It will not wait for it to 'disappear' however.
module_delete_verify: This will wait for the instance to be completely removed.

Specific tests may require more or less groups, depending on their makeup. The above example should give enough details to describe what the flow should look like.

All new tests should adhere to this 'new' standard so that we can preserve as much of the gate resources as possible.

Once all the tests are refactored, the last step will be to 'interleave' them such that wherever possible, 3 instances are being created at the same time, at all times (any more than 3 seem to cause the gate to fail due to not enough resources).

The comments below can be used to paste reviews of the work being done.

description: updated
Amrith Kumar (amrith) on 2016-04-16
Changed in trove:
importance: Undecided → Medium
milestone: none → ongoing
status: New → Confirmed

Fix proposed to branch: master
Review: https://review.openstack.org/309190

Changed in trove:
assignee: nobody → Peter Stachowski (peterstac)
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/309190
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=cc7559f26a3229cdeed97c3988e0707bbec8dbfb
Submitter: Jenkins
Branch: master

commit cc7559f26a3229cdeed97c3988e0707bbec8dbfb
Author: Peter Stachowski <email address hidden>
Date: Tue Apr 19 22:16:17 2016 -0400

    Refactor scenario tests to facilitate multi-group

    The scenario tests need to have the group classes broken
    up to allow interleaving of the different parts of the tests
    so that the gate can finish in a reasonable amount of time.

    The runners, however, need to be the same for each main group,
    thus allowing state to be preserved between groups of the same
    feature. The creation of runners was refactored to allow for this.

    The module tests were also modified to use the new paradym, which
    allowed the removal of the global varaibles in the runner.

    Dependencies on tests.api were also removed.

    Change-Id: I66fcb2bce8c6cbc99c9ff071c242e4741500056d
    Partial-Bug: #1571092

Fix proposed to branch: master
Review: https://review.openstack.org/322290

Fix proposed to branch: master
Review: https://review.openstack.org/322946

Fix proposed to branch: master
Review: https://review.openstack.org/323021

Changed in trove:
assignee: Peter Stachowski (peterstac) → Petr Malik (pmalik)

Fix proposed to branch: master
Review: https://review.openstack.org/323031

Fix proposed to branch: master
Review: https://review.openstack.org/323045

Fix proposed to branch: master
Review: https://review.openstack.org/323060

Changed in trove:
assignee: Petr Malik (pmalik) → Peter Stachowski (peterstac)
Changed in trove:
assignee: Peter Stachowski (peterstac) → Petr Malik (pmalik)
Changed in trove:
assignee: Petr Malik (pmalik) → Peter Stachowski (peterstac)

Fix proposed to branch: master
Review: https://review.openstack.org/324060

Fix proposed to branch: master
Review: https://review.openstack.org/325676

Changed in trove:
assignee: Peter Stachowski (peterstac) → mariam john (mariamj)
Changed in trove:
assignee: mariam john (mariamj) → Peter Stachowski (peterstac)

Fix proposed to branch: master
Review: https://review.openstack.org/331832

Change abandoned by Peter Stachowski (<email address hidden>) on branch: master
Review: https://review.openstack.org/324060

Change abandoned by Peter Stachowski (<email address hidden>) on branch: master
Review: https://review.openstack.org/325676

Change abandoned by Peter Stachowski (<email address hidden>) on branch: master
Review: https://review.openstack.org/324060

Reviewed: https://review.openstack.org/322289
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=b3bb151aede3cae2aa8055f723a1d0d6493c3952
Submitter: Jenkins
Branch: master

commit b3bb151aede3cae2aa8055f723a1d0d6493c3952
Author: Peter Stachowski <email address hidden>
Date: Fri May 27 01:35:04 2016 -0400

    Break apart module scenario tests

    The module scenario tests were broken into separate
    parts to facilitate parallel runs in the gate.

    Change-Id: Ie3262a9e1e5cf88a0b578f648a91e01d564b8812
    Partial-Bug: #1571092

Reviewed: https://review.openstack.org/323045
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=6fddd9922d19378d1c678d6bd2a8a1b157568e0f
Submitter: Jenkins
Branch: master

commit 6fddd9922d19378d1c678d6bd2a8a1b157568e0f
Author: Petr Malik <email address hidden>
Date: Mon May 30 20:39:10 2016 -0400

    Break apart root scenario tests

    The root actions group scenario tests were broken into separate parts
    to facilitate parallel runs in the gate.

    Change-Id: I63e995a2eda71183ef742d1091e7e871b53458a9
    Partial-Bug: 1571092

Reviewed: https://review.openstack.org/323031
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=4b3599a1b25fbffdae76c05fac9ea8c6a4fc1814
Submitter: Jenkins
Branch: master

commit 4b3599a1b25fbffdae76c05fac9ea8c6a4fc1814
Author: Petr Malik <email address hidden>
Date: Mon May 30 18:20:34 2016 -0400

    Break apart user scenario tests

    The user actions group scenario tests were broken into separate parts
    to facilitate parallel runs in the gate.

    Initialized instance delete action was also broken down into
    delete call itself and the wait for it to finish.

    Change-Id: I651219cc1a5dae36d6001548bb6c9ea518cbce97
    Partial-Bug: 1571092

Reviewed: https://review.openstack.org/323021
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=a43dc09d39e91bd84318d3a253093d1a30712c07
Submitter: Jenkins
Branch: master

commit a43dc09d39e91bd84318d3a253093d1a30712c07
Author: Petr Malik <email address hidden>
Date: Mon May 30 17:27:27 2016 -0400

    Break apart database scenario tests

    The database actions group scenario tests were broken into separate parts
    to facilitate parallel runs in the gate.

    Initialized instance delete action was also broken down into
    delete call itself and the wait for it to finish.

    Change-Id: I9f3064f26d83f33e351b6d77b32c7bde4de3eb88
    Partial-Bug: 1571092

Reviewed: https://review.openstack.org/322946
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=a080a69c5896624aa8326b653487eb51bf5e5fd5
Submitter: Jenkins
Branch: master

commit a080a69c5896624aa8326b653487eb51bf5e5fd5
Author: Peter Stachowski <email address hidden>
Date: Mon May 30 13:46:19 2016 -0400

    Break apart configuration group scenario tests

    The configuration group scenario tests were broken into separate
    parts to facilitate parallel runs in the gate.

    Change-Id: Id6f4b0a4060d6c88e547b62e74f97df881cf4425
    Partial-Bug: #1571092

Reviewed: https://review.openstack.org/331832
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=36b27cf6ac397eb60e70cb3b8f9f44cce3aef838
Submitter: Jenkins
Branch: master

commit 36b27cf6ac397eb60e70cb3b8f9f44cce3aef838
Author: Peter Stachowski <email address hidden>
Date: Wed Jun 1 13:46:07 2016 -0400

    Break apart backup scenario tests

    The backup scenario tests were broken into separate
    parts to facilitate parallel runs in the gate.

    Change-Id: Id14eec09174da3bb896a5828991a019fe09cf3e7
    Partial-Bug: #1571092

Reviewed: https://review.openstack.org/323060
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=1273b27d6010b2a753c3368f9b79b1077275e354
Submitter: Jenkins
Branch: master

commit 1273b27d6010b2a753c3368f9b79b1077275e354
Author: Peter Stachowski <email address hidden>
Date: Mon May 30 22:57:20 2016 -0400

    Break apart instance create/actions scenario tests

    The instance create and actions scenario tests were broken
    into separate parts to facilitate parallel runs in the gate.

    Change-Id: Ieeee9bcfe588c0f7ef80cb95eda33059d0486813
    Partial-Bug: #1571092

Reviewed: https://review.openstack.org/322290
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=2070147ae16005dd87e994dbeaa35f1deb5a603e
Submitter: Jenkins
Branch: master

commit 2070147ae16005dd87e994dbeaa35f1deb5a603e
Author: Peter Stachowski <email address hidden>
Date: Fri May 27 12:03:39 2016 -0400

    Break apart replication scenario tests

    The replication scenario tests were broken into separate
    parts to facilitate parallel runs in the gate.

    The verify_data helper was also enhanced to retry so that
    arbitrary sleeps are no longer necessary to ensure that
    data has time to replicate.

    Additional data was also added/verified once the replication
    network is back to its original configuration.

    Partial-Bug: #1571092
    Change-Id: I78bc945b36b1abba19e102cc2742984bd2375513

Reviewed: https://review.openstack.org/334135
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=e7bab492149ce54a04062645a5d9114018808923
Submitter: Jenkins
Branch: master

commit e7bab492149ce54a04062645a5d9114018808923
Author: Peter Stachowski <email address hidden>
Date: Thu Jun 2 13:16:28 2016 -0400

    Reorganize scenario test order to speed up run

    This changeset stitches together all the different groups
    in a way to have 3 instances being created at any given
    time (where possible). The groupings will be:

    -- module
    |- configuration
    -- instance_actions

    -- database_actions
    |- backup
    -- instance_actions

    -- user_actions
    |- root_actions
    -- guest_log

    -- replication

    Times (on a fast test machine) went from 2230s to 1720s, an
    improvement of 23%.

    The group labels were moved into the __init__.py file to facilitate
    being accessed by different group implementations (to avoid a
    circular import issue).

    Tests were also cleaned up and homogenized wherever possible.

    Change-Id: Idc3daab243261584ac5fa9350d952f5bb3984300
    Closes-Bug: #1571092

Changed in trove:
status: In Progress → Fix Released

This issue was fixed in the openstack/trove 6.0.0.0b2 development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers