Temptest tests are running in serial on the gate job

Bug #1428847 reported by Kevin Carter
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack-Ansible
Fix Released
Medium
Kevin Carter
Juno
Fix Released
Medium
Kevin Carter
Trunk
Fix Released
Medium
Kevin Carter

Bug Description

The tempest tests are running in parallel on the gate job which frequently results in a failure. To resolve this issue we should run the minimum basic tempest tests using the scripts in a serial mode. This will ensure that we are not overload the resource constrained gate nodes and ensure that the tests pass on their own merit without contending with one another.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to os-ansible-deployment (master)

Fix proposed to branch: master
Review: https://review.openstack.org/161912

Changed in openstack-ansible:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to os-ansible-deployment (master)

Reviewed: https://review.openstack.org/161912
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=255b94d425c0e6bb6a78868b12e5ef1fb4346354
Submitter: Jenkins
Branch: master

commit 255b94d425c0e6bb6a78868b12e5ef1fb4346354
Author: Kevin Carter <email address hidden>
Date: Thu Mar 5 15:35:52 2015 -0600

    Change testr to run in serial

    This change implements the change to allow testr to run in serial which
    should resolve most issues with tempest on a resource constrained host.

    The problem that is being resolved here is that testr will, by default,
    run in multi-threaded and has a relitively high probability to result in
    a 503. This primarily impacts the gate jobs as they are built on a hosts
    with 4 CPUs and 8GB of ram.

    By default if the tempest script is run, testr will run multi-threaded.
    However, if the environment variable RUN_TEMPEST_OPTS is set to
    '--serial' then it will run in serial. This is done for the
    gate-check-commit script so that the change only affects the AIO gate
    check.

    An additional environment variable TESTR_OPTS is added for further
    flexibility.

    Co-Authored-By: Jesse Pretorius <email address hidden>
    Closes-Bug: #1428847
    Change-Id: I8c7ac72ef46ae3f3af3d3503081bec0cbe7276e9

Changed in openstack-ansible:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to os-ansible-deployment (juno)

Fix proposed to branch: juno
Review: https://review.openstack.org/163807

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to os-ansible-deployment (icehouse)

Fix proposed to branch: icehouse
Review: https://review.openstack.org/163808

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to os-ansible-deployment (icehouse)

Reviewed: https://review.openstack.org/163808
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=2a63b0289e207cc71488170e3f7d79e24a91cb4c
Submitter: Jenkins
Branch: icehouse

commit 2a63b0289e207cc71488170e3f7d79e24a91cb4c
Author: Kevin Carter <email address hidden>
Date: Thu Mar 5 15:35:52 2015 -0600

    Change testr to run in serial

    This change implements the change to allow testr to run in serial which
    should resolve most issues with tempest on a resource constrained host.

    The problem that is being resolved here is that testr will, by default,
    run in multi-threaded and has a relitively high probability to result in
    a 503. This primarily impacts the gate jobs as they are built on a hosts
    with 4 CPUs and 8GB of ram.

    By default if the tempest script is run, testr will run multi-threaded.
    However, if the environment variable RUN_TEMPEST_OPTS is set to
    '--serial' then it will run in serial. This is done for the
    gate-check-commit script so that the change only affects the AIO gate
    check.

    An additional environment variable TESTR_OPTS is added for further
    flexibility.

    Conflicts:
            scripts/gate-check-commit.sh
            scripts/run-tempest.sh

    The above conflicts were due to the difference between master and this
    branch with regards to the environment variable. In master all variables
    are exported. This difference will be addressed in a subsequent patch.

    Co-Authored-By: Jesse Pretorius <email address hidden>
    Closes-Bug: #1428847
    Change-Id: I8c7ac72ef46ae3f3af3d3503081bec0cbe7276e9
    (cherry picked from commit 255b94d425c0e6bb6a78868b12e5ef1fb4346354)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to os-ansible-deployment (juno)

Reviewed: https://review.openstack.org/163807
Committed: https://git.openstack.org/cgit/stackforge/os-ansible-deployment/commit/?id=ea09b6e24e752adbf39019cbd5112a18d5be9a3b
Submitter: Jenkins
Branch: juno

commit ea09b6e24e752adbf39019cbd5112a18d5be9a3b
Author: Kevin Carter <email address hidden>
Date: Thu Mar 5 15:35:52 2015 -0600

    Change testr to run in serial

    This change implements the change to allow testr to run in serial which
    should resolve most issues with tempest on a resource constrained host.

    The problem that is being resolved here is that testr will, by default,
    run in multi-threaded and has a relitively high probability to result in
    a 503. This primarily impacts the gate jobs as they are built on a hosts
    with 4 CPUs and 8GB of ram.

    By default if the tempest script is run, testr will run multi-threaded.
    However, if the environment variable RUN_TEMPEST_OPTS is set to
    '--serial' then it will run in serial. This is done for the
    gate-check-commit script so that the change only affects the AIO gate
    check.

    An additional environment variable TESTR_OPTS is added for further
    flexibility.

    Conflicts:
            scripts/gate-check-commit.sh
            scripts/run-tempest.sh

    The above conflicts were due to the difference between master and this
    branch with regards to the environment variable. In master all variables
    are exported. This difference will be addressed in a subsequent patch.

    Co-Authored-By: Jesse Pretorius <email address hidden>
    Closes-Bug: #1428847
    Change-Id: I8c7ac72ef46ae3f3af3d3503081bec0cbe7276e9
    (cherry picked from commit 255b94d425c0e6bb6a78868b12e5ef1fb4346354)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.