Tempest and Neutron duplicate tests

Bug #1552960 reported by Assaf Muller on 2016-03-03
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
neutron
High
Unassigned

Bug Description

Problem statement:

1) Tests overlap between Tempest and Neutron repos - 264 tests last I checked. The effect is:
1.1) Confusion amongst QA & dev contributors and reviewers. I'm writing a test, where should it go? Someone just proposed a Tempest patch for a new Neutron API, what should I do with this patch?
1.2) Wasteful from a computing resources point of view - The same tests are being run multiple times for every Neutron patchset.
2) New APIs (Subnet pools, address scopes, QoS, RBAC, port security, service flavors and availability zones), are not covered by Tempest tests. Consumers have to adapt and run both the Tempest tests and the Neutron tests in two separate runs. This causes a surprising amount of grief.

Proposed solution:

For problem 1, we eliminate the overlap. We do this by defining a set of tests that will live in Tempest, and another set of tests that will live in Neutron. More information may be found here: https://review.openstack.org/#/c/280427/. After deciding on the line in the sand, we will delete any tests in Neutron that should continue to live in Tempest. Some Neutron tests were modified after they were copied from Tempest, these modifications will have to be examined and then proposed to Tempest. Afterwards these tests may be removed from Neutron, eliminating the overlap from the Neutron side. Once this is done, overlapping tests may be deleted from Tempest.

For problem 2, we will develop a Neutron Tempest plugin. This will be tracked in a separate bug. Note that there's already a patch for this up for review: https://review.openstack.org/#/c/274023/

* The work is also being tracked here: https://etherpad.openstack.org/p/neutron-tempest-defork

Assaf Muller (amuller) wrote :
description: updated
Changed in neutron:
milestone: none → mitaka-rc1

Reviewed: https://review.openstack.org/280427
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ddd5df95266cf872a852577d16d9c5d188b4e13a
Submitter: Jenkins
Branch: master

commit ddd5df95266cf872a852577d16d9c5d188b4e13a
Author: Assaf Muller <email address hidden>
Date: Thu Mar 3 17:19:13 2016 -0500

    Delete 118~ API tests from Neutron

    Deleted tests that duplicate tests from Tempest, according to the
    new criteria specified in TESTING.rst.

    Follow up steps are detailed here:
    https://etherpad.openstack.org/p/neutron-tempest-defork

    For reviewers, here's how to get a complete list of network
    tests from Tempest. From a Tempest directory, execute:
    testr list-tests tempest.api.network | cut -d"[" -f1 | cut -d"." -f4-
    I verified that every test removed here actually exists in that list.

    Here's a list of patches that modified tests after the fork:
    https://etherpad.openstack.org/p/neutron-tempest-defork-patches-since-initial-sync

    And the list of tests we care about:
    https://etherpad.openstack.org/p/neutron-tempest-defork, Ctrl-F for:
    'Tests that should be synced from Neutron'

    Related-bug: #1552960

    Change-Id: I685291058f221a7ef0b5b7485280cdabceaa4af3

This will spill into Newton.

Changed in neutron:
milestone: mitaka-rc1 → newton-1

Reviewed: https://review.openstack.org/291787
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cc863e07156e3cfe020c5acfcd9875daff477de6
Submitter: Jenkins
Branch: master

commit cc863e07156e3cfe020c5acfcd9875daff477de6
Author: Armando Migliaccio <email address hidden>
Date: Fri Mar 11 08:00:34 2016 -0800

    Remove test_external_network_visibility

    This test was left behind according to plan laid out in patch [1].
    Make its removal conditional to the Tempest sync-up patch.

    [1] https://review.openstack.org/#/c/280427/

    Related-bug: 1553595
    Related-bug: 1552960

    Change-Id: Ia85e7d03c32b5e2db901aed2c13bafe83896be0b
    Depends-on: I0219e17d2ec70939039344826d35cb533b3ce065

Reviewed: https://review.openstack.org/293776
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e9088d09328793495aba260a05d6161f92aae4cd
Submitter: Jenkins
Branch: stable/mitaka

commit e9088d09328793495aba260a05d6161f92aae4cd
Author: Armando Migliaccio <email address hidden>
Date: Fri Mar 11 08:00:34 2016 -0800

    Remove test_external_network_visibility

    This test was left behind according to plan laid out in patch [1].
    Make its removal conditional to the Tempest sync-up patch.

    [1] https://review.openstack.org/#/c/280427/

    Related-bug: 1553595
    Related-bug: 1552960

    (cherry-picked from commit cc863e07156e3cfe020c5acfcd9875daff477de6)
    Change-Id: Ia85e7d03c32b5e2db901aed2c13bafe83896be0b
    Depends-on: I0219e17d2ec70939039344826d35cb533b3ce065

tags: added: in-stable-mitaka

Reviewed: https://review.openstack.org/296987
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=af4492359b50383c95b5eac5041c08d8c7e92275
Submitter: Jenkins
Branch: stable/kilo

commit af4492359b50383c95b5eac5041c08d8c7e92275
Author: Armando Migliaccio <email address hidden>
Date: Fri Mar 11 08:00:34 2016 -0800

    Remove test_external_network_visibility

    This test was left behind according to plan laid out in patch [1].
    Make its removal conditional to the Tempest sync-up patch.

    [1] https://review.openstack.org/#/c/280427/

    Related-bug: 1553595
    Related-bug: 1552960

    Conflicts:
     neutron/tests/api/test_networks.py

    Change-Id: Ia85e7d03c32b5e2db901aed2c13bafe83896be0b
    (cherry picked from commit cc863e07156e3cfe020c5acfcd9875daff477de6)
    (cherry picked from commit ffd82f9ea4e3436eb750803eb7b15770d020f913)

tags: added: in-stable-kilo
tags: added: in-stable-liberty

Reviewed: https://review.openstack.org/296986
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e17f391ab75fa9338f3e7a2c6fa8e49ac43036c0
Submitter: Jenkins
Branch: stable/liberty

commit e17f391ab75fa9338f3e7a2c6fa8e49ac43036c0
Author: Armando Migliaccio <email address hidden>
Date: Fri Mar 11 08:00:34 2016 -0800

    Remove test_external_network_visibility

    This test was left behind according to plan laid out in patch [1].
    Make its removal conditional to the Tempest sync-up patch.

    [1] https://review.openstack.org/#/c/280427/

    Related-bug: 1553595
    Related-bug: 1552960

    Conflicts:
     neutron/tests/api/test_networks.py

    Change-Id: Ia85e7d03c32b5e2db901aed2c13bafe83896be0b
    (cherry picked from commit cc863e07156e3cfe020c5acfcd9875daff477de6)

Download full text (11.4 KiB)

Reviewed: https://review.openstack.org/302570
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=03ccdd6a6d8e4cea9145e39dd7a1614656024aed
Submitter: Jenkins
Branch: master

commit 3213eb124e40b130e174ac3a91067e2b196788dd
Author: Cedric Brandily <email address hidden>
Date: Tue Mar 29 18:06:44 2016 +0200

    Support Routes==2.3

    SubMapper.connect allows in Routes<=2.2 to define routes with/without a
    name:

     map.connect(route_name, route_path)
     map.connect(route_path)

    Routes==2.3 SubMapper.connect now requires a route name (but not
    Mapper.connect). This change adds a route name when needed to support
    Routes<=2.2 and Routes==2.3.

    Closes-Bug: #1563028
    Change-Id: I655bfe7f047945896d63dd94c584acfe60d81b38
    (cherry picked from commit c6efa6292cac9267428366dc3cee00f6fc73e208)

commit 4283a7e36b74823f229400f0f049742f630db28b
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Mar 30 11:48:37 2016 +0200

    Constraint requirements using mitaka upper-constraints.txt file

    Change-Id: I6a908470987ce05769c425c4623df404cef76a10

commit fc690974480611242d4fc6e1deaaf68171fcaed3
Author: OpenStack Proposal Bot <email address hidden>
Date: Wed Mar 30 06:21:15 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I78c24969927224277f381532ad2604cfde8741fc

commit 41be555eddb0f9947fdaa4e73fa74a72677d4d11
Author: OpenStack Proposal Bot <email address hidden>
Date: Tue Mar 29 06:07:19 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: Iafa43bfcfca4d6af0d0866206c07fa817f567de6

commit b435ec56af98d2c04de37dd779ed3613ec4583ad
Author: OpenStack Proposal Bot <email address hidden>
Date: Mon Mar 28 06:07:02 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I9e908579b4704c4294367113523b898839586426

commit bec65f6cdd159ec252c4aa0a63aabc9aea8fcfba
Author: YAMAMOTO Takashi <email address hidden>
Date: Fri Mar 25 15:25:39 2016 +0900

    api tests: Check correct extensions

    auto_allocate -> auto-allocated-topology
    rbac_policies -> rbac-policies

    Closes-Bug: #1561858

    (cherry picked from commit 9d3bab6b8d6b2122732faeb66f831927bee0ac7e)
    Change-Id: Ied13e52bf78afe571f7243e93005bf5c68fe44db

commit f0c3b19165a5ca73ea8f1dc0ddaad290eee35039
Author: OpenStack Proposal Bot <email address hidden>
Date: Fri Mar 25 06:06:36 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: Idb22eac32944a088d24f4d44e537d9bba09dbcc2

commit 99915fa76bd4ce13e0d36059616ba357f4f54ee2
Author: Hynek Mlnarik <email address hidden>
Date: Thu Mar...

Assaf Muller (amuller) on 2016-05-13
no longer affects: neutron/kilo

Related fix proposed to branch: master
Review: https://review.openstack.org/316283

Related fix proposed to branch: master
Review: https://review.openstack.org/316418

Related fix proposed to branch: master
Review: https://review.openstack.org/318495

Reviewed: https://review.openstack.org/316283
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a33b9d04350104bb5fdf33a6471629021fea403a
Submitter: Jenkins
Branch: master

commit a33b9d04350104bb5fdf33a6471629021fea403a
Author: Assaf Muller <email address hidden>
Date: Fri May 13 16:40:00 2016 -0400

    Remove a SGs API test following sync to Tempest

    The dependent patch syncs the test from Neutron to Tempest
    so it can be removed from the Neutron tree. Tests for core
    resources belong to Tempest as described in the bug report.

    Change-Id: I1e4d07fb659851755d5ffe7dd6fbae2d9f8a63fc
    Related-Bug: #1552960
    Depends-On: Ic77561757b5bd9ac61b1e059afdba19a98257126

Changed in neutron:
milestone: newton-1 → newton-2
tags: added: neutron-proactive-backport-potential

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/318495
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/318495
Reason: The tempest test is dead in the water.

Change abandoned by Zhi Chang (<email address hidden>) on branch: master
Review: https://review.openstack.org/316418

Changed in neutron:
milestone: newton-2 → newton-3
Changed in neutron:
milestone: newton-3 → newton-rc1

Not sure why it should be RC1 material. I suggest we postpone this one to O.

Changed in neutron:
milestone: newton-rc1 → ocata-1
Changed in neutron:
importance: High → Low
importance: Low → High
tags: removed: neutron-proactive-backport-potential

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/316269
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Changed in neutron:
milestone: ocata-1 → ocata-2
Changed in neutron:
milestone: ocata-2 → none
assignee: Assaf Muller (amuller) → nobody
status: In Progress → Confirmed

Reviewed: https://review.openstack.org/413374
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2d453e579241ba67e044d4e467bdc3d8d9f5e258
Submitter: Jenkins
Branch: master

commit 2d453e579241ba67e044d4e467bdc3d8d9f5e258
Author: Bruce Tan <email address hidden>
Date: Tue Dec 20 16:16:08 2016 +0000

    Remove test cases that moved to tempest

    Remove the follow three test cases that have been added (or updated if
    the same case already existed) in tempest [1]:
    - test_networks.NetworksTestJSON.test_show_network_fields
    - test_networks.NetworksTestJSON.test_list_networks_fields
    - test_networks.NetworksTestJSON.test_create_update_network_description

    [1] : https://review.openstack.org/#/c/374645/

    Change-Id: I919e11194939bb60be90455ed5e585cd6a894e5e
    Related-Bug: #1552960

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers