neutron_tempest_plugin.api.admin.test_shared_network_extension.RBACSharedNetworksTest, rbac policy in use across tenants.

[wes hayutin]

neutron_tempest_plugin.api.admin.test_shared_network_extension.RBACSharedNetworksTest failure

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/6cec620/tempest.html.gz

Details: {u'message': u'RBAC policy on object 3cfbd0a7-84f2-4e3f-917e-bf51b5995e20 cannot be removed because other objects depend on it.\nDetails: Callback neutron.plugins.ml2.plugin.Ml2Plugin.validate_network_rbac_policy_change--9223372036850840529 failed with "Unable to reconfigure sharing settings for network 3cfbd0a7-84f2-4e3f-917e-bf51b5995e20. Multiple tenants are using it.",Callback neutron.services.network_ip_availability.plugin.NetworkIPAvailabilityPlugin.validate_network_rbac_policy_change--9223372036853400817 failed with "Unable to reconfigure sharing settings for network 3cfbd0a7-84f2-4e3f-917e-bf51b5995e20. Multiple tenants are using it.",Callback neutron.services.network_ip_availability.plugin.NetworkIPAvailabilityPlugin.validate_network_rbac_policy_change--9223372036853463713 failed with "Unable to reconfigure sharing settings for network 3cfbd0a7-84f2-4e3f-917e-bf51b5995e20. Multiple tenants are using it."', u'type': u'RbacPolicyInUse', u'detail': u''}

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/549516

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/549516
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=4a3e4bc6e942f7cd37c413db98f9ef7aa65cc300
Submitter: Zuul
Branch: master

commit 4a3e4bc6e942f7cd37c413db98f9ef7aa65cc300
Author: Wes Hayutin <email address hidden>
Date: Sun Mar 4 03:09:15 2018 -0500

    two new tempest failures on master

    Related-Bug: #1753209
    Related-Bug: #1753211
    Change-Id: Ib9952abc0a2571663d51ed664bb2ef1bcd9e7def

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/549837

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/549837
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=e1d432115cd424ab6a4be474a7a1e1922c4553ed
Submitter: Zuul
Branch: master

commit e1d432115cd424ab6a4be474a7a1e1922c4553ed
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Mon Mar 5 20:00:22 2018 +0200

    Stop skipping volume encryption test

    Should work with dependent patch.

    Change-Id: I5f7b5e4e73dbfe4d6b819472452f478bbb206975
    Depends-On: Ibe4e5023b8773ee70ee914118628acc5c149f379
    Related-Bug: #1753209

[Ihar Hrachyshka (ihar-hrachyshka)]

So the reason of failure here is that one test case uses a network that was temporarily shared with all tenants while running a RBAC test case. When the network is for a short span of time shared with everyone, the other test case - VolumesSnapshotTestJSON:test_snapshot_create_delete_with_volume_in_use - creates an instance with the following request body:

    Body: {"server": {"flavorRef": "e443576a-50cd-4023-88e0-574b1ec1726e", "name": "tempest-VolumesSnapshotTestJSON-instance-1302082485", "imageRef": "9a98ccb7-cd30-43da-85a9-5cfd8126c5ae"}}
which, as you can see, doesn't specify network to use, so nova then apparently picks one of available networks to boot the instance on, and it happens to be the network that was momentarily shared in RBAC test case.

I don't think that the Volumes test case should rely on nova to pick a network for them, and instead pre-create a tenant network for this specific test case and then boot the instance on it. By doing so, we will avoid the race condition between those test cases b/c Volumes test case won't touch the shared network and create VIF ports on it.

This is probably a bug for cinder folks to solve since the offending test case is in VolumesSnapshot class.

One thing to mull over on neutron side though is whether the rbac test case, as written, could reduce its impact. One thing to consider is that the network that is momentarily shared with everyone will pop up for all tenants, incl. those unaware of tempest running in background. Since some operators execute tempest periodically against their cloud, they probably don't want their customers to get random networks popping up for a brief moment. Though the RBAC test case explicitly validates the wildcard RBAC behavior so not sure if we can easily get rid of it w/o loosing some coverage.

How does tempest core team look at such cases? Do we forbid any impact on other cloud users? If so, we could probably shorten the neutron api case to avoid it, even if it means a particular use case won't be covered with tempest.

[Ihar Hrachyshka (ihar-hrachyshka)]

Added neutron to affected projects since we may need to modify the api test case that is the root cause of the failure; also added tempest to clarify what's the recommendation for this type of test cases affecting other users.

[Ihar Hrachyshka (ihar-hrachyshka)]

My belief is that all test classes should use resources explicitly created for them; so in case of volume tests that are triggering a race in RBAC test class, they would boot instances with port/net specified instead of allowing nova to pick one.

Regardless, what neutron test class does is also wrong because tempest should not affect other tenants, and it does in this case by potentially breaking boot instance flow for them.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/553499

[Akihiro Motoki (amotoki)]

https://review.openstack.org/#/c/553499/ and https://review.openstack.org/#/c/550987/ move a test which shares a network via neutron RBAC mechanism. Do we also need to clean up all shared network case like a network with shared=True attribute? Do we also need to clean up 'shared' cases for other resources (like subnet pool, qos policy and so on)?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/553499
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6c772de103e469ad17cdbc6a86ed15655aa6b9cc
Submitter: Zuul
Branch: master

commit 6c772de103e469ad17cdbc6a86ed15655aa6b9cc
Author: Ihar Hrachyshka <email address hidden>
Date: Thu Mar 15 10:13:14 2018 -0700

    tests: port test_port_presence_prevents_network_rbac_policy_deletion

    The test was originally implemented in this form as a tempest api test
    case, but there is a problem with it being part of tempest suite,
    because the suite should be safe to execute on a running cloud and
    shouldn't affect other tenants. But sharing a network with everyone (*)
    briefly impacts other tenants because they see a new shared network
    popping up for a second in their net-list. It is also an issue because
    if we are unlucky enough, some other tenant may start an instance
    without specifying a particular port or network to use, in which case
    there is a chance that nova allocates a port in this shared network and
    attach it to the instance. When the rbac test case then tries to delete
    the policy and the network, it fails because it's still in use.

    The solution is to stop testing this scenario in tempest suite, instead
    moving it into unit test suite.

    A follow-up patch in neutron-tempest-plugin will clean up the bad test
    case.

    Change-Id: I199f639c95e8ae884ede46404370d7b64da3b309
    Needed-By: Iba89a53b2715cf3a9c7485f2089f27d547fea308
    Related-Bug: #1753209

[Ihar Hrachyshka (ihar-hrachyshka)]

@Akihiro, probably yes. Probably will require a separate bug report.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: master
Review: https://review.openstack.org/553501

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/571675

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/571676

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/571677

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/571681

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/571682

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/571675
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d9de2233d5edb7f978ed72e2ea8c4dc42a399e3d
Submitter: Zuul
Branch: stable/queens

commit d9de2233d5edb7f978ed72e2ea8c4dc42a399e3d
Author: Ihar Hrachyshka <email address hidden>
Date: Thu Mar 15 10:13:14 2018 -0700

    tests: port test_port_presence_prevents_network_rbac_policy_deletion

    The test was originally implemented in this form as a tempest api test
    case, but there is a problem with it being part of tempest suite,
    because the suite should be safe to execute on a running cloud and
    shouldn't affect other tenants. But sharing a network with everyone (*)
    briefly impacts other tenants because they see a new shared network
    popping up for a second in their net-list. It is also an issue because
    if we are unlucky enough, some other tenant may start an instance
    without specifying a particular port or network to use, in which case
    there is a chance that nova allocates a port in this shared network and
    attach it to the instance. When the rbac test case then tries to delete
    the policy and the network, it fails because it's still in use.

    The solution is to stop testing this scenario in tempest suite, instead
    moving it into unit test suite.

    A follow-up patch in neutron-tempest-plugin will clean up the bad test
    case.

    Change-Id: I199f639c95e8ae884ede46404370d7b64da3b309
    Needed-By: Iba89a53b2715cf3a9c7485f2089f27d547fea308
    Related-Bug: #1753209
    (cherry picked from commit 6c772de103e469ad17cdbc6a86ed15655aa6b9cc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/571676
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=48c5c84ee0327e2d6f5239a2e61d3787395635e1
Submitter: Zuul
Branch: stable/pike

commit 48c5c84ee0327e2d6f5239a2e61d3787395635e1
Author: Ihar Hrachyshka <email address hidden>
Date: Thu Mar 15 10:13:14 2018 -0700

    tests: port test_port_presence_prevents_network_rbac_policy_deletion

    The test was originally implemented in this form as a tempest api test
    case, but there is a problem with it being part of tempest suite,
    because the suite should be safe to execute on a running cloud and
    shouldn't affect other tenants. But sharing a network with everyone (*)
    briefly impacts other tenants because they see a new shared network
    popping up for a second in their net-list. It is also an issue because
    if we are unlucky enough, some other tenant may start an instance
    without specifying a particular port or network to use, in which case
    there is a chance that nova allocates a port in this shared network and
    attach it to the instance. When the rbac test case then tries to delete
    the policy and the network, it fails because it's still in use.

    The solution is to stop testing this scenario in tempest suite, instead
    moving it into unit test suite.

    A follow-up patch in neutron-tempest-plugin will clean up the bad test
    case.

    Change-Id: I199f639c95e8ae884ede46404370d7b64da3b309
    Needed-By: Iba89a53b2715cf3a9c7485f2089f27d547fea308
    Related-Bug: #1753209
    (cherry picked from commit 6c772de103e469ad17cdbc6a86ed15655aa6b9cc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/571677
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9f72d8387ecb2b81b9e7969004ea247a948de6f9
Submitter: Zuul
Branch: stable/ocata

commit 9f72d8387ecb2b81b9e7969004ea247a948de6f9
Author: Ihar Hrachyshka <email address hidden>
Date: Thu Mar 15 10:13:14 2018 -0700

    tests: port test_port_presence_prevents_network_rbac_policy_deletion

    The test was originally implemented in this form as a tempest api test
    case, but there is a problem with it being part of tempest suite,
    because the suite should be safe to execute on a running cloud and
    shouldn't affect other tenants. But sharing a network with everyone (*)
    briefly impacts other tenants because they see a new shared network
    popping up for a second in their net-list. It is also an issue because
    if we are unlucky enough, some other tenant may start an instance
    without specifying a particular port or network to use, in which case
    there is a chance that nova allocates a port in this shared network and
    attach it to the instance. When the rbac test case then tries to delete
    the policy and the network, it fails because it's still in use.

    The solution is to stop testing this scenario in tempest suite, instead
    moving it into unit test suite.

    A follow-up patch in neutron-tempest-plugin will clean up the bad test
    case.

    Change-Id: I199f639c95e8ae884ede46404370d7b64da3b309
    Needed-By: Iba89a53b2715cf3a9c7485f2089f27d547fea308
    Related-Bug: #1753209
    (cherry picked from commit 6c772de103e469ad17cdbc6a86ed15655aa6b9cc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/571682
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=19c367d5795e4f6c1f9a9d421e22a4cf1e5b15a2
Submitter: Zuul
Branch: stable/ocata

commit 19c367d5795e4f6c1f9a9d421e22a4cf1e5b15a2
Author: Slawek Kaplonski <email address hidden>
Date: Fri Jun 1 11:54:28 2018 +0200

    Don't share a network with all tenants

    Other tenants may appreciate random networks not popping up for
    miliseconds in their net-list; and other test cases executed in parallel
    may also benefit, especially when they e.g. attempt to validate net-list
    result; or boot an instance w/o specifying a port or network to use.

    There is still more work on tempest tests side.

    Original patch was for neutron-tempest-plugin:
    https://review.openstack.org/#/c/550987/
    and this is only backport to stable/Pike branch which still
    uses tempest tests from neutron repo.

    Depends-On: https://review.openstack.org/#/c/571677/
    Change-Id: I88eaa1bb65482557912c2d304800d782e730a6b6
    Closes-Bug: #1753209

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/571681
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=221df78612820fcbd3455b4b1092bdc8160c56cb
Submitter: Zuul
Branch: stable/pike

commit 221df78612820fcbd3455b4b1092bdc8160c56cb
Author: Slawek Kaplonski <email address hidden>
Date: Fri Jun 1 11:54:28 2018 +0200

    Don't share a network with all tenants

    Other tenants may appreciate random networks not popping up for
    miliseconds in their net-list; and other test cases executed in parallel
    may also benefit, especially when they e.g. attempt to validate net-list
    result; or boot an instance w/o specifying a port or network to use.

    There is still more work on tempest tests side.

    Original patch was for neutron-tempest-plugin:
    https://review.openstack.org/#/c/550987/
    and this is only backport to stable/Pike branch which still
    uses tempest tests from neutron repo.

    Depends-On: https://review.openstack.org/#/c/571676/
    Change-Id: I88eaa1bb65482557912c2d304800d782e730a6b6
    Closes-Bug: #1753209

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 10.0.7 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Federico Ressi (<email address hidden>) on branch: master
Review: https://review.openstack.org/604718

[Federico Ressi (fressi-redhat)]

I personally think the solution of this problem is a lack in base test case classes. When a test case doesn't specify a network to use, it should create one using some parameters specified in tempest.conf. Using a shared network by default (implicitly, without any notice, and without choosing it) makes testing shared networks near to impossible in the contest of running test cases in parallel and therefore should be forbidden and limited to very specific test cases where shared network is explicitly pick up by test case.

[chandan kumar (chkumar246)]

Based on master skip list update: https://tree.taiga.io/project/tripleo-ci-board/task/1088?kanban-status=1447275
Below is the tests are passing:
 "tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON": [
        "tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_snapshot_create_delete_with_volume_in_use",
        "tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_snapshot_create_get_list_update_delete",
        "tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_snapshot_create_offline_delete_online"
    ],

Currently there 5 tests runned under this group, we are moving rest of 2 tests under skip list.
* tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_volume_from_snapshot_no_size
* tempest.api.volume.test_volumes_snapshots.VolumesSnapshotTestJSON.test_volume_from_snapshot
*

[chandan kumar (chkumar246)]

Logs are here https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset021-master/7ab6f45/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-05-28_04_21_53

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/661694

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/661694
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=85bba9f24a79da8069d5d0b4a7515dd62893e202
Submitter: Zuul
Branch: master

commit 85bba9f24a79da8069d5d0b4a7515dd62893e202
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Tue May 28 13:29:09 2019 +0530

    Update skip list for master

    Bugs Fixed:
    https://bugs.launchpad.net/tripleo/+bug/1806075
    https://bugs.launchpad.net/keystone/+bug/1660603
    https://bugs.launchpad.net/tripleo/+bug/1753211
    https://bugs.launchpad.net/tripleo/+bug/1810325

    https://tree.taiga.io/project/tripleo-ci-board/task/1088

    Related-Bug: #1753209

    Change-Id: I291064cc6f7ebb40d5704ee64489dbf77a68475f
    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/698724

[OpenStack Infra (hudson-openstack)]

Change abandoned by Soniya Murlidhar Vyas (<email address hidden>) on branch: master
Review: https://review.opendev.org/698724
Reason: This patch is already existing:- https://review.opendev.org/#/c/696387

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/698887

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696387
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=361cc61be41fbce818abfd61f49c910fa7a77e0f
Submitter: Zuul
Branch: master

commit 361cc61be41fbce818abfd61f49c910fa7a77e0f
Author: Soniya Vyas <email address hidden>
Date: Thu Dec 12 19:44:50 2019 +0530

    [Train] Remove passing tests from skiplist

    Tests which are passing are being removed from the skiplist
    with this patch.

    Along with that, class
    neutron_tempest_plugin.api.test_networks has maximum no. of
    tests passing, so just kept failing test and other all tests
    are removed.

    Related-Bug: #1753209
    Related-Bug: #1793482

    Co-Authored-by: Soniya Vyas<email address hidden>
    Co-Authored-by: Suraj Patil<email address hidden>
    Change-Id: I6a18555ba74018495daee5e5e6479bd2bd0427fd

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/699394
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=f1b00f047569913eb33fe67c6cf714ca0e8b4373
Submitter: Zuul
Branch: master

commit f1b00f047569913eb33fe67c6cf714ca0e8b4373
Author: Soniya Vyas <email address hidden>
Date: Fri Jan 3 15:40:56 2020 +0530

    [Rocky] Removed passing tests from skiplist

    Class neutron_tempest_plugin.scenario has maximum
    number of tests passed. Hence, it is removed and
    only failing testsis kept.

    In addition to above, added two more tests from
    class 'TestSecurityGroupsBasicOps' and a test from
    class 'ServersOnMultiNodesTest'

    Related-bug: #1737940
    Related-bug: #1753209
    Related-bug: #1843259
    Related-bug: #1793482
    Related-bug: #1831223
    Related-bug: #1857365

    Signed-off by: Soniya Vyas<email address hidden>
    Change-Id: I8239bb694187d7f912163742e836a7362cdb1483

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/699275
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=5d393c80b8d525683f525c9d6d8a8a5a5798dbb5
Submitter: Zuul
Branch: master

commit 5d393c80b8d525683f525c9d6d8a8a5a5798dbb5
Author: Soniya Vyas <email address hidden>
Date: Tue Dec 17 00:20:06 2019 +0530

    [Stein] Remove passing tests from skiplist

    Tests which are passing are being removed from the skiplist
    with this patch.

    Along with that, class
    neutron_tempest_plugin.api.test_networks has maximum no. of
    tests passing, so just kept failing test and other all tests
    are removed.

    Related-Bug: #1753209
    Related-Bug: #1793482

    Signed-off by: Soniya Vyas<email address hidden>
    Change-Id: Iaf18f62f268b72148477a5416ac3b75a34344c75

[OpenStack Infra (hudson-openstack)]

Change abandoned by wes hayutin (<email address hidden>) on branch: master
Review: https://review.opendev.org/698887

[Martin Kopec (mkopec)]

Seems like this got fixed on neutron and neutron-tempest-plugin side ... the LP is marked Fix Committed for tempest party, however there hasn't been any commit merged/committed in tempest regarding this (or at least i haven't found any), therefore I'm changing the state to Invalid .. If my assumptions are not correct and the bug is still causing troubles, please feel free to raise it and change the status of the LP accordingly.