neutron

Neutron fails to create network with enforced scopes and new RBAC policies

Bug #1996150 reported by Slawek Kaplonski on 2022-11-10

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	High	Slawek Kaplonski

Bug Description

Error happened in test job run on https://review.opendev.org/c/openstack/tempest/+/614484: https://zuul.opendev.org/t/openstack/build/930366c016de49c4b7c26f69b371411a/log/controller/logs/screen-q-svc.txt#4406

Traceback:

27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers [None req-1e9ba6eb-84f1-4091-bf3a-862d6e6cb127 admin admin] Mechanism driver 'ovn' failed in create_network_postcommit: neutron_lib.exceptions.NetworkNotFound: Network 75ccd449-9526-4dc2-96c3-4f76ac9dcbe2 could not be found.

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers Traceback (most recent call last):

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers File "/opt/stack/neutron/neutron/plugins/ml2/managers.py", line 497, in _call_on_drivers

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context)

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers self._ovn_client.create_network(context._plugin_context, network)

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers self.create_metadata_port(context, network)

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers p_utils.create_port(self._plugin, context, port)

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.8/dist-packages/neutron_lib/plugins/utils.py", line 337, in create_port

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers return core_plugin.create_port(

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers File "/opt/stack/neutron/neutron/common/utils.py", line 704, in inner

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers return f(*args, **kwargs)

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers return f_with_retry(*args, **kwargs,

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers setattr(e, '_RETRY_EXCEEDED', True)

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers self.force_reraise()

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers raise self.value

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers return f(*args, **kwargs)

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers ectxt.value = e.inner_exc

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers self.force_reraise()

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers raise self.value

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers return f(*args, **kwargs)

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers context_reference.session.rollback()

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers self.force_reraise()

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers raise self.value

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers return f(*dup_args, **dup_kwargs)

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1570, in create_port

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers result, mech_context = self._create_port_db(context, port)

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers port_db = self.create_port_db(context, port)

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1461, in create_port_db

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers raise exc.NetworkNotFound(net_id=network_id)

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers neutron_lib.exceptions.NetworkNotFound: Network 75ccd449-9526-4dc2-96c3-4f76ac9dcbe2 could not be found.

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.managers

Nov 04 05:49:27.934555 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR neutron.plugins.ml2.plugin [None req-1e9ba6eb-84f1-4091-bf3a-862d6e6cb127 admin admin] mechanism_manager.create_network_postcommit failed, deleting network '75ccd449-9526-4dc2-96c3-4f76ac9dcbe2': neutron.plugins.ml2.common.exceptions.MechanismDriverError

Nov 04 05:49:27.936779 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: DEBUG neutron_lib.callbacks.manager [None req-1e9ba6eb-84f1-4091-bf3a-862d6e6cb127 admin admin] Publish callbacks ['neutron.plugins.ml2.plugin.External_net_db_mixin._before_network_delete_handler-1373484'] for network (75ccd449-9526-4dc2-96c3-4f76ac9dcbe2), before_delete {{(pid=94286) _notify_loop /usr/local/lib/python3.8/dist-packages/neutron_lib/callbacks/manager.py:176}}

Nov 04 05:49:27.997508 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: ERROR root [None req-1e9ba6eb-84f1-4091-bf3a-862d6e6cb127 admin admin] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1200, in _after_create_network\n self.mechanism_manager.create_network_postcommit(mech_context)\n', ' File "/opt/stack/neutron/neutron/plugins/ml2/managers.py", line 547, in create_network_postcommit\n self._call_on_drivers("create_network_postcommit", context)\n', ' File "/opt/stack/neutron/neutron/plugins/ml2/managers.py", line 513, in _call_on_drivers\n raise ml2_exc.MechanismDriverError(\n', 'neutron.plugins.ml2.common.exceptions.MechanismDriverError\n']: neutron_lib.exceptions.NetworkNotFound: Network 75ccd449-9526-4dc2-96c3-4f76ac9dcbe2 could not be found.

Nov 04 05:49:27.998066 ubuntu-focal-ovh-gra1-0031900066 neutron-server[94286]: INFO neutron.pecan_wsgi.hooks.translation [None req-1e9ba6eb-84f1-4091-bf3a-862d6e6cb127 admin admin] POST failed (client error): The resource could not be found.

Tags:

Revision history for this message

Rodolfo Alonso (rodolfo-alonso-hernandez) wrote on 2022-11-10:

Hi Slawek:

Please confirm https://bugs.launchpad.net/neutron/+bug/1996129 is a duplicate of this bug.

Thanks!

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-11-10: Fix proposed to neutron-lib (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/864213

Changed in neutron:
status:	Confirmed → In Progress

Revision history for this message

Slawek Kaplonski (slaweq) wrote on 2022-11-10:

@Rodolfo: yes, it is. I marked it as such :)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-11-16: Fix merged to neutron-lib (master)

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/864213
Committed: https://opendev.org/openstack/neutron-lib/commit/91759b17ea5d4d6563596612208e6014dee2ed88
Submitter: "Zuul (22348)"
Branch: master

commit 91759b17ea5d4d6563596612208e6014dee2ed88
Author: Slawek Kaplonski <email address hidden>
Date: Thu Nov 10 16:05:11 2022 +0100

Don't check context.system_scope to check project scope

    During first attempt of the Secure RBAC implementation [1] to
    function model_query_scope_is_project() there was added check
    which was checking if context.system_scope is "all" in case when
    scope enforcement was enabled. It was like that because that meant
    that it is SYSTEM_* persona context (e.g. SYSTEM_ADMIN).
    This is not needed now as later it was agreed to have only one ADMIN
    user which will still behave like old, "legacy" ADMIN user.

[1] https://review.opendev.org/c/openstack/neutron-lib/+/781075

Closes-bug: #1996150
Change-Id: If3a97c4d3a0f4cb6b4d06434f74cbe9d933a07a4

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-11-17: Fix included in openstack/neutron-lib 3.2.0

This issue was fixed in the openstack/neutron-lib 3.2.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-20: Fix proposed to neutron-lib (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron-lib/+/874394

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-20: Fix proposed to neutron-lib (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron-lib/+/874395

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-20: Fix proposed to neutron-lib (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron-lib/+/874396

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-20: Fix proposed to neutron-lib (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron-lib/+/874397

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-03-01: Fix merged to neutron-lib (stable/zed)

#10

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/874394
Committed: https://opendev.org/openstack/neutron-lib/commit/c40779d5522ed07c14a56f77ab2ad476537eff0f
Submitter: "Zuul (22348)"
Branch: stable/zed

commit c40779d5522ed07c14a56f77ab2ad476537eff0f
Author: Slawek Kaplonski <email address hidden>
Date: Thu Nov 10 16:05:11 2022 +0100

Don't check context.system_scope to check project scope

[1] https://review.opendev.org/c/openstack/neutron-lib/+/781075

    Closes-bug: #1996150
    Change-Id: If3a97c4d3a0f4cb6b4d06434f74cbe9d933a07a4
    (cherry picked from commit 91759b17ea5d4d6563596612208e6014dee2ed88)

tags:

added: in-stable-zed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-03-14: Fix included in openstack/neutron-lib 3.1.1

#11

This issue was fixed in the openstack/neutron-lib 3.1.1 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-03-30: Fix merged to neutron-lib (stable/xena)

#12

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/874396
Committed: https://opendev.org/openstack/neutron-lib/commit/150a1c96b93be52a589def0f96205166aaf58384
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 150a1c96b93be52a589def0f96205166aaf58384
Author: Slawek Kaplonski <email address hidden>
Date: Thu Nov 10 16:05:11 2022 +0100

Don't check context.system_scope to check project scope

[1] https://review.opendev.org/c/openstack/neutron-lib/+/781075

    Conflicts:
        neutron_lib/db/utils.py
        neutron_lib/tests/unit/db/test_utils.py

    Closes-bug: #1996150
    Change-Id: If3a97c4d3a0f4cb6b4d06434f74cbe9d933a07a4
    (cherry picked from commit 91759b17ea5d4d6563596612208e6014dee2ed88)

tags:	added: in-stable-xena
tags:	added: in-stable-yoga

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-03-30: Fix merged to neutron-lib (stable/yoga)

#13

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/874395
Committed: https://opendev.org/openstack/neutron-lib/commit/7f7e4aeef082e82c882d99c5964a89c3f646120a
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 7f7e4aeef082e82c882d99c5964a89c3f646120a
Author: Slawek Kaplonski <email address hidden>
Date: Thu Nov 10 16:05:11 2022 +0100

Don't check context.system_scope to check project scope

[1] https://review.opendev.org/c/openstack/neutron-lib/+/781075

    Conflicts:
        neutron_lib/db/utils.py
        neutron_lib/tests/unit/db/test_utils.py

    Closes-bug: #1996150
    Change-Id: If3a97c4d3a0f4cb6b4d06434f74cbe9d933a07a4
    (cherry picked from commit 91759b17ea5d4d6563596612208e6014dee2ed88)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-04-04: Fix merged to neutron-lib (stable/wallaby)

#14

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/874397
Committed: https://opendev.org/openstack/neutron-lib/commit/0dadfca1ffe72f9526617c596cf76ee6f7a3fd78
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 0dadfca1ffe72f9526617c596cf76ee6f7a3fd78
Author: Slawek Kaplonski <email address hidden>
Date: Thu Nov 10 16:05:11 2022 +0100

Don't check context.system_scope to check project scope

[1] https://review.opendev.org/c/openstack/neutron-lib/+/781075

    Conflicts:
        neutron_lib/db/utils.py
        neutron_lib/tests/unit/db/test_utils.py

    Closes-bug: #1996150
    Change-Id: If3a97c4d3a0f4cb6b4d06434f74cbe9d933a07a4
    (cherry picked from commit 91759b17ea5d4d6563596612208e6014dee2ed88)