Segment-aware scheduling fails for non-admin users

[Andrew Bonney]

This is a follow-up to https://bugs.launchpad.net/nova/+bug/1967314

Having deployed the Nova scheduler configuration for routed provider networks as follows (Xena deployment @ 7df9379d6661233174d49fb7be8eda0828a5e5ca), this was found to resolve issues around scheduling of instances to appropriate hypervisors, but it appears to have surfaced a side effect.

[scheduler]
query_placement_for_routed_network_aggregates = True

When the above configuration is enabled, creation of new instances for admin users works correctly, but for non-admin users against the same networks results in the following error:

285768 ERROR oslo_messaging.rpc.server [req-79ca3cb3-eb52-4755-bba1-4c840c8ae5fc c35a1473225f422c90a6f75b25188bf2 d96f0cd70c6a4adbbbcf993502b264dc - default default] Exception during message handling: K>
285768 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
285768 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
285768 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
285768 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
285768 ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/manager.py", line 154, in select_destinations
285768 ERROR oslo_messaging.rpc.server request_filter.process_reqspec(context, spec_obj)
285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/request_filter.py", line 387, in process_reqspec
285768 ERROR oslo_messaging.rpc.server filter(ctxt, request_spec)
285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/request_filter.py", line 41, in wrapper
285768 ERROR oslo_messaging.rpc.server ran = fn(ctxt, request_spec)
285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/request_filter.py", line 348, in routed_networks_filter
285768 ERROR oslo_messaging.rpc.server aggregates = utils.get_aggregates_for_routed_network(
285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/scheduler/utils.py", line 1406, in get_aggregates_for_routed_network
285768 ERROR oslo_messaging.rpc.server segment_ids = network_api.get_segment_ids_for_network(
285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/network/neutron.py", line 3721, in get_segment_ids_for_network
285768 ERROR oslo_messaging.rpc.server return [subnet['segment_id'] for subnet in subnets
285768 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-24.0.0.0rc1/lib/python3.8/site-packages/nova/network/neutron.py", line 3722, in <listcomp>
285768 ERROR oslo_messaging.rpc.server if subnet['segment_id'] is not None]
285768 ERROR oslo_messaging.rpc.server KeyError: 'segment_id'
285768 ERROR oslo_messaging.rpc.server

It appears that the subnet dictionaries are returned empty from the Neutron client library in this case, causing the KeyError.

As far as I can see, a matching command line request for 'openstack subnet show X' as the same requesting user correctly includes the 'segment_id', but I don't know how similar this code path and the permissions handling is.

I'd be happy to test out other requests or obtain additional logs if useful.

[Balazs Gibizer (balazs-gibizer)]

Thanks for the report! I was able to reproduce the problem on master as well.

The nova code at [1] creates the neutron client based on the request context. If the request is coming from an admin user then the neutron client will be an admin client too. In this case neutron returns the segment_id field in the GET /subnets response.

However if the requester is a non-admin then nova will query neutron with a non-admin client too. And base on my quick testing in a devstack from master neutron indeed does not return the segment_id field for a non-admin user.

So this is valid bug.

The easiest solution would be to change [2] to always create and neutron admin client with:

    get_client(context, admin=True)

I'm setting this to Triaged. Feel free to assign this to yourself and propose a patch fixing this.

[1] https://github.com/openstack/nova/blob/a1f006d799d2294234d381395a9ae9c22a2d80b9/nova/network/neutron.py#L3858-L3867
[2] https://github.com/openstack/nova/blob/a1f006d799d2294234d381395a9ae9c22a2d80b9/nova/network/neutron.py#L3858

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/839361

[Andrew Bonney (andrewbonney)]

Thanks for confirming. I've submitted a patch which I've tested locally and confirmed it does fix the issue.

I'm not sure whether this is something that warrants new tests, and if so how it would be best to go about this. If there's a similar example already I'd be happy to try and work from that, but otherwise it might be best left to someone more experienced.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/839361
Committed: https://opendev.org/openstack/nova/commit/ee32934f34afd8e6df467361e9d71788cd36f6ee
Submitter: "Zuul (22348)"
Branch: master

commit ee32934f34afd8e6df467361e9d71788cd36f6ee
Author: Andrew Bonney <email address hidden>
Date: Tue Apr 26 11:35:38 2022 +0100

    Fix segment-aware scheduling permissions error

    Resolves a bug encountered when setting the Nova scheduler to
    be aware of Neutron routed provider network segments, by using
    'query_placement_for_routed_network_aggregates'.

    Non-admin users attempting to access the 'segment_id' attribute
    of a subnet caused a traceback, resulting in instance creation
    failure.

    This patch ensures the Neutron client is initialised with an
    administrative context no matter what the requesting user's
    permissions are.

    Change-Id: Ic0f25e4d2395560fc2b68f3b469e266ac59abaa2
    Closes-Bug: #1970383

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/840732

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/840732
Committed: https://opendev.org/openstack/nova/commit/60548e804219d91d8c68ab3d74dd0ae956cd33f3
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 60548e804219d91d8c68ab3d74dd0ae956cd33f3
Author: Andrew Bonney <email address hidden>
Date: Tue Apr 26 11:35:38 2022 +0100

    Fix segment-aware scheduling permissions error

    Resolves a bug encountered when setting the Nova scheduler to
    be aware of Neutron routed provider network segments, by using
    'query_placement_for_routed_network_aggregates'.

    Non-admin users attempting to access the 'segment_id' attribute
    of a subnet caused a traceback, resulting in instance creation
    failure.

    This patch ensures the Neutron client is initialised with an
    administrative context no matter what the requesting user's
    permissions are.

    Change-Id: Ic0f25e4d2395560fc2b68f3b469e266ac59abaa2
    Closes-Bug: #1970383
    (cherry picked from commit ee32934f34afd8e6df467361e9d71788cd36f6ee)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 25.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 26.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/884808

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/884830

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova xena-eom release.