when enabled enforce_new_defaults,create server failed

[luffy Wong]

Description
===========
When enabled enforce_new_defaults in nova.conf. system scope admin failed to create a server.It occure an error in neutron log(controller node).

2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova Traceback (most recent call last):
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova File "/usr/lib/python3.6/site-packages/neutron/notifiers/nova.py", line 266, in send_events
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova batched_events)
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova File "/usr/lib/python3.6/site-packages/novaclient/v2/server_external_events.py", line 39, in create
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova return_raw=True)
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova File "/usr/lib/python3.6/site-packages/novaclient/base.py", line 363, in _create
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova resp, body = self.api.client.post(url, body=body)
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 401, in post
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova return self.request(url, 'POST', **kwargs)
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova File "/usr/lib/python3.6/site-packages/novaclient/client.py", line 78, in request
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova raise exceptions.from_response(resp, body, url, method)
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova novaclient.exceptions.Forbidden: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed. (HTTP 403) (Request-ID: req-928afad8-32b9-420
8-8e5e-e2bc9061a56a)

Steps to reproduce
==================
1、enabled enforce_new_defaults in nova.conf and restart nova
2、empty policy.yaml >/etc/nova/policy.yaml
3、use admin(system scope) to create a server
4、create server failed
5、disabled enforce_new_defaults ,admin could create server successfully.

Expected result
===============
admin user create the server successfully.

Actual result
=============
The status of server stuck in "BUILD" ,after 5 mimutes,it become "error".

It occure an error in neutron log(controller node).

2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova Traceback (most recent call last):
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova File "/usr/lib/python3.6/site-packages/neutron/notifiers/nova.py", line 266, in send_events
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova batched_events)
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova File "/usr/lib/python3.6/site-packages/novaclient/v2/server_external_events.py", line 39, in create
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova return_raw=True)
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova File "/usr/lib/python3.6/site-packages/novaclient/base.py", line 363, in _create
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova resp, body = self.api.client.post(url, body=body)
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 401, in post
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova return self.request(url, 'POST', **kwargs)
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova File "/usr/lib/python3.6/site-packages/novaclient/client.py", line 78, in request
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova raise exceptions.from_response(resp, body, url, method)
2022-04-14 09:36:29.743 876530 ERROR neutron.notifiers.nova novaclient.exceptions.Forbidden: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed. (HTTP 403) (Request-ID: req-928afad8-32b9-420
8-8e5e-e2bc9061a56a)

Environment
===========
OS release centos8.2
openstack victoria
nova 22.2.2
neutron 17.2
keystone 18.0

[luffy Wong (luffy963)]

Accroding to the default policy,system scope admin shoud have top priviledge.But in my case,admin
is not allowed to perform os_compute_api:os-server-external-events:create

[Sylvain Bauza (sylvain-bauza)]

Sounds legit, but we need more confirmation. Will look at it later.

[Ghanshyam Mann (ghanshyammann)]

This is a valid bug and we fixed it in Yoga (nova 25.0.0). Adding details below:

Query:
------

- you mentioned that you are able to create the server with system admin? but policy (with enforce_new_defaults=true) does not allow system admin to create server and even system admin does not have project_id in the token for who server needs to be created. I am sure you might have used project admin token in this case. Please confirm.

Bug root cause:
---------------

nova 22.2.2 is victoria version where we had the new RBAC with old direction (we moved to new design of new RBAC in Yoga[1]). This is exactly the issue we solved in OpenStack Yoga (nova 25.0.0).

In Ussuri, we tightly coupled the new default with scope and due to that new defaults were not usable (like this issue happened).

* nova 22.2.2:
** Server create default policy in nova 22.2.2 was PROJECT_MEMBER (so project admin can create too)
- https://github.com/openstack/nova/blob/stable/victoria/nova/policies/servers.py#L168-L176

** 'os_compute_api:os-server-external-events:create' default policy was SYSTEM_ADMIN, which is 'role:admin and system_scope:all'. Here you see "system_scope:all" hard string made new defaults tighly coupled with scope type.
- https://github.com/openstack/nova/blob/stable/victoria/nova/policies/server_external_events.py#L27

With that when neutron called nova server external event API with their service token (project admin might be if configured) it fail as nova API default was expecting System admin.

How to solve:
-------------
We have a few of the options/workaround to solve this:
1. nova 25.0.0: it is fixed in the 25.0.0 version. There we have decoupled the scope type from defaults and new defaults for server external create policy is 'Admin' (if the scope is disabled then any admin system or project). With that neutron will be able to call this API with admin token.
- https://github.com/openstack/nova/blob/a1f006d799d2294234d381395a9ae9c22a2d80b9/nova/policies/server_external_events.py#L35

2. override the 'os_compute_api:os-server-external-events:create' policy in your policy file to admin, like below:
"os_compute_api:os-server-external-events:create": "role:admin"

Please check if the above things work for you? NOTE: we still have not moved our integration testing to new defaults and scope which should give more clarity if there is any issue existing like this. This is plan for this cycle to test dependent services like nova, and neutron with new default and see if all work fine.

[1] https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html

[Sylvain Bauza (sylvain-bauza)]

Marking the bug as WONTFIX as we fixed the root cause in the Yoga release.