Generated policy.json in Ussuri is broken by default

[Thomas Goirand]

Looks like the generated policy.json is broken by default and can't be used by operators as-is, as it doesn't include the deprecated options which are unfortunately needed for it to work.

With the default policy.json as generated by the nova namespace, the admin user can't even do simple things like:

- openstack flavor create
- openstack hypervisor list

and probably many more...

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/723645

[Ghanshyam Mann (ghanshyammann)]

We discussed this on IRC, there is no clear way to differentiate whether re-generated policy file is 1. A way to move to new defaults or 2. expecting the old defaults also work with new defaults.

We definitely need more documents on re-generated of policy file or usage if policy file when the operator still want to use policy defaults. Basically this bug: https://bugs.launchpad.net/oslo.policy/+bug/1853170

What happened here:

During policy new defaults work, old defaults are kept supported as a deprecated rule along with new defaults. For deployment relying on policy defaults, it will keep working because of old defaults are still supported as a deprecated rule. If the deployment has overridden the policy rule then defaults are not into picture and token are checked as per the overriden rule value.

In this case, policy.json is re-generated with 'oslopolicy-sample-generator' which does not include the deprecated rule in the generated file. This means rule is overridden with the new-default value ("system_admin_api": "role:admin and system_scope:all") only. Policy checks the existing token with these new values in policy.json and fail. This is the exact same way when the operator wants to switch to new defaults by overriding the rule with new defaults in policy.json so that deprecated old rule gets disappear (though we introduced a new way to achieve the new defaults behaviour by enabling a flag 'oslo_policy.enforce_new_defaults' in Ussuri).

Ont thing we can do is to mention this in release notes and expect operators to either not override the rule they want to reply on defaults or update their token to work as per overriden value.

http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2020-04-27.log.html#t2020-04-27T13:46:37

[Balazs Gibizer (balazs-gibizer)]

My understanding of the current behavior:
* Tokens get project scope by default from keystone.
* Project scoped policies allow the request with the default project scoped token even if enforce_scope=False is in the nova.conf
* System scoped policies reject the request with the default project scoped token regardless of enforce_scope config option.
* The in-code default policies in nova code allows the request with the default project scoped token regardless the enforce_scope=False is in the nova.conf
* oslopolicy-sample-generator generates only the new scoped policy defaults

My understanding that we need to support the following use cases:
* Upgrading with an empty or non empty policy.json with default enforce_scope=False
=> Works today
* Freshly installing with empty policy.json with default enforce_scope=False
=> Works today
* Freshly installing with generated policy.json with default enforce_scope=False
=> Does not work today as the new generated policies only compatible with enforce_scope=True

What I see as possible way forward top of the already proposed reno warning: Change the oslopolicy-sample-generator to generate the same defaults as what is in nova code. This helps with the use case that doesn't work today. Add an upgrade check to Victoria that will warn the admin if he/she uses a policy file that has deprecated policies. This will help us in W to remove the deprecated rules and default enforce_scope to True.

Would this approach work?
Can we do the generator change for RC2 or it is only possible after Ussuri GA?

[Thomas Goirand (thomas-goirand)]

I very much agree with this approach. Having the generated policy.json match what's currently the Nova's reality is exactly what I, as an operator, expect. This is also the only sane way to inspect what the defaults are. Thanks for writing what's above.

[Ghanshyam Mann (ghanshyammann)]

That is correct what you wrote above. But tricky part here is generated policy.json from 'oslopolicy-sample-generator' tool without deprecated rules is right thing or wrong. Because it can be seen as one of the valid usgae for deployer who want to switch to new defaults. The only way for them till ussuri(till we introduced new flag in oslo) is to overwrite the policy file with new default (which is what 'oslopolicy-sample-generator 'generate).

If we add arg option in 'oslopolicy-sample-generator ' to add a deprecated rule (say --add-deprecated-rules) that also should not be default and deployer need to change the usage of that tool to pass the new arg. Opinion ? also other challenge is we need to check with Oslo team if that can be done now for ussuri. adding oslo also as an affected project.

Also, what we are missing here is this bug actually - https://bugs.launchpad.net/oslo.policy/+bug/1853170

[Balazs Gibizer (balazs-gibizer)]

From the IRC discussion I feel that changing the oslopolicy-sample-generator is not feasible in Ussuri GA timeframe [1] and might seemed even as a step backward.

We don't see any good option right now that supports all the use cases. So we will document the problem with broken use case and we added a topic for the PTG to further discuss the possibilities[2].

[1]http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2020-04-28.log.html#t2020-04-28T14:21:05
[2]https://etherpad.opendev.org/p/nova-victoria-ptg

[Sylvain Bauza (sylvain-bauza)]

We at least need to try to provide https://review.opendev.org/#/c/723645/ for a next RC.

[Ghanshyam Mann (ghanshyammann)]

uploading the result for oslo generated yaml format file works fine and json format file end up with 403 (this bug) .

--format yaml

openstack@openstack-VirtualBox:/opt/stack/nova$ oslopolicy-sample-generator --namespace nova --format yaml --output-file /etc/nova/policy.yaml
openstack@openstack-VirtualBox:/opt/stack/nova$ vi /etc/nova/nova.conf
openstack@openstack-VirtualBox:/opt/stack/nova$ grep policy /etc/nova/nova.conf[oslo_policy]
policy_file=/etc/nova/policy.yaml
openstack@openstack-VirtualBox:/opt/stack/nova$ nova hypervisor-list
+--------------------------------------+----------------------+-------+---------+
| ID | Hypervisor hostname | State | Status |
+--------------------------------------+----------------------+-------+---------+
| 5b3388dc-5149-45dd-ba88-2c4ff9ec9f12 | openstack-VirtualBox | up | enabled |
+--------------------------------------+----------------------+-------+---------+
openstack@openstack-VirtualBox:/opt/stack/nova$

--format json

openstack@openstack-VirtualBox:/opt/stack/nova$ oslopolicy-sample-generator --namespace nova --format json --output-file /etc/nova/policy.json
openstack@openstack-VirtualBox:/opt/stack/nova$ grep policy /etc/nova/nova.conf[oslo_policy]
policy_file=/etc/nova/policy.json
openstack@openstack-VirtualBox:/opt/stack/nova$ nova hypervisor-list
ERROR (Forbidden): Policy doesn't allow os_compute_api:os-hypervisors:list to be performed. (HTTP 403) (Request-ID: req-7df8af5d-5ac8-4827-a5c5-62187807e019)
openstack@openstack-VirtualBox:/opt/stack/nova$

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/723645
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d4af91f349b9d3fe4f840878906032a62b589324
Submitter: Zuul
Branch: master

commit d4af91f349b9d3fe4f840878906032a62b589324
Author: Ghanshyam Mann <email address hidden>
Date: Mon Apr 27 13:46:07 2020 -0500

    Add nova-status upgrade check and reno for policy new defaults

    There are cases where policy file is re-generated freshly
    and end up having the new defaults only but expectation is that
    old deprecated rule keep working.

    If a rule is present in policy file then, that has priority over
    its defaults so either rules should not be present in policy file
    or users need to update their token to match the overridden rule
    permission.

    This issue was always present when any policy defaults were changed
    with old defaults being supported as deprecated. This is we have
    changed all the policy for new defaults so it came up as broken case.

    Adding nova-status upgrade check also to detect such policy file.

    Related-Bug: #1875418

    Change-Id: Id9cd65877e53577bff22e408ca07bbeec4407f6e

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/725146

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/725146
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=dd3cc59ccf8c963e078359d4b27dacf7d54a14ee
Submitter: Zuul
Branch: stable/ussuri

commit dd3cc59ccf8c963e078359d4b27dacf7d54a14ee
Author: Ghanshyam Mann <email address hidden>
Date: Mon Apr 27 13:46:07 2020 -0500

    Add nova-status upgrade check and reno for policy new defaults

    There are cases where policy file is re-generated freshly
    and end up having the new defaults only but expectation is that
    old deprecated rule keep working.

    If a rule is present in policy file then, that has priority over
    its defaults so either rules should not be present in policy file
    or users need to update their token to match the overridden rule
    permission.

    This issue was always present when any policy defaults were changed
    with old defaults being supported as deprecated. This is we have
    changed all the policy for new defaults so it came up as broken case.

    Adding nova-status upgrade check also to detect such policy file.

    Related-Bug: #1875418

    Change-Id: Id9cd65877e53577bff22e408ca07bbeec4407f6e
    (cherry picked from commit d4af91f349b9d3fe4f840878906032a62b589324)

[Balazs Gibizer (balazs-gibizer)]

Removed the ussuri-rc-potential tag as the documentation and upgrade check part of the fix is merged to stable/ussuri and released in RC2.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/748059
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fe545dbe5fb434d5fce2d4d0f24c6c4a6bdd7d21
Submitter: Zuul
Branch: master

commit fe545dbe5fb434d5fce2d4d0f24c6c4a6bdd7d21
Author: Ghanshyam Mann <email address hidden>
Date: Thu Sep 3 14:25:43 2020 -0500

    Migrate default policy file from JSON to YAML

    Default value of 'CONF.oslo_policy.policy_file' config option
    has been changed from 'policy.json' to 'policy.yaml'. If new default
    file 'policy.yaml' does not exist but old default 'policy.json' exist
    then fallback to use old default file.

    An upgrade checks is added to check the policy_file format and
    fail upgrade checks if it is JSON formatted.

    Added a warning in policy doc about JSON formatted file is deprecated,
    also removed all the reference to policy.json file in doc as well as
    in tests.

    Related Blueprint: policy-json-to-yaml

    Closes-Bug: #1875418

    Change-Id: Ic4d3b998bb9701cb1e3ef12d9bb6f4d91cc19c18

[Ghanshyam Mann (ghanshyammann)]

closing it for oslo polocy also as all required work is done under - https://review.opendev.org/#/q/topic:migrate-to-focal-oslo+(status:open+OR+status:merged)