Nova API not restarted when nova policy is updated

[Sébastien Gremion]

I was modifying the nova policy option in order to allow custom role to execute certain actions but it didn't work as planned. After having added my new configuration in the variable "nova_policy_overrides", I've launched the playbook os-nova-install.yml. The execution was successful and I could see my changes in the file /etc/nova/policy.json. Howerver, the access was still forbidden. It was only after having restarted the service nova-api that my changes became active.

To reproduce (on an existing OpenStack installation) :
1) Modify the content of the variable nova_policy_overrides
2) Execute playbook os-nova-install.yml
3) Test the API for which you have updated the rule in step 1) --> Should not work and report that the policy doesn't now allow
4) Restart the service nova-api manually
5) Execute the same test as in step 3) --> Should work this time

Possible fix:
- Restart the nova-api service if a change in the nova policy.json is detected

[Dmitriy Rabotyagov (noonedeadpunk)]

Eventually API service restart is not required to apply policy. OSLO loads policies resilently, without need to restart service itself. Might be smth weird with nova specificly though...

[Dmitriy Rabotyagov (noonedeadpunk)]

Can you kindly specify what OSA version are you running? As eventually oslo also states, that changes to policy should not require service restart: https://docs.openstack.org/oslo.policy/victoria/admin/policy-yaml-file.html

[Sébastien Gremion (sgremion)]

Thanks for your help Dmitriy. I'm running OSA in version 22.1.2

[Dmitriy Rabotyagov (noonedeadpunk)]

Well, I had spawned instance on master only, and policy has been applied without service restart:

(openstackclient) root@aio1:~# openstack server list

(openstackclient) root@aio1:~# openstack server list --all-projects
Policy doesn't allow os_compute_api:servers:detail:get_all_tenants to be performed. (HTTP 403) (Request-ID: req-4f41008e-566c-4be2-be3d-5a56a2d551f7)
(openstackclient) root@aio1:~# stat /var/lib/lxc/aio1_nova_api_container-a32981bc/rootfs/etc/nova/policy.yaml
stat: cannot stat '/var/lib/lxc/aio1_nova_api_container-a32981bc/rootfs/etc/nova/policy.yaml': No such file or directory
(openstackclient) root@aio1:~# echo '"os_compute_api:servers:detail:get_all_tenants": "rule:system_reader_api or rule:project_member_api"' > /var/lib/lxc/aio1_nova_api_container-a32981bc/rootfs/etc/nova/policy.yaml
(openstackclient) root@aio1:~# openstack server list --all-projects
+--------------------------------------+------+--------+-----------------------+--------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------+--------+-----------------------+--------+--------+
| 03167f5d-0ef1-47bf-8a04-80672ca437e1 | test | ACTIVE | private=192.168.74.10 | cirros | test |
+--------------------------------------+------+--------+-----------------------+--------+--------+
(openstackclient) root@aio1:~#

Will spawn 22.1.2 and see how things are there.

[Dmitriy Rabotyagov (noonedeadpunk)]

Sorry, I can't reproduce this behaviour even on 22.1.2 :(

```
(openstackclient) root@aio:~/openstack-ansible# git branch
* (HEAD detached at 22.1.2)
  master
(openstackclient) root@aio:~/openstack-ansible# openstack server list

(openstackclient) root@aio:~/openstack-ansible# openstack server list --all-projects
Policy doesn't allow os_compute_api:servers:detail:get_all_tenants to be performed. (HTTP 403) (Request-ID: req-0c9a0c09-a34a-41c7-92c3-386dce82a8f8)
(openstackclient) root@aio:~/openstack-ansible# stat /var/lib/lxc/aio1_nova_api_container-d8a994b5/rootfs/etc/nova/policy.yaml
stat: cannot stat '/var/lib/lxc/aio1_nova_api_container-d8a994b5/rootfs/etc/nova/policy.yaml': No such file or directory
(openstackclient) root@aio:~/openstack-ansible# echo '"os_compute_api:servers:detail:get_all_tenants": "rule:system_reader_api or rule:project_member_api"' > /var/lib/lxc/aio1_nova_api_container-d8a994b5/rootfs/etc/nova/policy.yaml
(openstackclient) root@aio:~/openstack-ansible# openstack server list --all-projects
+--------------------------------------+------+--------+----------------------+--------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------+--------+----------------------+--------+--------+
| 1eae58d7-0fd3-4345-aebd-6cb7a14e106a | test | ACTIVE | private=192.168.74.3 | cirros | cirros |
+--------------------------------------+------+--------+----------------------+--------+--------+
(openstackclient) root@aio:~/openstack-ansible#

```

[Dmitriy Rabotyagov (noonedeadpunk)]

Oh, wait a second. I was able to reproduce it with json file: http://paste.openstack.org/show/806326/

But it's not really role issue... While we can workaround this with service restart, I believe it should be fixed on oslo side.

[Sébastien Gremion (sgremion)]

Isn't OSA using the json format when overriding the policy ?

I used the variable nova_policy_overrides to customized the nova policy

[Dmitriy Rabotyagov (noonedeadpunk)]

Yeah, I used json afterwards and was able to partially reproduce the problem. But it's not that service doesn't read updated file but it just does randomly. And in my test env I had single container, so it was different replies from the exact same api endpoints.

But still I believe it's oslo.policy bug.

With using nova_policy_overrides I got exactly same effect - that sometimes service does respond with applied policy and sometimes not.