Heat API services re-read policy.json on each request

[Ilya Shakhat]

Heat API services (heat-api, heat-api-cfn) re-read policy.json file on each request despite caching settings in oslo_policy.

According to logs:
2017-01-20 06:58:46.814 DEBUG eventlet.wsgi.server [-] (30375) accepted ('192.168.122.58', 51722) from (pid=30375) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868
2017-01-20 06:58:46.815 DEBUG heat.api.middleware.version_negotiation [-] Processing request: GET /v1/d0d64850ea344ed78255e9726800c171/stacks Accept: application/json from (pid=30375) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:50
2017-01-20 06:58:46.815 DEBUG heat.api.middleware.version_negotiation [-] Matched versioned URI. Version: 1.0 from (pid=30375) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:65

2017-01-20 06:58:46.842 DEBUG oslo_policy._cache_handler [req-c5d3ca35-67d6-4418-b84b-d46f31498260 None demo] Reloading cached file /etc/heat/policy.json from (pid=30375) read_cached_file /usr/local/lib/python2.7/dist-packages/oslo_policy/_cache_handler.py:40
2017-01-20 06:58:46.844 DEBUG oslo_policy.policy [req-c5d3ca35-67d6-4418-b84b-d46f31498260 None demo] Reloaded policy file: /etc/heat/policy.json from (pid=30375) _load_policy_file /usr/local/lib/python2.7/dist-packages/oslo_policy/policy.py:666

2017-01-20 06:58:46.845 DEBUG heat.common.wsgi [req-c5d3ca35-67d6-4418-b84b-d46f31498260 None demo] Calling <heat.api.openstack.v1.stacks.StackController object at 0x7fe93517ee50> : index from (pid=30375) __call__ /opt/stack/heat/heat/common/wsgi.py:836
2017-01-20 06:58:46.846 DEBUG oslo_messaging._drivers.amqpdriver [req-c5d3ca35-67d6-4418-b84b-d46f31498260 None demo] CALL msg_id: 2b4121a745c84ea9a90a1e45e2cbe33d exchange 'heat' topic 'engine' from (pid=30375) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:442
2017-01-20 06:58:46.867 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 2b4121a745c84ea9a90a1e45e2cbe33d from (pid=30375) __call__ /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:299
2017-01-20 06:58:46.869 DEBUG heat.common.serializers [req-c5d3ca35-67d6-4418-b84b-d46f31498260 None demo] JSON response : {"stacks": []} from (pid=30375) to_json /opt/stack/heat/heat/common/serializers.py:40
2017-01-20 06:58:46.870 INFO eventlet.wsgi.server [req-c5d3ca35-67d6-4418-b84b-d46f31498260 None demo] 192.168.122.58 - - [20/Jan/2017 06:58:46] "GET /v1/d0d64850ea344ed78255e9726800c171/stacks HTTP/1.1" 200 227 0.054964

It's expected that oslo_policy should cache policy.json and re-read it only after changes.

The reason of the issue is that policy enforcer is created per-request and does not persist between requests:

  * policy.Enforcer() is created in constructor of RequestContext (https://github.com/openstack/heat/blob/master/heat/common/context.py#L118), it also creates oslo_policy.Enforcer()
  * self.policy.check_is_admin() is called (#123)
  * in check_is_admin() (https://github.com/openstack/heat/blob/master/heat/common/policy.py#L82) _check() is called which in turn calls self.enforcer.enforce() (#66)
  * Enforcer.enforce() reloads the policy

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/469472

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/469472
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=71c72340f58d28d7f865cb22917ecd18703cec74
Submitter: Jenkins
Branch: master

commit 71c72340f58d28d7f865cb22917ecd18703cec74
Author: Thomas Herve <email address hidden>
Date: Wed May 31 15:07:40 2017 +0200

    Cache policy enforcer

    The policy enforcer can be used as a global object, there is no create
    one for every context creation. This caches the object in the module and
    reuse it in context.

    Change-Id: I03087b7bc139c8aaef95809043fc57f52b7900a1
    Closes-Bug: #1658083

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/487654

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 9.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/487654
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9f1496f5632c89b0be47db42cf9a5d7234dfdbe4
Submitter: Jenkins
Branch: stable/ocata

commit 9f1496f5632c89b0be47db42cf9a5d7234dfdbe4
Author: Thomas Herve <email address hidden>
Date: Wed May 31 15:07:40 2017 +0200

    Cache policy enforcer

    The policy enforcer can be used as a global object, there is no create
    one for every context creation. This caches the object in the module and
    reuse it in context.

    Change-Id: I03087b7bc139c8aaef95809043fc57f52b7900a1
    Closes-Bug: #1658083
    (cherry picked from commit 71c72340f58d28d7f865cb22917ecd18703cec74)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 8.0.4 release.