oslo.policy policy_dirs support is missing

[Dmitrii Shcherbakov]

In order to augment policy definitions either a charm-supplied policy file needs to be modified or a drop-in mechanism needs to be used.

oslo.policy provides policy_dirs option ('policy.d' relative to conf_dir by default) which is inspected if policy_file exists (which is the case for most of the openstack charms but not for every charm).

https://github.com/openstack/oslo.policy/blame/stable/ocata/oslo_policy/opts.py#L25-L47

Projects that use oslo.policy's Enforcers automatically gain support for this functionality.

https://github.com/openstack/oslo.policy/blob/stable/ocata/oslo_policy/policy.py#L428-L445
https://github.com/openstack/oslo.policy/blob/stable/ocata/oslo_policy/policy.py#L502-L542

There needs to be a way for charms to utilize this by rendering a service-specific policy file.

Support for this is needed across classic and reactive openstack charms (primary service charms, plugin charms, subordinate charms). For subordinate charms this would mean that a common mechanism to trigger service restart would be needed across different charms on subordinate policy config change.

[Dmitrii Shcherbakov (dmitriis)]

charm-helpers:
https://github.com/juju/charm-helpers/pull/84

keystone:
https://review.openstack.org/#/c/531614/

[Dmitrii Shcherbakov (dmitriis)]

Modified to include an ability to supply a jinja2 template as a drop-in and use the same variables as in policy.json template file.

https://github.com/juju/charm-helpers/pull/87
https://github.com/juju/charm-helpers/pull/88
https://review.openstack.org/#/c/531614/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/539269

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/538688
Committed: https://git.openstack.org/cgit/openstack/charms.openstack/commit/?id=1ca4bd0ab941f6c7aebf86bdf5ff47fdc1d2d3e2
Submitter: Zuul
Branch: master

commit 1ca4bd0ab941f6c7aebf86bdf5ff47fdc1d2d3e2
Author: Dmitrii Shcherbakov <email address hidden>
Date: Sat Jan 27 22:42:16 2018 +0300

    add string template rendering capability

    In some cases software deliberately allows drop-in config file usage
    capabilities, for example, when it comes to enforcing policy, the
    desired behavior varies from an operator to operator. For that reason it
    is sometimes desirable to supply custom templates via config options.

    Another use-case is templates that are passed from subordinates for a
    primary charm to render.

    Given that properties and desired adapters can be arbitrary the change
    uses a dict of meta tuples of the following format to render templates
    from strings based on adapter properties:

    {config_file_path: (relation_name, adapter property)}

    relation names must be normalized (lowercase, underscores instead of
    dashes. "options" relation name is used for a config adapter as usual.

    In summary a string config file path should be used:

    1. in the restart_map for a given derived class;
    2. in string_templates dict as a key for a meta tuple

    Change-Id: Ic85b22d0e5d497c49c75243e3c280140f940df66
    Closes-Bug: #1741723

[Dmitrii Shcherbakov (dmitriis)]

openstack-dashboard is a bit unique as it needs to have many per-service policy files and they need to be in sync with individual policy files for other charms.

https://docs.openstack.org/horizon/latest/contributor/topics/policy.html
"The implementation in Horizon is based on copies of policy files found in the service’s source code."

Therefore, horizon charm will need to populate per-service policy dirs and this is something only available in Queens for Horizon project itself:

https://blueprints.launchpad.net/horizon/+spec/policy-dirs

So, for horizon, a single extra-policy option will need to have policy file templates for multiple services which makes it quite complex.

Maybe in case of horizon it would be necessary to get policy files via relation data but it is necessary to get 3 types of policy:

1. a render of policy-in-code for a given service (not applicable for older OpenStack releases);
2. policy files embedded into charms;
3. extra-policy configs from individual charms.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Frode Nordahl (<email address hidden>) on branch: master
Review: https://review.openstack.org/531614
Reason: Marking this review as abandoned due to lack of activity in the past 6 months.
Feel free to restore it again if you want to pick up and continue the work.

[Paul Henien (phenien)]

A customer asked asked making changes in the policy.json on the cinder and openstack-dashboard charms to enable consistency groups and consistency group snapshots.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Kavanagh (tinwood) (<email address hidden>) on branch: master
Review: https://review.opendev.org/685331
Reason: neutron-gateway isn't actually an API charm and so doesn't need this change!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/685973

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/685982

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/685726
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=f7f6fa295c82ea88953ec08695c68c04694fc1a5
Submitter: Zuul
Branch: master

commit f7f6fa295c82ea88953ec08695c68c04694fc1a5
Author: Alex Kavanagh <email address hidden>
Date: Mon Sep 30 15:47:53 2019 +0100

    Policyd override implementation

    This patchset implements policy overrides for nova-cloud-controler.

    This change includes a charm-helpers sync to bring in the policyd helper
    code.

    Note there are no functional tests for this feature as the charm still
    uses the old style non-zaza amulet framework. The Related-Bug below is
    tracking this issue.

    Change-Id: Ia5f3f8189d4a7b7b46a827707d964ebe40740aeb
    Closes-Bug: #1741723
    Related-Bug: #1845639

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/685376
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=697ca00bcf29c6a6f3dbca4b13c16a87804592ab
Submitter: Zuul
Branch: master

commit 697ca00bcf29c6a6f3dbca4b13c16a87804592ab
Author: Alex Kavanagh <email address hidden>
Date: Fri Sep 27 16:12:19 2019 +0100

    Policyd override implementation

    This patchset implements policy overrides for neutron-gateway.

    This change includes a charm-helpers sync to bring in the policyd helper
    code.

    Change-Id: I89f1f4b5d58843017e428a8d2cfada840dde14de
    Closes-Bug: #1741723

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/679420
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=186769cc056f4306111c1c26e899dd4697ff468f
Submitter: Zuul
Branch: master

commit 186769cc056f4306111c1c26e899dd4697ff468f
Author: Alex Kavanagh <email address hidden>
Date: Fri Aug 30 11:58:04 2019 +0100

    Policyd override implementation

    This patchset implements policy overrides for keystone. It uses the
    code in charmhelpers.

    Closed-Bug: #1741723
    Change-Id: I187f4493392178d87ef7dbd67de841bbeae0c65d

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/685973
Committed: https://git.openstack.org/cgit/openstack/charm-glance/commit/?id=97152f55a1f045484ad3fde69a99a84fce8f349b
Submitter: Zuul
Branch: master

commit 97152f55a1f045484ad3fde69a99a84fce8f349b
Author: Alex Kavanagh <email address hidden>
Date: Tue Oct 1 14:55:29 2019 +0100

    Policyd override implementation

    This patchset implements policy overrides for glance. It uses the
    code in charmhelpers.

    Change-Id: I0586326ff87fdf03f2c88e4c459627f4085c3367
    Closed-Bug: #1741723

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Kavanagh (tinwood) (<email address hidden>) on branch: master
Review: https://review.opendev.org/686006
Reason: OpenStack dashboard requires a much more complex set of policy overrides than the system in charm-helpers can support. It'll require a completely different approach, but can be based on the existing code.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/685982
Committed: https://git.openstack.org/cgit/openstack/charm-cinder/commit/?id=6ee32006e54f67e7415a042f00b3489ddbcd113a
Submitter: Zuul
Branch: master

commit 6ee32006e54f67e7415a042f00b3489ddbcd113a
Author: Alex Kavanagh <email address hidden>
Date: Tue Oct 1 15:37:29 2019 +0100

    Policyd override implementation

    This patchset implements policy overrides for cinder. It uses the
    code in charmhelpers.

    It also fixes several bugs in the bundles where the actual version of
    cinder that was being installed was the distro default rather than the
    one that the bundle described.

    Change-Id: Ic979dcb96ddb931fadb1fa4a4b36108244ddf306
    Closed-Bug: #1741723

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/693162

[Alex Kavanagh (ajkavanagh)]

Note that swift software doesn't (yet) support the oslo.policy Enforcer class, which means that policy.d overrides aren't possible with it (yet).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693162
Committed: https://git.openstack.org/cgit/openstack/charm-heat/commit/?id=98de623820f58dc562ba770431e3464193d15a27
Submitter: Zuul
Branch: master

commit 98de623820f58dc562ba770431e3464193d15a27
Author: Alex Kavanagh <email address hidden>
Date: Wed Nov 6 11:44:06 2019 +0000

    Policyd override implementation

    This patchset implements policy overrides for heat. It uses the
    code in charmhelpers.

    It also fixes a bug in the actions/domain-setup where it assumes that
    the python2 version of openstackclient should be installed, and corrects
    this via code in hooks/install and hooks/upgrade-charm.

    A sync of charm-helpers is included to bring the latest policyd changes
    through to the charm.

    func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/111

    Change-Id: Ia607dc9120cfb03902efb019041b43cf12ade2d3
    Closed-Bug: #1741723

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/693407

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/689469
Committed: https://git.openstack.org/cgit/openstack/charm-openstack-dashboard/commit/?id=1b1e7c583c68a166ba2b826b7fd2606e10d358ce
Submitter: Zuul
Branch: master

commit 1b1e7c583c68a166ba2b826b7fd2606e10d358ce
Author: Alex Kavanagh <email address hidden>
Date: Fri Oct 18 14:55:26 2019 +0100

    Policyd override implementation

    This patchset implements policy overrides for octavia. It uses the
    code in charmhelpers [1] which has been modified to support the richer
    and more complex approach to handling policy overrides.

    [1]: https://github.com/juju/charm-helpers/pull/393

    func-test-pr: https://github.com/openstack-charmers/zaza-openstack-tests/pull/126

    Change-Id: Ib51fd2c7c540c680083c2928eab4ce4df0d43e23
    Closed-Bug: #1741723

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693407
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=5e3e942cd88db2fe6c15fa2182a64796944b9b51
Submitter: Zuul
Branch: master

commit 5e3e942cd88db2fe6c15fa2182a64796944b9b51
Author: Alex Kavanagh <email address hidden>
Date: Thu Nov 7 18:50:25 2019 +0000

    Policyd override implementation

    This patchset implements policy overrides for octavia. It uses the
    code in charmhelpers.

    Change-Id: Ia46e7aa885f6fadfe45cd8ea1db76400a5f6c34e
    Closed-Bug: #1741723

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dmitrii Shcherbakov (<email address hidden>) on branch: master
Review: https://review.opendev.org/539269

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/739274

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/739275

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/739274
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=7bec3ddee6163a566d2a9c0aaadacc1be27be9a7
Submitter: Zuul
Branch: master

commit 7bec3ddee6163a566d2a9c0aaadacc1be27be9a7
Author: Arif Ali <email address hidden>
Date: Fri Jul 3 16:14:04 2020 +0100

    Policyd override implementation

    This patchset implements policy overrides for gnocchi

    Change-Id: Ie7a02c536d5085cd2548b84d94b7a6bf8ac6a315
    Closed-Bug: #1741723

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738745
Committed: https://git.openstack.org/cgit/openstack/charm-swift-proxy/commit/?id=d21648e67106c48e222a3f698eeb2255593326dd
Submitter: Zuul
Branch: master

commit d21648e67106c48e222a3f698eeb2255593326dd
Author: Arif Ali <email address hidden>
Date: Tue Jun 30 19:04:37 2020 +0000

    Policyd override implementation

    This patchset implements policy overrides for swift-proxy. It uses
    the code in charmhelpers.

    Closed-Bug: #1741723
    Change-Id: Ic51ee5d181558b63dfd968c5b0c7d40760a5ac59

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/739275
Committed: https://git.openstack.org/cgit/openstack/charm-aodh/commit/?id=2e500d38c2f4c6779dd06f5a841e1b268055e7ab
Submitter: Zuul
Branch: master

commit 2e500d38c2f4c6779dd06f5a841e1b268055e7ab
Author: Arif Ali <email address hidden>
Date: Fri Jul 3 15:41:55 2020 +0100

    Policyd override implementation

    This patchset implements policy overrides for aodh. Updates
    xenial-ocata and trusty-mitaka functional tests

    Change-Id: Ic2477ed4ccc05c5bef384c4d9dc9a1776de9f6f6
    Closed-Bug: #1741723