Make pod security policies configurable

[Jorge Niedbalski]

[Environment]

Kubernetes 1.19

[Description]

There is no configuration option to allow for a more restrictive PSP policy to be provided when using rbac different to the privileged one provided as a template.

Default cluster role bindings defined for rbac pod security policies [0] are rendered once in the leader unit when setting up the kubernetes-master nodes. Those will get overwritten
if the juju leader changes for any reason (that's up to the consensus protocol and can't be guaranteed).

[0] https://github.com/charmed-kubernetes/charm-kubernetes-master/blob/e03d8394eac686ecd8c061f051ca01c02bfe375f/reactive/kubernetes_master.py#L1715
[1] https://github.com/charmed-kubernetes/charm-kubernetes-master/blob/master/templates/rbac-pod-security-policy.yaml

[Possible Solution]

Allow a custom rbad-pod-security-policy to be passed via a configuration directive to prevent
the charm to overwrite them.

This is linked to https://bugs.launchpad.net/cdk-addons/+bug/1906303

[Jorge Niedbalski (niedbalski)]

Submitted PR https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/139

[Paul Goins (vultaire)]

Based on reviewing the CDK 1.20 version of the kubernetes-master charm, this seems like it will occur on the first time any kubernetes-master unit is promoted to leader.

While dirty, a mitigation which can be performed on newly added units (assuming they don't get promoted to leader right away) is to run this SQL query against the /var/lib/juju/agents/<unit>/charm/.unit-state.db SQLite3 DB:

  insert into kv (key, data) values ("reactive.states.kubernetes-master.pod-security-policy.applied", "null")

This will set the flag which indicates the settings have already been applied, and thus they won't be re-applied.

[Paul Goins (vultaire)]

(Note: I have not tested the above workaround, but based upon code review, I'm pretty sure it should work.)

[Paul Goins (vultaire)]

Actually, just got a chance to test the workaround above for preventing re-writing of the privileged cluster security policies - it does seem to work, for what it's worth.

[Jeff Hillman (jhillman)]

Subscribed Field-High.

[George Kraft (cynerva)]

PR: https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/178

[George Kraft (cynerva)]

Cherry-pick to stable branch: https://github.com/charmed-kubernetes/charm-kubernetes-master/commit/bfc2ffb5b156f20933b77bcf37d2f80d56b012c1