"Waiting to retry addon deployment" status after relating ceph-client

[George Kraft]

After adding a kubernetes-control-plane:ceph-client relation, kubernetes-control-plane can enter a "waiting" status with the message: "Waiting to retry addon deployment"

From juju debug-log:

unit-kubernetes-control-plane-1: 2022-05-03 18:49:18 WARNING unit.kubernetes-control-plane/1.update-status Error from server (Invalid): error when creating "/root/snap/cdk-addons/10961/addons/csi-rbdplugin.yaml": DaemonSet.apps
 "csi-rbdplugin" is invalid: [spec.template.spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policy, spec.template.spec.containers[1].securityContext.privileged: Forbidden: disallowed by cluster policy, spec.template.spec.containers[2].securityContext.privileged: Forbidden: disallowed by cluster policy]

This happens when the charm is configured with allow-privileged=auto. Adding the ceph-client relation should have restarted kube-apiserver with allow-privileged=true, but it didn't.

[George Kraft (cynerva)]

PR: https://github.com/charmed-kubernetes/charm-kubernetes-control-plane/pull/223

[George Kraft (cynerva)]

Cherry-pick to stable branch: https://github.com/charmed-kubernetes/charm-kubernetes-control-plane/commit/db817508331455516de1da4d6bbb21618208f64a