[k8s] Juju controller and model operators should run with higher priority

[Pedro Guimarães]

On some stress tests, the team noticed that modeloperator pod got evicted from a node that was having memory contention.

Although this behavior is expected in k8s, Juju should take provisions to not have its own control plane evicted and/or not scheduling due to lack of resources.

The recommendation is to create a PriorityClass with higher value and setting it to all juju control plane pods (controller, model-operators).

Documentation: https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass

That means:
1. These pods will have higher priority when k8s needs to decide which pods to evict from a given node
2. These pods get priority when (re)scheduling.

[Pedro Guimarães (pguimaraes)]

I was working in another similar bug, and stumbled in this again. Another thing we were looking into is PodDisruptionBudget: https://kubernetes.io/docs/tasks/run-application/configure-pdb/

Reference bug: https://github.com/canonical/postgresql-k8s-operator/issues/269

[Ian Booth (wallyworld)]

The next 2.9.46 candidate release will not include a fix for this bug and we don't plan on any more 2.9 releases. As such it is being removed from its 2.9 milestone.

If the bug is still important to you, let us know and we can consider it for inclusion on a 3.x milestone.