[k8s] Juju doesn't set CPU / memory requests for its internal pods

[Junien F]

Hi,

When Juju creates its internal k8s pods (the model operator and the applications operators), it doesn't specify any CPU or memory request [1] - which mean they're effectively 0.

This confuses the k8s scheduler since it relies on these values to properly schedule pods to the appropriate nodes.

Juju should likely set a non-zero value for CPU and memory requests, and probably CPU and memory limits as well. I'm not sure if it should provide a way to override these defaults, I'll leave this up to you.

If you don't want to provide default values, at least Juju should let the operators specify values.

I think charm developers can set requests and limits via k8s_resources.

Note : a workaround to this bug is to specify CPU/memory requests and limits per namespace. Perhaps juju should set these on the namespaces it creates. The values would then be applied to all the pods in the namespace, unless overriden by a pod spec.

https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
https://kubernetes.io/docs/concepts/policy/limit-range/
https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/cpu-constraint-namespace/

[John A Meinel (jameinel)]

We do have some default heuristics (IIRC 2-4GB memory, 2 cpus, 32G disk),
that we use for normal instances, we could probably be using them for k8s
containers as well.

On Mon, Dec 14, 2020 at 1:10 PM Junien Fridrick <email address hidden>
wrote:

> Public bug reported:
>
> Hi,
>
> When Juju creates its internal k8s pods (the model operator and the
> applications operators), it doesn't specify any CPU or memory request
> [1] - which mean they're effectively 0.
>
> This confuses the k8s scheduler since it relies on these values to
> properly schedule pods to the appropriate nodes.
>
> Juju should likely set a non-zero value for CPU and memory requests, and
> probably CPU and memory limits as well. I'm not sure if it should
> provide a way to override these defaults, I'll leave this up to you.
>
> If you don't want to provide default values, at least Juju should let
> the operators specify values.
>
> I think charm developers can set requests and limits via k8s_resources.
>
> Note : a workaround to this bug is to specify CPU/memory requests and
> limits per namespace. Perhaps juju should set these on the namespaces it
> creates. The values would then be applied to all the pods in the
> namespace, unless overriden by a pod spec.
>
>
> https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
> https://kubernetes.io/docs/concepts/policy/limit-range/
>
> https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/cpu-constraint-namespace/
>
> ** Affects: juju
> Importance: Undecided
> Status: New
>
>
> ** Tags: k8s
>
> ** Tags added: k8s
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1908110
>
> Title:
> [k8s] Juju doesn't set CPU / memory requests for its internal pods
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1908110/+subscriptions
>

[Ian Booth (wallyworld)]

We do limit the memory and CPU on the controller pod itself using bootstrap constraints (mem defaults to 1.5G). We don't have a way to allow separate constraints for the controller container and the mongo container so we just use the same value for both.

But yes, model operator and application operators do not have any limits defined.

[Junien F (axino)]

Got bit by this today again.

On top of setting CPU and mem requests for its own pods, I'd STRONGLY suggest that Juju also set CPU and mem requests for any pod it creates, if the charm doesn't provide any.

[Tom Haddon (mthaddon)]

Just to elaborate on the issue, pods not having CPU and mem requests means k8s has no idea where to schedule pods, and tends to just schedule them all on one node. Until that is overloaded and reaches a "NotReady" state. Worth noting https://bugs.launchpad.net/juju/+bug/1919976 here ("Need a way to differentiate between resource limits and requests in kubernetes constraints"), which is related.