Need a way to differentiate between resource limits and requests in kubernetes constraints

[Tom Haddon]

Deploying a kubernetes charm with constraints of "mem=100M cpu-power=100" (for instance) will produce the following (from describe pod):

Containers:
[...]
    Limits:
      cpu: 100m
      memory: 100Mi
    Requests:
      cpu: 100m
      memory: 100Mi

However, being able to set these to different values would be useful, and is good practice in Kubernetes. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for more details.

[Harry Pidcock (hpidcock)]

Thanks Tom, we have this on out roadmap, I'm going to target this to 2.9.1 as its fairly important.

[Giuseppe Petralia (peppepetra)]

Is the support for hugepages constraints in the roadmap as well?

[Ian Booth (wallyworld)]

We've started a spec for this work
https://docs.google.com/document/d/1t4gvqA-tQ_Nv1lYiTl38QZDwQeGkD7tG_VlIfKcpbQA

[Paulo Machado (paulomachado)]

Hello there! Is this fix planned to be backported to 2.9 series? Thanks

[Alex Lutay (taurus)]

Dear Juju Team,

Can you please consider to increase priority for this ticket.
It will be a hot topic for the next customer mysql-k8s and postgresql-k8s DB deployment by Juju.

We have to deploy DB with RAM constraints to properly perform RAM management inside the unit (auto-tuning) but Juju applies the same constrains on "charm" and "workload" which is resource in-optimal (charm container do not need constraints on 64GB RAM), but also K8s do not deploy charm on node due to resource limits. Example:

> `juju deploy mysql-k8s --constraints mem=64G`
^ Pod will NOT be scheduled for K8s node with 64GB.
The constraint requires 128GB+ availability.

> `juju deploy mysql-k8s --constraints mem=32G`
^ Pod still will NOT be scheduled for K8s node with 64GB RAM! <== Boom.
Charm DB application cannot be deployed at all if all your K8s nodes have 64GB RAM.

Tnx!

[John A Meinel (jameinel)]

If the charm wouldn't be scheduled with '32GB' constraint but 64GB nodes,
are you sure that it would be scheduled without juju in the mix and a 64GB
request? That at least sounds like K8s is reserving some memory on the node
for other activities, and you need to go to something like 30GB. At least
for testing purposes.

As far as actually being able to prioritize this, we certainly understand
that you have a business case for it, but not really the concrete
implications and whether there is a workaround, to be able to prioritize it
relative to other activities.

We certainly should get this into our product feedback prioritization
queue, though.

On Tue, Sep 5, 2023 at 10:06 AM Alex Lutay <email address hidden>
wrote:

> Dear Juju Team,
>
> Can you please consider to increase priority for this ticket.
> It will be a hot topic for the next customer mysql-k8s and postgresql-k8s
> DB deployment by Juju.
>
> We have to deploy DB with RAM constraints to properly perform RAM
> management inside the unit (auto-tuning) but Juju applies the same
> constrains on "charm" and "workload" which is resource in-optimal (charm
> container do not need constraints on 64GB RAM), but also K8s do not
> deploy charm on node due to resource limits. Example:
>
>
> > `juju deploy mysql-k8s --constraints mem=64G`
> ^ Pod will NOT be scheduled for K8s node with 64GB.
> The constraint requires 128GB+ availability.
>
> > `juju deploy mysql-k8s --constraints mem=32G`
> ^ Pod still will NOT be scheduled for K8s node with 64GB RAM! <== Boom.
> Charm DB application cannot be deployed at all if all your K8s nodes have
> 64GB RAM.
>
> Tnx!
>
> --
> You received this bug notification because you are subscribed to
> Canonical Juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1919976
>
> Title:
> Need a way to differentiate between resource limits and requests in
> kubernetes constraints
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1919976/+subscriptions
>
>

[Alex Lutay (taurus)]

> ... That at least sounds like K8s is reserving some memory on the node for other activities...

Absolutely! Details: https://github.com/canonical/mysql-k8s-operator/issues/254#issuecomment-1632365691

TL;DR:
```
> kubectl describe node gke-taurus-20152-default-pool-dd74c68c-7cq0
...
Capacity:
...
  memory: 15358168Ki
Allocatable:
...
  memory: 12658904Ki
```

So, K8s will NOT schedule DB pod launch (juju deploy mysql-k8s --constraints mem=8GB) on K8s node with 16GB real RAM, as 12GB is "Allocatable" only and Juju requests 8GB for charm container + 8GB for workload container.

[Simon Aronsson (0x12b)]

Workaround until Juju properly supports this: https://github.com/canonical/observability-libs/blob/main/lib/charms/observability_libs/v0/kubernetes_compute_resources_patch.py