KubeletTooManyPods alert is incorrecly sent

[Bartłomiej Poniecki-Klotz]

I have a Charmed Kuberentes cluster of:
- 3 control plane nodes
- 3 worker nodes

I deployed 133 Pods, so it's far from capacity.

On each control node, I get KubeletTooManyPods alert (pasted below).

name: KubeletTooManyPods
expr: count by (cluster, node) ((kube_pod_status_phase{job="kube-state-metrics",juju_application="kubernetes-control-plane",juju_model="kubernetes-sm",juju_model_uuid="2f45d4ac-212c-4878-857b-8df46e8c9d3a",phase="Running"} == 1) * on (instance, pod, namespace, cluster) group_left (node) topk by (instance, pod, namespace, cluster) (1, kube_pod_info{job="kube-state-metrics",juju_application="kubernetes-control-plane",juju_model="kubernetes-sm",juju_model_uuid="2f45d4ac-212c-4878-857b-8df46e8c9d3a"})) / max by (cluster, node) (kube_node_status_capacity{job="kube-state-metrics",juju_application="kubernetes-control-plane",juju_model="kubernetes-sm",juju_model_uuid="2f45d4ac-212c-4878-857b-8df46e8c9d3a",resource="pods"} != 1) > 0.95
for: 15m
labels:
juju_application: kubernetes-control-plane
juju_charm: kubernetes-control-plane
juju_model: kubernetes-sm
juju_model_uuid: 2f45d4ac-212c-4878-857b-8df46e8c9d3a
severity: info
annotations:
description: Kubelet '{{ $labels.node }}' is running at {{ $value | humanizePercentage }} of its Pod capacity.
runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubelettoomanypods
summary: Kubelet is running at capacity.

Versions Charmed K8s - 1.29/beta charms, 1.28/stable K8s API (snap)

COS:
App Version Status Scale Charm Channel Rev Address Exposed Message
alertmanager 0.25.0 active 1 alertmanager-k8s stable 96 10.152.183.33 no
catalogue active 1 catalogue-k8s stable 31 10.152.183.65 no
grafana 9.2.1 active 1 grafana-k8s stable 93 10.152.183.236 no
loki 2.7.4 active 1 loki-k8s stable 105 10.152.183.209 no
prometheus 2.47.2 active 1 prometheus-k8s stable 156 10.152.183.48 no
prometheus-scrape-config n/a active 1 prometheus-scrape-config-k8s latest/stable 44 10.152.183.190 no
traefik 2.10.4 active 1 traefik-k8s stable 166 172.29.200.100 no