Metrics from kube-scheduler, kube-controller-manager and kube-proxy need to be exported to prometheus charm

[Seyeong Kim]

For now, In prometheus UI, I can't see scheduler_ prefixed metrics.

I've analyzed this a few and found out kubernetes-master doesn't register manual_job for kube-scheduler

so I added template manually like below

##
scrape_interval: 30s
metrics_path: /metrics
scheme: http
static_configs:
- targets:
- 10.0.0.9:10251 ( kubernetes-master/leader 's ipaddress )
##

then I can see scheduler_ prefixed metrics in prometheus UI.

It would be better if there is way to export kube-scheduler's metrics as well.

ADDED

In addition to kube-scheduler, kube-controller-manager and kube-proxy metrics also not exported to prometheus.

For kube-proxy, kube-proxy snap need to be changed as it doesn't export it's metric for now.

--metrics-bind-address ipport Default: 127.0.0.1:10249 ( https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/ )

ADDED for @cynerva's

Just adding template can't solve this issue as

e.g kube-state-metrics exposed by using service & endpoint with k8s deployment ( for kube-state-metrics)

So it can be access with metrics_path: /api/v1/namespaces/kube-system/services/kube-state-metrics:8081/proxy/metrics

But for kube-scheduler, kube-controller-manager, kube-proxy are not exposed.

I tried to create service & endpoint for kube-scheduler, but it is local service inside k8s-master unit not deployment.

I think we need to change deployment structure to expose those.

Please give me some more advice or please engage canonical k8s team for this.

Thanks.

[George Kraft (cynerva)]

It should be possible to add this to the kubernetes-master charm's prometheus job templates[1].

[1]: https://github.com/charmed-kubernetes/charm-kubernetes-master/tree/bb3328c490f2e05b20010f688be55296623fb7e1/templates/prometheus

[Seyeong Kim (seyeongkim)]

In addition to kube-scheduler, kube-controller-manager and kube-proxy metrics also not exported to prometheus.

[Seyeong Kim (seyeongkim)]

@cynerva

I updated description for your comment,
kube-state-metric is exported by service, then we can access via kube-api-loadbalancer,
but the others aren't.

so just creating template doesn't solve this. ( if we specify only one node ( leader ) it could be ok but it doesn't make sense to me )

[George Kraft (cynerva)]

Yep, so it looks like the existing jobs point to a single IP for the Kubernetes API, but the situation is different here since multiple IPs are needed.

It looks like you'll need to expand on the register_prometheus_jobs handler[1] in one of two ways. Either:

1) Collect IPs for all kubernetes-master units, and include them when rendering templates for services like kube-scheduler, or

2) Create a new handler, without the leadership.is_leader flag, so each kubernetes-master unit can create scrabe jobs for itself.

That's about all the guidance I can offer at the moment.

[1]: https://github.com/charmed-kubernetes/charm-kubernetes-master/blob/bb3328c490f2e05b20010f688be55296623fb7e1/reactive/kubernetes_master.py#L3177

[Alex Pearce (alexpearce)]

Am I right in thinking that this would also affect a non-charm Prometheus from scraping these components (kube-scheduler, kube-controller-manager, and kube-proxy)? I install kube-prometheus-stack into my Charmed Kubernetes cluster and the default alerts complain that these components cannot be scraped.