Impossible to modify nginx parameter client_max_body_size

[foot]

Hello,

Problem:

Trying to install helm chart stable/prometheus-operator 9.3.1 will results with following error:
"Error: create: failed to create: the server responded with the status code 413 but did not return more information (post secrets)"

Description:

Nginx service on kubeapi-load-balancer configured with default client_max_body_size parameter that equals to 1Mb. prometheus-operator helm chart generates secret bigger than 1Mb that leads to error:

/var/log/nginx.error.log:
"2020/08/31 16:06:14 [error] 20918#20918: *10354770 client intended to send too large body: 1054818 bytes, client: 10.100.251.5, server: _, request: "POST /api/v1/namespaces/monitoring/secrets HTTP/2.0", host: "10.100.38.59:443""

/var/log/nginx.access.log:
10.100.251.5 - admin [31/Aug/2020:16:06:14 +0000] "POST /api/v1/namespaces/monitoring/secrets HTTP/2.0" 413 208 "-" "helm/v0.0.0 (linux/amd64) kubernetes/$Format"

[John Puskar (jpuskar-amtrust)]

I just wanted to confirm that this is indeed still a problem, and that setting client_max_body_size to 10M or so (and running helm very quickly before it gets reverted) is the only workaround I've found so far.

It's one of the only manual steps in my cluster onboarding process and a real pain to work around.

Also, the OP states that this is about secrets, but more precisely it's the total manifest size that helm renders out of the kube-prometheus-stack chart.

[John Puskar (jpuskar-amtrust)]

I found a better workaround!
Adding a nginx conf file (which in hindsight seems fairly obvious).

---
/home/ubuntu# cat /etc/nginx/conf.d/client_max_body_size.conf
client_max_body_size 10M;
---

[Nikolay Vinogradov (nikolay.vinogradov)]

Just stumbled across this one. I was following the deployment guide of Harbor Operator v.1.3.0 from https://github.com/goharbor/harbor-operator/blob/ae23ba2c42d9d4f89a0c65459be133ed3a6e8036/docs/installation/by-helm-chart.md and running the full-stack deployment command :

helm upgrade --namespace harbor-operator-ns --install harbor-operator charts/harbor-operator-v1.3.0.tgz --set-string image.repository=ghcr.io/goharbor/harbor-operator --set-string image.tag=v1.3.0 --set installCRDs=true --set redis-operator.enabled=true --set minio-operator.enabled=true --set postgres-operator.enabled=true

I got error from helm:

...
Error: create: failed to create: the server responded with the status code 413 but did not return more information (post secrets)
helm.go:88: [debug] the server responded with the status code 413 but did not return more information (post secrets)
...

I've not had such problem on Microk8s.

I think the default values of request limits should be set according to the configured limits of etcd and kubeapi of Charmed Kubernetes by default and be configurable.

[George Kraft (cynerva)]

I agree that adding a config option for this would be good.

The max request body size in kube-apiserver appears to be 3MB[1] and unconfigurable. I suspect 3MB would be a good default for kubeapi-load-balancer's client_max_body_size config as well.

[1]: https://github.com/kubernetes/kubernetes/blob/d29e3bd7aa4978f10a01b9dfdebb2a647634f8fe/staging/src/k8s.io/apiserver/pkg/server/config.go#L407

[Kellen Renshaw (krenshaw)]

Has this change been made? I am running into this issue as well.

[George Kraft (cynerva)]

Nope, this hasn't been fixed. If this is a frequent problem or you've found it to have a big impact, let us know and we can re-prioritize it. Or feel free to submit a PR to https://github.com/charmed-kubernetes/charm-kubeapi-load-balancer.

[Adam Dyess (addyess)]

Perhaps we should create another config option 'nginx-http-config' similar to [0] in the PR https://github.com/charmed-kubernetes/charm-kubeapi-load-balancer/pull/31

[0] https://github.com/charmed-kubernetes/charm-kubeapi-load-balancer/blob/main/config.yaml#L35-L48

This would allow the user to configure the charm:

juju config kubeapi-load-balancer nginx-http-config='{client_max_body_size size 3m}'

[Kevin W Monroe (kwmonroe)]

PR for review:

https://github.com/charmed-kubernetes/charm-kubeapi-load-balancer/pull/32

[Adam Dyess (addyess)]

backport completed after pulling commits from main up to
https://github.com/charmed-kubernetes/charm-kubeapi-load-balancer/pull/33

[George Kraft (cynerva)]

Due to https://bugs.launchpad.net/bugs/2038347 and https://bugs.launchpad.net/bugs/2038348, we had to roll back kubeapi-load-balancer to a version prior to this fix. This is now targeted for 1.29.