limit the number of auth-webhook worker processes

Bug #1904747 reported by Kevin W Monroe on 2020-11-18
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Kubernetes Master Charm
High
Kevin W Monroe

Bug Description

Gunicorn recommends setting (2*cores)+1 as the number of workers for an application:

https://docs.gunicorn.org/en/stable/design.html#how-many-workers

It also says that 4-12 workers should be enough to handle 1000s of reqs per second. K8s master currently does not put a ceiling on the number of workers it spawns, so a 64-core system will fire up 129 workers. This is overkill.

Changed in charm-kubernetes-master:
status: New → In Progress
assignee: nobody → Kevin W Monroe (kwmonroe)
milestone: none → 1.20
Kevin W Monroe (kwmonroe) wrote :
tags: added: review-needed
Changed in charm-kubernetes-master:
importance: Undecided → High
status: In Progress → Fix Committed
tags: removed: review-needed
Peter De Sousa (pjds) wrote :

subscribing field high as this is affecting a kubeflow deployment

Chris Sanders (chris.sanders) wrote :

Peter if you are experience a definitive impact to the deployment please provide logs or some description of that effect. This is landing in 1.20 either way, but as far as I'm aware this is a matter of preventative adjustment to avoid issues.

Peter De Sousa (pjds) wrote :

Adding some context to how this bug is exhibiting itself on a deployment.

We have machines with 49 threads per core with 2 cores per machine (resulting in 97 gunicorn workers per master) - this is exhibiting itself with quite persistent timeouts on kubectl or internal calls to etcd whereby we see 'context exceeded'. When the number was reduced the timeouts were also reduced significantly.

Changed in charm-kubernetes-master:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers