[Feature Request] Minimize operator pod volume usage

[Kenneth Koski]

The background / motivating use case is this: The Kubeflow bundle is comprised of approximately 30 charms, with more slated to get included. Right now, Juju creates a PV/PVC per operator pod, and one operator pod per charm. This runs into issues with the fact that you can only mount so many volumes onto a single node instance with many providers, such as AWS. Concretely, when I spin up a Charmed Kubernetes stack on AWS and deploy Kubeflow to it, any one node can only have 26 volumes mounted on it.

The end result is that I'm unable to scale up with Charmed Kubernetes / Juju, the only option I have is to scale out, since all of the charms can't fit onto a single node instance. This isn't the worst thing in the world, but in a microservices world, that can translate to a lot of wasted capacity if you can't place enough microservices onto a node to maximize utilization.

There's two solutions that seem like they would solve this issue for me. Neither one seems particularly easy, hence this being a feature request:

1. The ability to run operator pods in a stateless manner, with no volumes.

This would probably require some work put into making the operator code idempotent, which won't be easy, but there are some pretty great benefits in fault-tolerance that this would bring.

2. The ability to coalesce multiple operators into fewer stateful pods

This would probably be easier than above, but may run into issues with complexities around logging, multi-threading, etc.

[Tim McNamara (tim-clicks)]

Hi Kenneth, do you mind posting this to Discourse to facilitate discussion?
It feels like there is a fairly wide design space to navigate.

On Wed, 21 Aug 2019, 08:30 Kenneth Koski <<email address hidden> wrote:

> Public bug reported:
>
> The background / motivating use case is this: The Kubeflow bundle is
> comprised of approximately 30 charms, with more slated to get included.
> Right now, Juju creates a PV/PVC per operator pod, and one operator pod
> per charm. This runs into issues with the fact that you can only mount
> so many volumes onto a single node instance with many providers, such as
> AWS. Concretely, when I spin up a Charmed Kubernetes stack on AWS and
> deploy Kubeflow to it, any one node can only have 26 volumes mounted on
> it.
>
> The end result is that I'm unable to scale up with Charmed Kubernetes /
> Juju, the only option I have is to scale out, since all of the charms
> can't fit onto a single node instance. This isn't the worst thing in the
> world, but in a microservices world, that can translate to a lot of
> wasted capacity if you can't place enough microservices onto a node to
> maximize utilization.
>
> There's two solutions that seem like they would solve this issue for me.
> Neither one seems particularly easy, hence this being a feature request:
>
>
> 1. The ability to run operator pods in a stateless manner, with no volumes.
>
> This would probably require some work put into making the operator code
> idempotent, which won't be easy, but there are some pretty great
> benefits in fault-tolerance that this would bring.
>
>
> 2. The ability to coalesce multiple operators into fewer stateful pods
>
> This would probably be easier than above, but may run into issues with
> complexities around logging, multi-threading, etc.
>
> ** Affects: juju
> Importance: Undecided
> Status: New
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju-bugs, juju-project-bugs
> https://bugs.launchpad.net/bugs/1840841
>
> Title:
> [Feature Request] Minimize operator pod volume usage
>
> Status in juju:
> New
>
> Bug description:
> The background / motivating use case is this: The Kubeflow bundle is
> comprised of approximately 30 charms, with more slated to get
> included. Right now, Juju creates a PV/PVC per operator pod, and one
> operator pod per charm. This runs into issues with the fact that you
> can only mount so many volumes onto a single node instance with many
> providers, such as AWS. Concretely, when I spin up a Charmed
> Kubernetes stack on AWS and deploy Kubeflow to it, any one node can
> only have 26 volumes mounted on it.
>
> The end result is that I'm unable to scale up with Charmed Kubernetes
> / Juju, the only option I have is to scale out, since all of the
> charms can't fit onto a single node instance. This isn't the worst
> thing in the world, but in a microservices world, that can translate
> to a lot of wasted capacity if you can't place enough microservices
> onto a node to maximize utilization.
>
> There's two solutions that seem like they would solve this issue for
> me. Neither one seems particula...

[Kenneth Koski (knkski)]

Thanks, opened a discussion item here:

https://discourse.jujucharms.com/t/minimizing-caas-operator-pod-volume-usage/1968

[John A Meinel (jameinel)]

I think it would be interesting to be able to use a different storage pool for operators. I can imagine that on an AWS instance if you are putting everything onto one node, you'd rather use node local storage instead of AWS EBS volumes.

[Ian Booth (wallyworld)]

You can use a different storage pool for operators by setting the operator-storage model config attribute. You'd point this to a k8s storage class configured to provision storage from something other than the cloud volume provisioner, eg NFS or whatever

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.