Juju has no way of waiting until Kubernetes charms are actually ready

[Kenneth Koski]

I would like to have a way of saying "Wait until every is 100% up and ready" in regular Juju. Right now there's the `juju-wait` plugin (which seems like it should be part of core juju), but that doesn't actually wait for the Kubernetes resources spun up by charms to complete setting themselves up, it only waits for the charms to report ready. See here for an example of how we have to do additional waiting in the kubeflow bundle setup:

https://github.com/juju-solutions/bundle-kubeflow/blob/master/scripts/cli.py#L423-L433

Additionally, there's a rare race condition that I've encountered where even the above isn't enough to ensure readiness. The steps that it goes through are:

 - script configures dex-auth with public-url configuration option
 - scripts runs `juju wait -wv` and `microk8s kubectl wait` as linked above
 - script determines that everything has settled and exits with successful completion message
 - juju actually configures dex-auth charm with new public-url configuration option
 - dex-auth charm deploys new dex-auth pod, triggering a rolling update
 - tests that are now running in CI fail due to dex-auth not being ready

So in other words, I would like some way to say "wait until all configuration changes that are in the pipeline have been processed, and everything is then 100% up and responsive"

[Jason Hobbs (jason-hobbs)]

juju-wait works pretty well, but it sounds like your charms are lying when they say they are Ready. Shouldn't they not report Ready until they're actually ready?

[Pen Gale (pengale)]

@jason-hobbs I think that there are places where it is difficult for a charm to know whether it is ready or not, especially in complex deployments, or when there are some levels of indirection, as with the kubernetes charms.

We've done some work addressing this with goal state, but goal state won't address @knkski's specific needs here.

I'm going to put this bug on the wishlist for now, but keep it in mind as we continue to work on the new kubernetes operators this cycle. There might be some things we can do to start to address this as we work on that.

[Kenneth Koski (knkski)]

Yeah, to expand on what petevg says above, in the K8s world, charms pass Juju a pod spec YAML document that declares what resources to create for it. After that, it's up to Juju to go off and create everything. Charm authors aren't really able to interact with the resources to check that they're ready.

As far as how Juju checks that an application is up 100%, it seems like it could easily just proxy that off to Kubernetes' livenessProbe and readinessProbe bits of functionality.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.