kubernetes caas operator doesn't reveal errors

Bug #1946999 reported by Adam Dyess
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Triaged
High
Unassigned

Bug Description

while deploying kubeflow atop microk8s, i encountered an issue where the controller didn't provide sufficient logs to what prevented the pods from starting.

controller-0: 18:34:02 ERROR juju.worker.dependency "caas-operator-provisioner" manifold worker returned unexpected error: failed to generate operator config for "kfp-api": updating agent config: no existing agent conf found and no new password generated for "kfp-api" operator

This same message ^^ above was presented for most of the operators starting in the model.

Studying the controller code around this [error](
https://github.com/juju/juju/blob/41051adf37af635239a42f0a3842e62f01d7ca77/worker/caasoperatorprovisioner/worker.go#L244) i discovered that there was a linkage with the kubernetes implementation of "Operator" [here](https://github.com/juju/juju/blob/41051adf37af635239a42f0a3842e62f01d7ca77/caas/kubernetes/provider/operator.go#L635)

I would ask that the controller bubble out errors from the Operator classes. In my case, the StatefulSet the k8s provider operator uses to create the pods was in error, and had a clear reason as to the issue. it wasn't clear from any of the debug-logs in either the controller or application models to point to an issue with the statefulsets being in a retry error loop.

Adam Dyess (addyess)
tags: added: kubernetes
tags: added: statefulset
tags: added: kubeflow
tags: added: ju-21
tags: removed: ju-21
Revision history for this message
Harry Pidcock (hpidcock) wrote :

Both the handling of this should be better and the error reporting should be more pronounced.

If you see this again, could we get the kubectl yaml output of the operator StatefulSet and the ConfigMap (if there is any).

Changed in juju:
importance: Undecided → High
status: New → Triaged
milestone: none → 3.0.0
Changed in juju:
milestone: 3.0.0 → 3.0.1
Changed in juju:
milestone: 3.0.1 → 3.0.2
Changed in juju:
milestone: 3.0.2 → 3.0.3
Changed in juju:
milestone: 3.0.3 → 3.0.4
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.