Application status could indicate service failures

[Barry Price]

After deploying the integrator to a cluster, 'juju status' suggests all is well:

App Version Status Scale Charm Store Rev OS Notes
openstack-integrator train active 1 openstack-integrator local 0 ubuntu

Unit Workload Agent Machine Public address Ports Message
openstack-integrator/0* active idle 13 10.15.210.18 Ready

And the storage class certainly shows up:

$ kubectl get sc --all-namespaces
NAME PROVISIONER AGE
cdk-cinder csi-cinderplugin 9d

But on closer inspection:

$ kubectl get po --all-namespaces | grep csi-cinder
kube-system csi-cinder-controllerplugin-0 2/4 CrashLoopBackOff 1300 45h
kube-system csi-cinder-nodeplugin-p6lzs 1/2 CrashLoopBackOff 3391 9d
kube-system csi-cinder-nodeplugin-schgg 1/2 CrashLoopBackOff 2622 9d

See LP:1868368 for the above issue itself, but is it possible (and if so, is it a good idea) for the charm to notice that the pods haven't started, and adjust application status to e.g. "blocked"?

[Martin Kalcok (martin-kalcok)]

Pods mentioned in the description (along with some other k8s resources) are currently "managed" by cdk-addons.
When "openstack-integrator" joins with "kubernetes-master", the k8s-master executes "cdk-addons" with parameter "openstack-enabled=true" which deploys appropriate k8s resources. However, there's no mechanism that would let "kubernetes-master" unit know which resources were created so it can not monitor if they were created successfully.

There's currently an effort to make cdk-addons obsolete and replace them with standalone charms. I think that instead of inventing a mechanism that would let "kubernetes-master" know which kubernetes resources were created by cdk-addons, we could instead make it a responsibility of "openstack-integrator" to create and monitor these resources.
In practice, this would involve moving k8s resource templates from cdk-addons to "openstack-integrator" and using k8s API to directly create resources from the charm.

There's only one drawback to this approach, we would have to require that the "openstack-integrator" is a subordinate charm of the "kubernetes-master" so that it can have direct access to the kube config, and use kubernetes API directly.

What do you all think about this approach?

[George Kraft (cynerva)]

Neither kubernetes-master nor openstack-integrator should be responsible for creating or monitoring the openstack-cloud-controller-manager and csi-cinder pods. The plan is to create new standalone k8s charms for those instead, with support for cross-model relations to openstack-integrator or kubernetes-master as needed. The new charms would be responsible for monitoring their own workloads.

I think the correct answer here is to mark this issue as "won't fix" since the pods in question are about to be deprecated. What do you think?