Juju 2.9 failing to create ClusterRoleBinding

[Kenneth Koski]

I am deploying charms as part of this CI run:

https://github.com/canonical/notebook-operators/runs/2953294102

If you examine the output from the `kubectl get pods -A -oyaml` step, you can see that a `jupyter-ui` Pod was created, with `serviceAccount: jupyter-ui`. If you examine the `kubectl get clusterroles -A -oyaml`, you can also see that a `kubeflow-jupyter-ui` ClusterRole was created. However, if you examine the `kubectl get clusterrolebindings -A -oyaml` step, no corresponding ClusterRoleBinding was created, meaning that the `jupyter-ui` Pod doesn't get any permissions, and returns errors for any requests, failing the CI process.

This only seems to happen sometimes, as the the other run for that commit completed successfully. You can view the errors that are being returned by downloading the `selenium-har` artifact and uploading it to http://www.softwareishard.com/har/viewer/. You'll see that there are several 403 responses from jupyter-ui, as it can't list various Kubernetes resources that it requires to function.

[John A Meinel (jameinel)]

In the run link there is a link to Artifacts, which has the 'introspection-reports'. Downloading that and expanding it does have a 'juju' subdirectory, which includes the juju controller debug logs.

I don't see a smoking gun in there (of some sort of api call that failed, which we aren't retrying, etc).

Looking at the log I do see a lot of warnings about:
4abab061-5a6d-44c2-8b31-752631a38a3f: controller-0 2021-06-30 14:42:44 INFO juju.kubernetes.klog klog.go:56 apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apie
xtensions.k8s.io/v1 CustomResourceDefinition

However https://bugs.launchpad.net/juju/+bug/1921553 seems to say that we should be trying to use apiextensions.k8s.io/v1 first.

(I don't know that this is the cause of the issue, but it certainly would be good to get rid of the noise in the logs to help identify the actual issue.)

[Yang Kelvin Liu (kelvin.liu)]

https://github.com/juju/juju/pull/13129 will be landed to 2.9 to fix this issue