Can't specify apiVersion for resources listed in pod spec's kubernetesResources

[Kenneth Koski]

I have a CRD that looks like this:

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: viewers.kubeflow.org
spec:
  group: kubeflow.org
  names:
    kind: Viewer
    listKind: ViewerList
    plural: viewers
    shortNames:
    - vi
    singular: viewer
  scope: Namespaced
  versions:
  - name: v1beta1
    served: true
    storage: true

Note that the apiVersion is listed as "apiextensions.k8s.io/v1beta1". I pass that file in to the pod spec like this:

k8s_resources={"kubernetesResources": { "customResourceDefinitions": [
    {"name": crd["metadata"]["name"], "spec": crd["spec"]}
    for crd in yaml.safe_load_all(Path("src/crds.yaml").read_text())
]}}

There is no way for me to specify the apiVersion, and either Juju or Kubernetes is defaulting to "apiextensions.k8s.io/v1". This is an issue because that apiVersion requires the use of the "schema" property, triggering errors like this:

creating or updating custom resource definitions: ensuring custom resource definition "scheduledworkflows.kubeflow.org" with version "v1": CustomResourceDefinition.apiextensions.k8s.io "scheduledworkflows.kubeflow.org" is invalid: spec.versions[0].schema.openAPIV3Schema: Required value: schemas are required

[Ian Booth (wallyworld)]

Can you migrate the charm to use the v1 api?
We'd rather not have to try and support the beta api.

[Kenneth Koski (knkski)]

As a workaround, I was able to add this to the CRD definition to create it on recent Kubernetes versions:

  validation:
    openAPIV3Schema:
      type: object
      x-kubernetes-preserve-unknown-fields: true

This does happen to fix the issue that I encountered. However, if an upstream service were to attempt to list the CRDs themselves (as opposed to the custom resources), it would fail due to the apiVersion not matching.

[Cory Johns (johnsca)]

It looks like apiextensions.k8s.io/v1 was introduced in 1.16, at which point apiextensions.k8s.io/v1beta1 was deprecated and will be removed entirely in 1.22 [1].

From what I can tell, when you create a CRD with apiextensions.k8s.io/v1beta1, it is automatically converted by K8s to apiextensions.k8s.io/v1 with the preserveUnknownFields automatically set [1]. Requests to list or get CRDs will return them no matter what version they were created with and creating resources based on those CRDs will work the same regardless of which version the CRD itself was created with.

[1]: https://kubernetes.io/docs/reference/using-api/deprecation-guide/#customresourcedefinition-v122

[2]: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#field-pruning

[Yang Kelvin Liu (kelvin.liu)]

Hi Ken,
The current multi-version support was a quick win, and it just simply try to parse the spec using the current latest version(in this case, it's v1) then retry using v1beta1 if it was failed.
As you have already figured out, you will be able to use v1beta1 if the spec has some v1beta1 specific fields(like the top-level `validation`). It is probably the workaround to solve the issue.
More accurate multi-version support is definitely a good feature to implement in the future.

As Cory mentioned, any versions of resources will be all converted to the latest version after deployed to the k8s side. And as far as I know, the k8s list/delete/get operations actually ignores the API versions.
I don't think it will have any issues when the upstream service tries to list these CRDs.

[Canonical Juju QA Bot (juju-qa-bot)]

This Medium-priority bug has not been updated in 60 days, so we're marking it Low importance. If you believe this is incorrect, please update the importance.