Pod spec not able to process certain fields of CRDs

[Camille Rodriguez]

When creating a CRD through the kubernetes resources of the pod spec, an error is encountered when fields other than name, storage and served are present under "versions".

For example, with this CRD yaml file :

-----
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: certificates.networking.internal.knative.dev
  labels:
    serving.knative.dev/release: "v0.19.0"
    knative.dev/crd-install: "true"
spec:
  group: networking.internal.knative.dev
  versions:
    - name: v1alpha1
      served: true
      storage: true
      subresources:
        status: {}
      schema:
        openAPIV3Schema:
          type: object
          # this is a work around so we don't need to flush out the
          # schema for each version at this time
          #
          # see issue: https://github.com/knative/serving/issues/912
          x-kubernetes-preserve-unknown-fields: true
      additionalPrinterColumns:
        - name: Ready
          type: string
          jsonPath: ".status.conditions[?(@.type==\"Ready\")].status"
        - name: Reason
          type: string
          jsonPath: ".status.conditions[?(@.type==\"Ready\")].reason"
  names:
    kind: Certificate
    plural: certificates
    singular: certificate
    categories:
      - knative-internal
      - networking
    shortNames:
      - kcert
  scope: Namespaced

----

and with the kubernetes resources defined as :

k8s_resources={
            'kubernetesResources': {
                'customResourceDefinitions': [
                    {'name': crd['metadata']['name'], 'spec': crd['spec']}
                    for crd in yaml.safe_load_all(Path("files/serving-crds.yaml").read_text())
                ]
            }
}

Then The following error occurs:

 creating or updating custom resource definitions: ensuring custom resource definition "certificates.networking.internal.knative.dev": CustomResourceDefinition.apiextensions.k8s.io "certificates.networking.internal.knative.dev" is invalid: [spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:true, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(0xc017891b10), Subresources:(*apiextensions.CustomResourceSubresources)(0xc0126a97d0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:"Ready", Type:"string", Format:"", Description:"", Priority:0, JSONPath:".status.conditions[?(@.type==\"Ready\")].status"}}}}: per-version schemas may not all be set to identical values (top-level validation should be used instead), spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:true, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(0xc017891b10), Subresources:(*apiextensions.CustomResourceSubresources)(0xc0126a97d0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:"Ready", Type:"string", Format:"", Description:"", Priority:0, JSONPath:".status.conditions[?(@.type==\"Ready\")].status"}}}}: per-version subresources may not all be set to identical values (top-level subresources should be used instead), spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:true, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(0xc017891b10), Subresources:(*apiextensions.CustomResourceSubresources)(0xc0126a97d0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:"Ready", Type:"string", Format:"", Description:"", Priority:0, JSONPath:".status.conditions[?(@.type==\"Ready\")].status"}}}}: per-version additionalPrinterColumns may not all be set to identical values (top-level additionalPrinterColumns should be used instead)]

At first I thought it might be a problem specifically with the additionalPrinterColumns, so I removed that section from the CRD yaml file. But I still got a similar error of the sort:

controller-0: 20:45:47 ERROR juju.worker.caasunitprovisioner creating or updating custom resource definitions: ensuring custom resource definition "certificates.networking.internal.knative.dev": CustomResourceDefinition.apiextensions.k8s.io "certificates.networking.internal.knative.dev" is invalid: [spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:true, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(0xc0202d4680), Subresources:(*apiextensions.CustomResourceSubresources)(0xc0145cdbf0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: per-version schemas may not all be set to identical values (top-level validation should be used instead), spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:true, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(0xc0202d4680), Subresources:(*apiextensions.CustomResourceSubresources)(0xc0145cdbf0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: per-version subresources may not all be set to identical values (top-level subresources should be used instead)]

So I removed then the sections subresources, schema and additionalPrinterColumns from the CRD. It finally went through with success.

To compare the result, I was able to apply the original CRD file directly with kubectl without any issue. This points to a bug in how juju handles the CRD.

Let me know if any additional information is needed.

[Ian Booth (wallyworld)]

Looks like you are hitting this bug

https://github.com/kubernetes/kubernetes/issues/82443

Juju currently still uses "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1"
Seems in k8s 1.16 v1 became available.

The above k8s bug does have a work around on how to structure the CRD until Juju can be updated.

[Ian Booth (wallyworld)]

I can confirm that using the non-beta v1 k8s API does allow a pod spec with this CRD to be deployed. Since there's only one version entry, you can easily adjust the CRD to conform to the beta1 syntax to get things working with juju 2.8

  customResourceDefinitions:
    - name: certificates.networking.internal.knative.dev
      spec:
        scope: Namespaced
        group: networking.internal.knative.dev
        names:
          kind: Certificate
          plural: certificates
          singular: certificate
          categories:
            - knative-internal
            - networking
          shortNames:
            - kcert
        versions:
          - name: v1alpha1
            served: true
            storage: true
            subresources:
              status: {}
            schema:
              openAPIV3Schema:
                type: object
                # this is a work around so we don't need to flush out the
                # schema for each version at this time
                #
                # see issue: https://github.com/knative/serving/issues/912
                x-kubernetes-preserve-unknown-fields: true
            additionalPrinterColumns:
              - name: Ready
                type: string
                jsonPath: ".status.conditions[?(@.type==\"Ready\")].status"
              - name: Reason
                type: string
                jsonPath: ".status.conditions[?(@.type==\"Ready\")].reason"

[Camille Rodriguez (camille.rodriguez)]

Hi Ian, thanks for the quick update.

In the previous comment, you say that changing the version fixes the behavior. I tried changing the version name to either v1 or v1beta1 and I am still hitting the issue. The block you pasted still uses "v1alpha1". Which value did you mean to set it to?

[Ian Booth (wallyworld)]

Sorry, I meant I had to change the juju code to use the updated k8s SDK APIs. The CRD as posted was from the bug description - I just added it to a pod spec to test that updating the SDK APIs juju uses allows the newer CRD format to be used.

[Yang Kelvin Liu (kelvin.liu)]

https://github.com/juju/juju/pull/12515 will be landed to 2.8 to support both v1 and v1beta1 API versions for CRD. This bug should be fixed by this patch.