When creating a CRD through the kubernetes resources of the pod spec, an error is encountered when fields other than name, storage and served are present under "versions".
For example, with this CRD yaml file :
-----
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: certificates.networking.internal.knative.dev
labels:
serving.knative.dev/release: "v0.19.0"
knative.dev/crd-install: "true"
spec:
group: networking.internal.knative.dev
versions:
- name: v1alpha1
served: true
storage: true
subresources:
status: {}
schema:
openAPIV3Schema:
type: object
# this is a work around so we don't need to flush out the
# schema for each version at this time
#
# see issue: https://github.com/knative/serving/issues/912
x-kubernetes-preserve-unknown-fields: true
additionalPrinterColumns:
- name: Ready
type: string
jsonPath: ".status.conditions[?(@.type==\"Ready\")].status"
- name: Reason
type: string
jsonPath: ".status.conditions[?(@.type==\"Ready\")].reason"
names:
kind: Certificate
plural: certificates
singular: certificate
categories:
- knative-internal
- networking
shortNames:
- kcert
scope: Namespaced
----
and with the kubernetes resources defined as :
k8s_resources={
'kubernetesResources': {
'customResourceDefinitions': [
{'name': crd['metadata']['name'], 'spec': crd['spec']}
for crd in yaml.safe_load_all(Path("files/serving-crds.yaml").read_text())
]
}
}
Then The following error occurs:
creating or updating custom resource definitions: ensuring custom resource definition "certificates.networking.internal.knative.dev": CustomResourceDefinition.apiextensions.k8s.io "certificates.networking.internal.knative.dev" is invalid: [spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:true, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(0xc017891b10), Subresources:(*apiextensions.CustomResourceSubresources)(0xc0126a97d0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:"Ready", Type:"string", Format:"", Description:"", Priority:0, JSONPath:".status.conditions[?(@.type==\"Ready\")].status"}}}}: per-version schemas may not all be set to identical values (top-level validation should be used instead), spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:true, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(0xc017891b10), Subresources:(*apiextensions.CustomResourceSubresources)(0xc0126a97d0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:"Ready", Type:"string", Format:"", Description:"", Priority:0, JSONPath:".status.conditions[?(@.type==\"Ready\")].status"}}}}: per-version subresources may not all be set to identical values (top-level subresources should be used instead), spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:true, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(0xc017891b10), Subresources:(*apiextensions.CustomResourceSubresources)(0xc0126a97d0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition{apiextensions.CustomResourceColumnDefinition{Name:"Ready", Type:"string", Format:"", Description:"", Priority:0, JSONPath:".status.conditions[?(@.type==\"Ready\")].status"}}}}: per-version additionalPrinterColumns may not all be set to identical values (top-level additionalPrinterColumns should be used instead)]
At first I thought it might be a problem specifically with the additionalPrinterColumns, so I removed that section from the CRD yaml file. But I still got a similar error of the sort:
controller-0: 20:45:47 ERROR juju.worker.caasunitprovisioner creating or updating custom resource definitions: ensuring custom resource definition "certificates.networking.internal.knative.dev": CustomResourceDefinition.apiextensions.k8s.io "certificates.networking.internal.knative.dev" is invalid: [spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:true, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(0xc0202d4680), Subresources:(*apiextensions.CustomResourceSubresources)(0xc0145cdbf0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: per-version schemas may not all be set to identical values (top-level validation should be used instead), spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion{apiextensions.CustomResourceDefinitionVersion{Name:"v1alpha1", Served:true, Storage:true, Deprecated:false, DeprecationWarning:(*string)(nil), Schema:(*apiextensions.CustomResourceValidation)(0xc0202d4680), Subresources:(*apiextensions.CustomResourceSubresources)(0xc0145cdbf0), AdditionalPrinterColumns:[]apiextensions.CustomResourceColumnDefinition(nil)}}: per-version subresources may not all be set to identical values (top-level subresources should be used instead)]
So I removed then the sections subresources, schema and additionalPrinterColumns from the CRD. It finally went through with success.
To compare the result, I was able to apply the original CRD file directly with kubectl without any issue. This points to a bug in how juju handles the CRD.
Let me know if any additional information is needed.
Looks like you are hitting this bug
https:/ /github. com/kubernetes/ kubernetes/ issues/ 82443
Juju currently still uses "k8s.io/ apiextensions- apiserver/ pkg/apis/ apiextensions/ v1beta1"
Seems in k8s 1.16 v1 became available.
The above k8s bug does have a work around on how to structure the CRD until Juju can be updated.