k8s pod-spec-set charm erroring with "server could not find the requested resource"

[Tom Haddon]

Following the deployment instructions on https://charmhub.io/mattermost-k8s as follows in a clean juju k8s model:

```
juju deploy postgresql-k8s
juju deploy mattermost-k8s
juju relate mattermost-k8s postgresql-k8s:db
```

I get the following error for the `mattermost-k8s` application:
```
creating or updating ingress resources: ensuring ingress "mattermost-k8s-ingress" with version "v1beta1": the server could not find the requested resource
```

Juju version is 2.9.9, microk8s version is v1.22.0.

This doesn't seem to be charm-specific as there's no mention of v1beta1 in the charm source code, and I same a similar error deploying cs:dex-auth:
```
creating or updating custom resource definitions: ensuring custom resource definition "authcodes.dex.coreos.com" with version "v1beta1": the server could not find the requested
resource
```

[Tom Haddon (mthaddon)]

And confirmed the same error on 2.9.10, fwiw.

[Tom Haddon (mthaddon)]

And confirmed this is related to the version of microk8s. If I follow the same steps on v1.21.3 I don't get this error and the charms deploy fine. https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/#api-changes is likely relevant here.

[Dominik Fleischmann (dominik.f)]

Regarding the ingress error in the mattermost charm, this resource is created by adding the ingress information into the "ingressResource" section under the "kubernetesResources" section of the Pod Spec.

This seems to be creating an "Ingress" resource still using the "v1beta1" version of the API which has been deprecated with 1.22 (see Tom's link in the comment above).

Regarding the dex issue, we use an existing CRD from which we pass through the "metadata.name" and the "spec" section to Pod Spec. The version (although our existing CRD also uses v1beta1) is set by juju which seems to be using v1beta1 which also is deprecated with 1.22.

Original CRD manifest: https://github.com/canonical/dex-auth-operator/blob/main/resources/crds.yaml

What is passed through in Pod Spec: https://github.com/canonical/dex-auth-operator/blob/3ed73d0c9385fbfaeaadfcef53870c4b8443c740/src/charm.py#L200

[John A Meinel (jameinel)]

It sounds less like it was deprecated and more that it was completely
removed with 1.22. Can you confirm? If it was deprecated it would work, but
there would be some sort of warning.

On Mon, Aug 9, 2021 at 6:20 AM Dominik Fleischmann <
<email address hidden>> wrote:

> Regarding the ingress error in the mattermost charm, this resource is
> created by adding the ingress information into the "ingressResource"
> section under the "kubernetesResources" section of the Pod Spec.
>
> This seems to be creating an "Ingress" resource still using the
> "v1beta1" version of the API which has been deprecated with 1.22 (see
> Tom's link in the comment above).
>
> Regarding the dex issue, we use an existing CRD from which we pass
> through the "metadata.name" and the "spec" section to Pod Spec. The
> version (although our existing CRD also uses v1beta1) is set by juju
> which seems to be using v1beta1 which also is deprecated with 1.22.
>
> Original CRD manifest: https://github.com/canonical/dex-auth-
> operator/blob/main/resources/crds.yaml
>
> What is passed through in Pod Spec: https://github.com/canonical/dex-
> auth-
> operator/blob/3ed73d0c9385fbfaeaadfcef53870c4b8443c740/src/charm.py#L200
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1939278
>
> Title:
> k8s pod-spec-set charm erroring with "server could not find the
> requested resource"
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1939278/+subscriptions
>
>

[Tom Haddon (mthaddon)]

https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/#api-changes confirms these are removals not deprecations.

[John A Meinel (jameinel)]

We should at least investigate this, because we thought we handled the deprecation of v1beta1 => v1. It may be a case of 'compat' mode that got triggered because of something in the pod spec, or it may be that we just missed fixing this one.

[Yang Kelvin Liu (kelvin.liu)]

hi Tom,
Juju does support both v1beta1 and v1.
The issue here is that the charm submits a v1beta1 ingress spec to Juju via set-pod-spec.
But the k8s cluster does not support v1beta1 anymore.
So the fix could be to upgrade the ingress spec to v1 in the charm and set the `.deployment.min-version==1.19` in metadata.yaml.
(ingress v1 was introduced in k8s version 1.19)

thanks

[Tom Haddon (mthaddon)]

I'm not sure if you're saying we need to make charm changes for this (if so, I think this would affect _every_ pod-spec-set charm that defines ingress). The charm has no mention of v1beta1 ingress, it simply passes an "ingressResources" dictionary within "kubernetesResources" that's then passed to `self.model.pod.set_spec`.

We'd definitely prefer if this didn't require charm changes, but obviously if that's unavoidable then let us know.

[Yang Kelvin Liu (kelvin.liu)]

yeah, unfortunately, I think we either need to upgrade the podspec to v1 format or downgrade the k8s cluster to <1.21.
This is purely a k8s API version support issue and it is not controlled by Juju.

As you can see the spec in the charm is using the old format(v1beta1).
- https://docs.konghq.com/kubernetes-ingress-controller/1.3.x/concepts/ingress-versions/
- https://git.launchpad.net/charm-k8s-mattermost/tree/src/charm.py#n300

Cheers

[Tom Haddon (mthaddon)]

Sorry, I don't quite understand what we're being asked to do here. I don't see anywhere in the pod-spec-set code where we reference v1 or v1beta1 API versions, so what are we supposed to change here?

[John A Meinel (jameinel)]

So the issue for dex-auth-operator is clear that it was using the v1beta1 api. From the link:
https://github.com/canonical/dex-auth-operator/blob/main/resources/crds.yaml

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: authcodes.dex.coreos.com
spec:
...

Looking at: https://git.launchpad.net/charm-k8s-mattermost/tree/src/charm.py

I don't know which bit is v1 vs v1beta. I know things like CRDs have changed their representation for K8s v1.22.

The code Juju uses is:
https://github.com/juju/juju/blob/72e225cfaa4d1f2714d3048c7b92086bed1eb662/caas/kubernetes/provider/specs/customresourcedefinition.go#L31

Which is 'try to parse as SpecV1 in strict mode' else fall back to SpecV1Beta mode.

The specs involved are:
https://github.com/kubernetes/apiextensions-apiserver/blob/ebfb0f733cdfccc3c866e2cd0ff575b2a7870ce6/pkg/apis/apiextensions/v1beta1/types.go#L41
vs
https://github.com/kubernetes/apiextensions-apiserver/blob/ebfb0f733cdfccc3c866e2cd0ff575b2a7870ce6/pkg/apis/apiextensions/v1/types.go#L41

Doing a diff is about 200 lines (a lot of comments). Some of the bits are:
Removing "Version at the top level" in favor of Versions as a list, etc.

Digging a bit more.

[Barry Price (barryprice)]

I've come across the same issue with another k8s charm today - if the pod spec syntax needs to be rewritten then we can deal with that, but I don't quite understand which parts need to change.

This seems likely to crop up again for more charms in future, so I think it's worth figuring out a migration guide to point people to, whether in this bug or elsewhere.

Charm code in question is here, anchored to the stanza where we set up our ingressResource:

https://git.launchpad.net/charm-k8s-discourse/tree/src/charm.py?h=v2.8.0.beta7-20.04#n108

[Yang Kelvin Liu (kelvin.liu)]

Hi Tom, Barry:
As John pointed out, Juju is 'try to parse as SpecV1 in strict mode' else fall back to SpecV1Beta mode.

So, the charm pod spec does not have a `version` field configurable.
As mentioned in the #9 comment, you can find the `.spec` `diff` between v1 and v1beta1.
Just change the `.spec` in the charm to v1 format.

[Barry Price (barryprice)]

Sure - and here is that diff between John's two spec links:

https://paste.ubuntu.com/p/HVmn2j3wNn/

It does seem like the Version->Versions change might be relevant here, although that's the pod spec itself rather than the ingressResource I linked to.

I just can't tell from that diff exactly what change is required here, it'd be useful to have something to point others to when they run into this issue. I'll try a few things and see what I can come up with.

[Tom Haddon (mthaddon)]

I've been working on an MP against the mattermost charm to see if I can get the pod spec translated over as needed. I haven't yet been able to test this, and can't yet see how to define TLS and annotations, but https://code.launchpad.net/~mthaddon/charm-k8s-mattermost/+git/charm-k8s-mattermost/+merge/411033 is what I have so far in case anyone else is looking at this in parallel.

[Ian Booth (wallyworld)]

This PR https://github.com/juju/juju/pull/13493 will automatically translate beta or v1 ingress resources to match what's supported by the cluster. I recommend writing charms to use v1 and juju will take care of translating back to beta1 for older clusters.

[Ian Booth (wallyworld)]

I'll mark as fix committed, if CRDs are an issue, can we raise a new bug for that.