CSIMigrationAWS and CSIMigrationGCE deprecated in k8s 1.25

[George Kraft]

Test run: https://solutions.qa.canonical.com/testruns/testRun/bce25cca-023e-4585-8541-4cc86a102515

kubernetes-control-plane, kubernetes-worker, and calico are all stuck in Waiting status:

k8s-cp - Waiting for auth-webhook tokens
k8s-worker - Waiting for cluster credentials.
calico - Waiting to retry Calico node configuration

kube-apiserver fails to start with:

Error: invalid argument "CSIMigrationAWS=false" for "--feature-gates" flag: cannot set feature gate CSIMigrationAWS to false, feature is locked to true

Per release notes[1], the CSIMigrationAWS and CSIMigrationGCE features are now GA and locked to true. We will need to remove the code in kubernetes-control-plane and kubernetes-worker that sets those.

[1]: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.25.md#feature

[George Kraft (cynerva)]

Since we're no longer able to disable those feature gates, in-tree cloud provider storage on AWS and GCE will no longer work.

[George Kraft (cynerva)]

PRs:
https://github.com/charmed-kubernetes/charm-kubernetes-control-plane/pull/244
https://github.com/charmed-kubernetes/layer-kubernetes-common/pull/32

[George Kraft (cynerva)]

Cherry-picks to release_1.25 branches:
https://github.com/charmed-kubernetes/charm-kubernetes-control-plane/commit/40c0975abc28330c08a2aba0e30f7e0cf864b595
https://github.com/charmed-kubernetes/layer-kubernetes-common/commit/fbf594f868d91c8407d12423aef38795ca294ad2

[George Kraft (cynerva)]

The fix has been released to the latest/beta channels of kubernetes-control-plane (rev 183) and kubernetes-worker (rev 52).

[Adam Dyess (addyess)]

Workaround for AWS Storage:

Deploy the cluster with the aws-integrator charm as normal

Create a secret for use by the out-of-tree storage controllers
> kubectl create secret generic aws-secret \
    --namespace kube-system \
    --from-literal "key_id=${AWS_ACCESS_KEY_ID}" \
    --from-literal "access_key=${AWS_SECRET_ACCESS_KEY}"

Deploy the out-of-tree storage controller
> kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-1.11"

Create a storage class to use the provisioner (ebs.csi.aws.com)

See this for more details
https://github.com/kubernetes-sigs/aws-ebs-csi-driver/tree/master/examples/kubernetes/dynamic-provisioning

[Kevin W Monroe (kwmonroe)]

Workaround per comment #5 gets us past the bug on AWS. Future work will include charming aws-ebs-csi to be delivered in a ckX bugfix release or next GA.

Investigation will continue on GCP to confirm a similar workaround / charm requirement.

[George Kraft (cynerva)]

Is there remaining work for the aws-k8s-storage and gcp-k8s-storage charms? Can those be marked Fix Committed for 1.26?