1.29 config default-storage ineffective

[Adam Dyess]

the default-storage annotation on a storage class can be marked on MANY storage classes, but is only effective when it there is a single storage class marked.

The kubernetes-control-plane config option `default-storage` seems like a good charm config for this setting -- to ensure the cluster has 1 or 0 storage class marked as such. It just doesn't work as it was only ever designed to mark storage classes created via cdk-addons.

Charms like aws-k8s-storage, gcp-k8s-storage, vsphere-cloud-controller, cinder-csi, ceph-csi, which apply storage classes themselves may overwrite any annotations kcp applies on those storage classes.

We should find a way for these "storage" charms to communicate their storage class with KCP (maybe via relation) so k8s can anoint the right one to mark it as the default storage class for the cluster

[Adam Dyess (addyess)]

perhaps sharing the default-storage class via kube-control is a good option here, and have kcp try to mark it with the annotation if the storage class exists