impossible to delete k8s services when octavia is detected

[Junien F]

Hi,

I investigated a k8s service that wouldn't delete today, the "describe" was like this :
https://pastebin.ubuntu.com/p/wWjVSH3MZ6/

As it turns out, by default, openstack users don't have access to Octavia (at least in Ussuri, which is the openstack version we're using here) :
https://docs.openstack.org/octavia/ussuri/configuration/policy.html (first line).

I had to add the "load-balancer_observer" role to the openstack user to allow the k8s service to be deleted.

If a k8s service can be created without octavia access, I think one should be able to delete it as well without octavia access.

Thanks

PS : the version of openstack-integrator I have is :
App Version Status Scale Charm Channel Rev Exposed Message
openstack-integrator wallaby active 1 openstack-integrator stable 140 no Ready

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.1", GitCommit:"e4d4e1ab7cf1bf15273ef97303551b279f0920a9", GitTreeState:"clean", BuildDate:"2022-09-16T02:32:15Z", GoVersion:"go1.19", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.14", GitCommit:"0f77da5bd4809927e15d1658fb4aa8f13ad890a5", GitTreeState:"clean", BuildDate:"2022-06-17T21:57:40Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.25) and server (1.21) exceeds the supported minor version skew of +/-1

[George Kraft (cynerva)]

Interesting. The code that openstack-integrator uses to decide if Octavia should be used or not is in detect_octavia[1], which checks for the presence of octavia in `openstack catalog list`. I'm guessing this does not take into account the roles of the user.

That said, we require Octavia[2], and we require it because the upstream openstack-cloud-controller-manager project that we use to manage k8s LoadBalancer services also requires Octavia[3].

I recommend two fixes:
1. Update the openstack-integration documentation to list required user roles
2. Make the openstack-integrator charm check and verify user roles, and enter Blocked status if required roles are missing

[1]: https://github.com/juju-solutions/charm-openstack-integrator/blob/91984db6176c005340429061c2aef02654b543ad/lib/charms/layer/openstack.py#L131
[2]: https://ubuntu.com/kubernetes/docs/openstack-integration
[3]: https://github.com/kubernetes/cloud-provider-openstack/blob/ec0e52924d107a039524b29e19cb11937b37961e/docs/openstack-cloud-controller-manager/using-openstack-cloud-controller-manager.md

[George Kraft (cynerva)]

After discussing this with Junien, I understand that LoadBalancer support is not needed or desired in this cluster, so Octavia roles should not be required. The relation to openstack-integrator is still needed to provide PersistentVolume support via Cinder CSI.

We need to expose a way to disable LoadBalancer support. We can do this by passing the [LoadBalancer] enabled=false config[1] to openstack-cloud-controller-manager. We may need to do this in two places: one in layer-kubernetes-common's generate_openstack_cloud_config[2] and one in openstack-cloud-controller-operator's build_cloud_config[3].

New recommended fix:
1. Add a config option to openstack-integrator to disable LoadBalancer integration
2. Make the openstack-integrator charm check and verify user roles, and enter Blocked status if required roles are missing, BUT don't require Octavia roles if LoadBalancer support is disabled
3. Update the openstack-integration documentation to clarify what roles are required and when

[1]: https://github.com/kubernetes/cloud-provider-openstack/blob/ec0e52924d107a039524b29e19cb11937b37961e/docs/openstack-cloud-controller-manager/using-openstack-cloud-controller-manager.md#load-balancer
[2]: https://github.com/charmed-kubernetes/layer-kubernetes-common/blob/2d388da1bbdc78a82bb7fc649fe9a0cb09af0803/lib/charms/layer/kubernetes_common.py#L550
[3]: https://github.com/canonical/openstack-cloud-controller-operator/blob/5eafef3506344a741d81d26bbb4c6f4fb7caeff8/src/backend.py#L79

[George Kraft (cynerva)]

PRs:
https://github.com/juju-solutions/interface-openstack-integration/pull/13
https://github.com/juju-solutions/charm-openstack-integrator/pull/67
https://github.com/charmed-kubernetes/layer-kubernetes-common/pull/40
https://github.com/canonical/openstack-cloud-controller-operator/pull/3