Juju controller restart change svc type from LoadBalancer to ClusterIP

[Giuseppe Petralia]

This has been observed on a COS deployed with juju 3.1.6 on top of charmed Kubernetes on top of OpenStack.

Charmed Kubernetes uses openstack-integrator to create Octavia LoadBalancers when a service with type=LoadBalancer is created on K8s.

The behavior observed is the following:

1. Traefik is deployed on K8s using juju controller 3.1.6

2. A svc type=LoadBalancer is created and an Octavia LB and a FIP are created in the underlying OpenStack

3. The user configures DNS entry and firewall rules assuming the ingress of their COS is that FIP

4. If traefik loses connectivity with the juju controller, upon reconnection the service type of traefik is changed from LoadBalancer -> ClusterIP

5. The svc type change triggers a deletion of the underlying Octavia LoadBalancer and its associated FIP

6. Traefik charm then patch the service changing the type back from ClusterIP -> LoadBalancer but a new Octavia LB is not created

7. The user patch the annotations of the traefik svc with the following

```
kubectl patch svc traefik -p '{"metadata":{"annotations":{"loadbalancer.openstack.org/load-balancer-id":null}}}'
```

8. A new Octavia LB and FIP are instantiated on the underlying OpenStack

9. All configurations done at step 3 are no longer valid, since a new FIP has been created on OpenStack

[Harry Pidcock (hpidcock)]

My feeling with this one is the charm should not be modifying the service created by Juju, instead creating its own. We don't make any guarantees that juju won't manipulate k8s resources that itself owns. There is some future work that we've internally discussed to improve Juju's patching of resources to preserve alterations made by other actors, but this isn't planned for a while.

[Aliaksandr Vasiuk (valexby)]

Hi,

The very much related github issue was confirmed by traeffik team:
https://github.com/canonical/traefik-k8s-operator/issues/319
Leaving it here for clarity.

Best Regards,
Alex.