`kubectl expose` of cdk-addons deployments is reverted by cdk-addons

[Andrey Grebennikov]

Deploying CDK on bare metal servers with MetalLB (helm-based).
Exposing metric server, prometheus and kubernetes-dashboard with type "LoadBalancer".
After 1-2 minutes each service is being deleted and I have to-recreate it again.
The logs of k8s-master only show this:

Jun 22 23:04:32 juju-4efb95-0-lxd-4 kube-apiserver.daemon[2861]: I0622 23:04:32.413194 2861 httplog.go:90] DELETE /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/kubernetes-dashboard: (19.223639ms) 200 [kubectl/v1.16.10 (linux/amd64) kubernetes/f3add64 192.168.122.30:52454]
Jun 22 23:04:32 juju-4efb95-0-lxd-4 kube-apiserver.daemon[2861]: I0622 23:04:32.620733 2861 store.go:229] deletion of /registry/services/endpoints/kubernetes-dashboard/dash-ext failed because of a conflict, going to retry
Jun 22 23:04:32 juju-4efb95-0-lxd-4 kube-apiserver.daemon[2861]: I0622 23:04:32.622498 2861 httplog.go:90] DELETE /api/v1/namespaces/kubernetes-dashboard/endpoints/dash-ext: (28.287606ms) 404 [kube-controller-manager/v1.16.11 (linux/amd64) kubernetes/436254b/system:serviceaccount:kube-system:endpoint-controller 192.168.122.30:52100]
Jun 22 23:04:32 juju-4efb95-0-lxd-4 kube-apiserver.daemon[2861]: I0622 23:04:32.651591 2861 httplog.go:90] DELETE /api/v1/namespaces/kubernetes-dashboard/services/dash-ext: (83.684494ms) 200 [kubectl/v1.16.10 (linux/amd64) kubernetes/f3add64 192.168.122.30:52456]

[George Kraft (cynerva)]

Can you describe in more detail what you're trying to do? We need reproduction steps.

If you're trying to expose services that are created by cdk-addons, then I would expect cdk-addons to revert those changes. We don't currently support changing the service type of addons provided by CDK.

If you're creating your own services, please post reproduction steps that include YAML for the services you're creating.

[Andrey Grebennikov (agrebennikov)]

George, I'm creating another services to get access to the dashboard and the metric server/prometheus from the external hosts.

kubectl expose -n kubernetes-dashboard deploy kubernetes-dashboard --name dash-ext --type=LoadBalancer

This is one of the examples, and it applies to other ones in the kube-system namespace (deployment metrics-server-v0.3.6 for example).

Every additional service for deployments created by CDK itself is being removed within a few minutes, sometimes within a few seconds.

This is very annoying because it doesn't allow to connect to the metric server from the outside, and users lose access to the dashboard after a while (as original exposure via ClusterIP is only available from inside the cluster)

[George Kraft (cynerva)]

Thanks Andrey. This behavior is a bit surprising, but I can reproduce it.

Apparently when you `kubectl expose` a deployment, the service that gets created inherits labels from the deployment:

$ kubectl expose -n kubernetes-dashboard deploy kubernetes-dashboard --name dash-ext --type=LoadBalancer
$ kubectl get svc -n kubernetes-dashboard dash-ext -o yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    cdk-addons: "true"
    cdk-restart-on-ca-change: "true"
    k8s-app: kubernetes-dashboard
...

The cdk-addons label causes cdk-addons to think this is a resource that it's supposed to manage. It doesn't recognize the service, however, so it deletes it.

You can work around this by overriding the cdk-addons label when you expose the service:

kubectl expose -n kubernetes-dashboard deploy kubernetes-dashboard --name dash-ext --type=LoadBalancer -l cdk-addons=false