Juju fails to display LB address in k8s cloud

[Nick Veitch]

Juju status isn't picking up and or displaying in status the LB addresses from a GKE K8s cluster added as a k8s cloud.

- Create a GKE cluster
- juju add-k8s --gke the cloud
- Juju deploy Kubeflow (for example)

The google console reveals that the istio-gateway has been created and given external IPs correctly, but the juju status address field for these is blank.

[Nick Veitch (evilnick)]

Just some extra info, can confirm this also happens on Azure

[Ian Booth (wallyworld)]

juju uses the details of the service resource named after the application as that application's address in the juju model.

For LoadBalancer services, juju will use the k8s LoadBalancerIP attribute (if not empty), otherwise it picks the first Ingress address recorded on the service's Status.LoadBalancer attribute.

For ExternalName services, juju will use the ExternalName attribute.

For ClusterIP services, juj uwill use the ClusterIP attribute.

Can you use kubectl to see what type of service we are dealing with and what attributes it has?

[Thomas Miller (tlmiller)]

Hey Nick,

Can I ask how much time has passed to test juju status?

Also are you able to run a kubectl get svc <svc_name> -o yaml on the service for me and provide the contents back?

Cheers
tlm

[Nick Veitch (evilnick)]

Thanks for looking at this. I killed the clusters so I will run through it again and get more info

[Nick Veitch (evilnick)]

so kubectl gives:

kubectl get svc -n kubeflow istio-ingressgateway
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway LoadBalancer 10.41.174.99 35.187.22.184 15020:32184/TCP,80:30406/TCP,443:30228/TCP,15029:30008/TCP,15030:31494/TCP,15031:31650/TCP,15032:32142/TCP,15443:30215/TCP,15011:31596/TCP,8060:31720/TCP,853:31629/TCP 27m

with the external IP. Here is the YAML:

kubectl get svc -n kubeflow istio-ingressgateway -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/neg: '{"ingress":true}'
    juju.io/controller: ced6c50d-35f0-4027-873a-0a407609cf4c
    juju.io/model: 8339e9ae-1265-4359-8bf7-551c50c1d7a1
  creationTimestamp: "2020-11-30T16:15:44Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    juju-app: istio-ingressgateway
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:juju.io/controller: {}
          f:juju.io/model: {}
        f:labels:
          .: {}
          f:juju-app: {}
      f:spec:
        f:externalTrafficPolicy: {}
        f:ports:
          .: {}
          k:{"port":80,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
          k:{"port":443,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
          k:{"port":853,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
          k:{"port":8060,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
          k:{"port":15011,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
          k:{"port":15020,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
          k:{"port":15029,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
          k:{"port":15030,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
          k:{"port":15031,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
          k:{"port":15032,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
          k:{"port":15443,"protocol":"TCP"}:
            .: {}
            f:name: {}
            f:port: {}
            f:protocol: {}
            f:targetPort: {}
        f:selector:
          .: {...

[Thomas Miller (tlmiller)]

Hey Nick,

How are you? Mate I have ran up Kubeflow several times on GKE and can't produce the problem. Would you be able to give me the following information so I can try and help nail the problem down?

- Steps for how you are deploying Kubeflow
- Juju status output once it's full deployed
- kubectl get svc -n kubeflow
- export JUJU_DEV_FEATURE_FLAGS=developer-mode && juju dump-db'
- juju version

Cheers and hopefully this helps repo

[Nick Veitch (evilnick)]

Ok. I will try to get to that today. My update for now is that I got it to work, but so far only by creating a default GKE cluster - i.e. using the default us-central1 zone and the default version of kubernetes on GKE, which is pretty old - 1.16.13
So now I am thinking the issue is maybe to do with the Kubernetes version or the zone

[Thomas Miller (tlmiller)]

Fixed landed on 2.8 branch.

[Thomas Miller (tlmiller)]

PR: https://github.com/juju/juju/pull/12401