add-k8s ignores KUBECONFIG env var if ~/.kube/config exists

[George Kraft]

If I run `juju add-k8s` with the KUBECONFIG environment variable set, that variable seems to be ignored, but only if a kubeconfig exists in the default ~/.kube/config location.

I have the environment variable set, and kubectl commands work:

$ echo $KUBECONFIG
/home/gkk/kubeconfig
$ kubectl get po
No resources found in default namespace.

But juju add-k8s fails:

$ juju add-k8s --client k8s --skip-storage
ERROR making juju admin credentials in cluster: ensuring cluster role "juju-credential-20eae0b5" in namespace "kube-system": Get "https://10.246.154.123:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/juju-credential-20eae0b5": dial tcp 10.246.154.123:6443: connect: connection refused

The 10.246.154.123 IP comes from a stale kubeconfig located at ~/.kube/config whose cluster no longer exists. If I delete that kubeconfig, then try again, it now works:

$ rm ~/.kube/config
$ juju add-k8s --client k8s --skip-storage
k8s substrate added as cloud "k8s" with no configured storage provisioning capability.
You can now bootstrap to this cloud by running 'juju bootstrap k8s'.

$ juju version
3.1.2-genericlinux-amd64
$ snap list juju
Name Version Rev Tracking Publisher Notes
juju 3.1.2 22722 3.1/stable canonical✓ -

[George Kraft (cynerva)]

Command output with --debug:

$ juju add-k8s --client k8s --skip-storage --debug
16:13:38 INFO juju.cmd supercommand.go:56 running juju [3.1.2 f43d417390caeba51beece481aa7441ce3388519 gc go1.20.2]
16:13:38 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/22722/bin/juju", "add-k8s", "--client", "k8s", "--skip-storage", "--debug"}
16:13:38 DEBUG juju.environs.tools build.go:122 looking for: /snap/juju/22722/bin/juju
16:13:38 DEBUG juju.environs.tools versionfile.go:54 looking for sha256 f5f009bdc97044471643520b933824ff15dc9c5ed0f1ff42cbbcf319909955d0
16:13:38 DEBUG juju.kubernetes.provider detectcloud.go:33 failed to query local microk8s: "/var/snap/juju/22722/microk8s/credentials/client.config" does not exist: juju "3.1.2" can only work with strictly confined microk8s
ERROR making juju admin credentials in cluster: ensuring cluster role "juju-credential-9eef92ca" in namespace "kube-system": Get "https://10.246.154.123:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/juju-credential-9eef92ca": dial tcp 10.246.154.123:6443: connect: connection refused
16:13:38 DEBUG cmd supercommand.go:548 error stack:
Get "https://10.246.154.123:6443/apis/rbac.authorization.k8s.io/v1/clusterroles/juju-credential-9eef92ca": dial tcp 10.246.154.123:6443: connect: connection refused
github.com/juju/juju/caas/kubernetes/clientconfig.getOrCreateClusterRole:167:
github.com/juju/juju/caas/kubernetes/clientconfig.ensureJujuAdminServiceAccount:88: ensuring cluster role "juju-credential-9eef92ca" in namespace "kube-system"
github.com/juju/juju/cmd/juju/caas.(*AddCAASCommand).Run:503: making juju admin credentials in cluster

[Facundo Ciccioli (fandanbango)]

I hit this today. If for any reason you don't want to delete your stale .kube/config (like in, it isn't stale), you can pipe the config through stdin and it will use it:

cat $KUBECONFIG | juju add-k8s --client k8s --skip-storage