Juju+microk8s: very weird behaviour

[Jordan Barrett]

Yesterday I installed microk8s to work with Juju following this tutorial:
https://juju.is/docs/olm/microk8s
and it all worked fine.

Today I noticed some very strange behaviour from Juju. When I called `juju status`, it would hang for a few minutes, before finally returning
```
ERROR cannot connect to k8s api server; try running 'juju update-k8s --client <k8s cloud name>'
```

`juju controllers` worked fine, showing
```
Controller ...
localhost-localhost ...
microk8s-localhost* ...
```
however I could not `juju switch` to localhost, as I got the same error:
```
ERROR refreshing models cache: cannot connect to k8s api server; try running 'juju update-k8s --client <k8s cloud name>'
```

I tried to do as the error suggested, but ironically, I could not run `juju clouds` to get the cloud name, because it gave the same error. So did trying to remove the k8s cloud.

I tried to run `juju update-k8s --client <k8s cloud name>` with several different cloud names, but kept getting errors. (I'm positive one of them was the actual cloud name).

The only command that gave me a useful error message was `juju bootstrap`:
```
ERROR Insufficient permissions to access MicroK8s.
You can either try again with sudo or add the user jordan to the 'microk8s' group:

    sudo usermod -a -G microk8s jordan
    sudo chown -f -R jordan ~/.kube

After this, reload the user groups either via a reboot or by running 'newgrp microk8s'.
```
`newgrp microk8s` didn't help, but opening another terminal window did. I tried to bootstrap a new LXD controller, but it hung on the `Running machine configuration script` step, and I had to kill it after 20 mins. Not sure if that is connected.

Anyway, I still have no idea what happened, and whether this is a bug with Juju, with microk8s, or something else. At the very least, the error messages could be improved, as it seems this was a problem with system permissions, and only `juju bootstrap` told me that, while the other commands gave misleading errors.

[joe (emcp)]

I can attest, I too have started using microk8s and juju for first time.. Setup `kubeflow` and it still runs fine.. but the juju client is giving these same errors reported by OP

```
$ juju models
cannot connect to k8s api server; try running 'juju update-k8s --client <k8s cloud name>'
ERROR cannot connect to k8s api server; try running 'juju update-k8s --client <k8s cloud name>'
$ juju controllers
Use --refresh option with this command to see the latest information.

Controller Model User Access Cloud/Region Models Nodes HA Version
localhost-localhost stonks-beta admin superuser localhost/localhost 4 9 none 2.9.22
microk8s-localhost* kubeflow admin superuser microk8s/localhost 2 1 - 2.9.22

$ juju update-k8s --client microk8s-localhost
ERROR cloud microk8s-localhost not found
$ juju models
cannot connect to k8s api server; try running 'juju update-k8s --client <k8s cloud name>'
ERROR cannot connect to k8s api server; try running 'juju update-k8s --client <k8s cloud name>'
$ juju status
ERROR cannot connect to k8s api server; try running 'juju update-k8s --client <k8s cloud name>'

```

[joe (emcp)]

update on my post.. I didn't realize the cloud name for microk8s was simply that... once I got that solved.. I successfully performed the command needed and all seems well now

`juju update-k8s --client microk8s`

[Jordan Barrett (barrettj12)]

I'm also getting this error from the following:
```
$ microk8s stop
$ juju add-model foo
ERROR opening API connection: cannot connect to k8s api server; try running 'juju update-k8s --client <k8s cloud name>'
```
but the problem here is not an out-of-date k8s, it's that the server is not running. Looks like the error diagnosis could be a lot better here.

[Amanda Hager Lopes de Andrade Katz (amandahla)]

I had this exact same situation.

- Installed juju yesterday
- Everything was fine
- Turned off the machine and turned on today
- juju status hanging trying to connect with something:
```
tcp 0 1 192.168.0.105:43940 10.172.70.75:16443 SYN_SENT 47408/juju
```
- Checked my /etc/hosts, user permissions and .kube
```
id
uid=1000(amanda) gid=1000(amanda) groups=1000(amanda),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),122(lpadmin),134(lxd),135(sambashare),999(microk8s)

drwxr-x--- 3 amanda microk8s 4096 out 17 18:09 .kube

cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 amoreira indico.local
```
- Checked if microk8s as up and running
```
microk8s status
microk8s is running

microk8s kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-mpmkv 1/1 Running 7 (13m ago) 39h
indico modeloperator-7f4567579-tmrkv 1/1 Running 6 (18m ago) 38h
indico postgresql-k8s-operator-0 1/1 Running 3 (18m ago) 23h
controller-tutorial-controller modeloperator-787dbdd7dc-jv2cq 1/1 Running 6 (18m ago) 38h
indico redis-broker-operator-0 1/1 Running 3 (18m ago) 23h
kube-system hostpath-provisioner-78cb89d65b-wxf44 1/1 Running 6 (18m ago) 38h
indico redis-cache-operator-0 1/1 Running 3 (18m ago) 23h
kube-system calico-kube-controllers-976b9cd7f-rrrhb 1/1 Running 6 (18m ago) 39h
kube-system coredns-66bcf65bb8-tnck6 1/1 Running 6 (18m ago) 38h
ingress nginx-ingress-microk8s-controller-c5mgt 1/1 Running 1 (18m ago) 16h
controller-tutorial-controller controller-0 2/2 Running 21 (18m ago) 38h
indico redis-broker-0 1/1 Running 3 (18m ago) 23h
indico redis-cache-0 1/1 Running 3 (18m ago) 23h
indico nginx-ingress-integrator-0 1/1 Running 1 (18m ago) 16h
indico postgresql-k8s-0 1/1 Running 3 (18m ago) 23h
indico indico-0 5/5 Running 49 (3m41s ago) 23h
```
- Fixed by Joe suggestion:
```
juju update-k8s --client microk8s
```

- Working fine
```
juju status
Model Controller Cloud/Region Version SLA Timestamp
indico tutorial-controller microk8s/localhost 2.9.35 unsupported 09:15:35-03:00

App Version Status Scale Charm Channel Rev Address Exposed Message
indico 3.2 active 1 indico stable 83 10...

[Vitaly Antonenko (anvial)]

Please, reopen if it's still an issue.

[Launchpad Janitor (janitor)]

[Expired for Canonical Juju because there has been no activity for 60 days.]