"connect-plug-peers": error: cannot communicate with server

[David]

Hello,

My colleague Mark Beierl faced this issue installing Microk8s and Juju:

+ sudo snap install microk8s --classic
microk8s (1.19/stable) v1.19.0 from Canonical✓ installed
++ whoami
+ sudo usermod -a -G microk8s mark
+ mkdir -p /home/mark/.kube
++ whoami
+ sudo chown -f -R mark /home/mark/.kube
+ KUBEGRP=microk8s
+ sudo snap install juju --classic --channel=2.8/stable
Run hook connect-plug-peers of snap "juju" \
error: cannot perform the following tasks:
- Run hook connect-plug-peers of snap "juju" (run hook "connect-plug-peers": error: cannot communicate with server: timeout exceeded while waiting for response)

I think juju has an interface that when it detects microk8s it will pull the controller docker image in the background. May be that's something that should be optional because the host machine might be behind a proxy, or the server might be down.

May be Mark can provide more information regarding the environment in which he faced the issue.

[Pen Gale (pengale)]

The following steps do not reproduce the issue:

```
# Launch a test environment and connect to it
kvm-ok # Verify kvm extensions working
multipass launch bionic --name test-connect-plug-peers --mem 8G --disk 40G --cpus 4
multipass shell test-connect-plug-peers

# Duplicate steps above in test environment
sudo snap install microk8s --classic
whoami
sudo usermod -a -G microk8s ubuntu
mkdir -p /home/ubuntu/.kube
sudo chown -f -R ubuntu /home/ubuntu/.kube
KUBEGRP=microk8s
sudo snap install juju --classic --channel=2.8/stable
```

I concur that it is probably an egress issue, or temporary issue w/ the server hosting the Docker image.

Next steps:

1. Add steps to the repro case above to simulate the egress/connection issue.
2. Determine whether we want to be more verbose about the error, or simply skip the image and move on.

[Pen Gale (pengale)]

The following script reproduces the error in the original bug (it just hits docker.io with a hammer on the test machine):

```
# Launch a test environment and connect to it
kvm-ok # Verify kvm extensions working
multipass launch bionic --name test-connect-plug-peers --mem 8G --disk 40G --cpus 4
multipass shell test-connect-plug-peers

# Duplicate steps above in test environment
sudo snap install microk8s --classic
whoami
sudo usermod -a -G microk8s ubuntu
mkdir -p /home/ubuntu/.kube
sudo chown -f -R ubuntu /home/ubuntu/.kube
KUBEGRP=microk8s
echo "127.0.0.1 docker.io" | sudo tee -a /etc/hosts
sudo snap install juju --classic --channel=2.8/stable
```

Produces:

```
error: cannot perform the following tasks:
- Run hook connect-plug-peers of snap "juju" (run hook "connect-plug-peers": error: cannot communicate with server: timeout exceeded while waiting for response)
```

[Mark Beierl (mbeierl)]

Here's an interesting tidbit, if I put a sleep between the two installs, it works.

Test 1:

```
sudo snap install microk8s --classic ; sudo snap install juju --classic --channel=2.8/stable
Download snap "microk8s" (1667) from channel "1.19/stable" Run configure hook of "microk8s" snap if present microk8s (1.19/stable) v1.19.0 from Canonical✓ installed
Download snap "juju" (14248) from channel "2.8/stable" Run hook connect-plug-peers of snap "juju" error: cannot perform the following tasks:
- Run hook connect-plug-peers of snap "juju" (run hook "connect-plug-peers": error: cannot communicate with server: timeout exceeded while waiting for response)
```

Test 2 (with sleep):

```
mark@hackfest:~$ sudo snap install microk8s --classic ; sleep 10 ; sudo snap install juju --classic --channel=2.8/stable
Run configure hook of "microk8s" snap if present -
microk8s (1.19/stable) v1.19.0 from Canonical✓ installed

juju (2.8/stable) 2.8.3 from Canonical✓ installed
```
```

[Pen Gale (pengale)]

Can confirm that running without a sleep reproduces the issue.

Here are the repro steps:

```
# Launch a test environment and connect to it
kvm-ok # Verify kvm extensions working
multipass launch bionic --name test-connect-plug-peers --mem 8G --disk 40G --cpus 4
multipass shell test-connect-plug-peers

# Duplicate steps above in test environment
sudo usermod -a -G microk8s ubuntu
mkdir -p /home/ubuntu/.kube
sudo chown -f -R ubuntu /home/ubuntu/.kube
KUBEGRP=microk8s
sudo snap install microk8s --classic; sudo snap install juju --classic --channel=2.8/stable
```

After running a "sudo snap remove --purge microk8s", I can repeat the last step successfully by adding a sleep (it still doesn't work if you omit the sleep):

sudo snap install microk8s --classic; sleep 60; sudo snap install juju --classic --channel=2.8/stable

[Pen Gale (pengale)]

Note about the last comment: I need a much longer sleep, possibly reflective of the fact that I was working on a vm on a laptop that I was actively using for other things.

This feels like a service inside of microk8s isn't all the way up at the time that the install hook returns.

[Mark Beierl (mbeierl)]

Agreed. Sleep 10 just <happened> to work, but it is arbitrary and I have had it fail since with only 10s.

[Mark Beierl (mbeierl)]

And if I just do this:

microk8s status --wait-ready

*before* trying to install juju, that works. So perhaps a documentation change to say that a wait ready should be done to ensure microk8s is ready to accept juju?

[David (davigar15)]

I am able to see this error after doing the following:

```
<Microk8s installed, juju bootstrapped to k8s, and some k8s charms deployed>
$ sudo snap disable microk8s
$ sudo reboot
$ sudo snap enable microk8s
error: cannot perform the following tasks:
- Run hook connect-plug-peers of snap "juju" (run hook "connect-plug-peers": error: cannot communicate with server: timeout exceeded while waiting for response)
```

[Nick Moffitt (nick-moffitt)]

I managed to install juju on my rpi4 Groovy install after six tries, five previous ones hitting this bug.

[Yang Kelvin Liu (kelvin.liu)]

https://github.com/juju/juju/pull/13016 will be landed to 2.8, 2.9 and dev to fix this issue.

[Yang Kelvin Liu (kelvin.liu)]

https://github.com/juju/juju/pull/13017 will be landed to 2.8, 2.9 and dev to fix this issue.