Snap installation of microk8s fails when juju snap is also installed

[Kenneth Koski]

I am trying to install microk8s after having installed juju latest/candidate. The installation fails with this error message:

$ sudo snap install microk8s --classic
error: cannot perform the following tasks:
- Run hook connect-plug-peers of snap "juju" (run hook "connect-plug-peers":
-----
error: error running snapctl: cannot perform the following tasks:
- start of [juju.fetch-oci] (# systemctl start snap.juju.fetch-oci.service
Job for snap.juju.fetch-oci.service failed because the control process exited with error code.
See "systemctl status snap.juju.fetch-oci.service" and "journalctl -xe" for details.
)
- start of [juju.fetch-oci] (exit status 1)
-----)

Checking the logs from journalctl, I see these lines:

Jun 02 09:27:15 cat systemd[1]: Starting Service for snap application juju.fetch-oci...
Jun 02 09:27:15 cat juju.fetch-oci[337478]: /snap/bin/microk8s.kubectl
Jun 02 09:27:15 cat juju.fetch-oci[337479]: /snap/bin/microk8s.ctr
Jun 02 09:27:15 cat juju.fetch-oci[337460]: Wait for microk8s to be ready if needed.
Jun 02 09:27:15 cat juju.fetch-oci[337480]: Insufficient permissions to access MicroK8s.
Jun 02 09:27:15 cat juju.fetch-oci[337480]: /snap/microk8s/1422/actions/common/utils.sh: line 7: USER: unbound variable
Jun 02 09:27:15 cat systemd[1]: snap.juju.fetch-oci.service: Main process exited, code=exited, status=1/FAILURE
Jun 02 09:27:15 cat systemd[1]: snap.juju.fetch-oci.service: Failed with result 'exit-code'.
Jun 02 09:27:15 cat systemd[1]: Failed to start Service for snap application juju.fetch-oci.

I'm opening this issue in Juju's bug tracker because I checked that file, and those lines haven't been changed since 2019, whereas this installation error started happening recently.

[Ian Booth (wallyworld)]

It seems the issue here is that the user cannot run microk8s because they are not in the microk8s group.

https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/common/utils.sh#L7

So microk8s is not operational and the juju snap hook exits 1.
We'll land a tweak to the juju snap hook to always exit 0 so that even if there's an issue, it won't block anything. After all, it's just an optimisation that is being done.

[Ian Booth (wallyworld)]

https://github.com/juju/juju/pull/11662

[Konstantinos Tsakalozos (kos.tsakalozos)]

Did this patch make it to 2.8.0 release?

I see it on the 2.8.0 release:
```
jackal@aurora:~/workspace/microk8s$ sudo snap install microk8s --classic
[sudo] password for jackal:
error: cannot perform the following tasks:
- Run hook connect-plug-peers of snap "juju" (run hook "connect-plug-peers":
-----
error: error running snapctl: cannot perform the following tasks:
- start of [juju.fetch-oci] (# systemctl start snap.juju.fetch-oci.service
Job for snap.juju.fetch-oci.service failed because the control process exited with error code.
See "systemctl status snap.juju.fetch-oci.service" and "journalctl -xe" for details.
)
- start of [juju.fetch-oci] (exit status 1)
-----)
jackal@aurora:~/workspace/microk8s$ juju version
2.8.0-focal-amd64
jackal@aurora:~/workspace/microk8s$ sudo snap refresh juju
snap "juju" has no updates available
```

[Ian Booth (wallyworld)]

Something weird here. The fetch-oci script was updated and now exits 0 in all cases - it used to exit 1 if microk8s.status failed.

I again installed juju and then microk8s snap on a clean system and saw the error reported. Yet I then ran the fetch-oci script by hand and looking at the exit code showed it did behave as expected in that manual test case.

I could also manually run systemctl start snap.juju.fetch-oci.service without issue at that point.

Yet the system logs seem to indicate this chink of code causes exit 1 when done as part of the snap hook invocation.

echo "Wait for microk8s to be ready if needed."
wait_result=$(microk8s.status --wait-ready --timeout 30 2>&1)
if [ $? -ne 0 ]; then
    echo "${wait_result}"
    exit 0
fi

More investigation needed.

[Tilman Baumann (tilmanbaumann)]

I can confirm.
My juju has the patch from #2 but it still fails

tilmanb@zuse:~/git/testcharm$ sudo systemctl restart snap.juju.fetch-oci.service
tilmanb@zuse:~/git/testcharm$ sudo snap install --classic microk8s
error: cannot perform the following tasks:
- Run hook connect-plug-peers of snap "juju" (run hook "connect-plug-peers":
-----
error: error running snapctl: cannot perform the following tasks:
- start of [juju.fetch-oci] (# systemctl start snap.juju.fetch-oci.service
Job for snap.juju.fetch-oci.service failed because the control process exited with error code.
See "systemctl status snap.juju.fetch-oci.service" and "journalctl -xe" for details.
)
- start of [juju.fetch-oci] (exit status 1)
-----)

[Yang Kelvin Liu (kelvin.liu)]

https://github.com/juju/juju/pull/11680 will be landed at 2.8 to fix this issue!

[Calvin Hartwell (calvinh)]

This is still broken upstream. Fresh install of Juju yesterday for a demo today and microk8s wouldn't install for same issue highlighted by Tilman. 20.04 latest.

https://i.imgur.com/vCuhC8S.png

[Ian Booth (wallyworld)]

The fix in 2.8.0 wasn't quite right and and updated fix was landed for 2.8.1 (it is in the 2.8.1 edge snap).
Until then, you need to make sure microk8s is installed and operational before installing juju. The main issue is that the user needs to be in the microk8s group.

[Tytus Kurek (tkurek)]

The following sequence works for me:

sudo snap install juju --classic --edge
sudo snap install microk8s --classic

[Ian Booth (wallyworld)]

That order only works if microk8s has been installed previously, since the user is in the microk8s group. Until 2.8.1 is released, the best order to do things is install microk8s first and ensure that the user is in the microk8s group so that (for example) microk8s.status runs.

[Tim McNamara (tim-clicks)]

I've updated the "Using Juju with MicroK8s" document so that it's clear that MicroK8s should be installed first.