race on snap installation

[Paul Collins]

It looks like the charm creates missing directories when writing daemon_arguments, which causes problems because snapd expects "current" to be a symlink:

ubuntu@juju-79f88d-prod-jaas-candid-6:~$ sudo snap install --channel=2/stable prometheus
error: cannot perform the following tasks:
- Make snap "prometheus" (20) available to the system (cannot cleanup failed attempt at making snap "prometheus" available to the system: cannot remove snap current symlink: remove /var/snap/prometheus/current: directory not empty)
- Make snap "prometheus" (20) available to the system (symlink 20 /var/snap/prometheus/current: file exists)
ubuntu@juju-79f88d-prod-jaas-candid-6:~$ ls -lR /var/snap/prometheus
/var/snap/prometheus:
total 4
drwxr-xr-x 2 root root 4096 Jul 6 05:31 current

/var/snap/prometheus/current:
total 4
-r--r--r-- 1 root root 2337 Jul 6 05:31 daemon_arguments
ubuntu@juju-79f88d-prod-jaas-candid-6:~$ _

[Stuart Bishop (stub)]

Is the race that the charm is configuring the snap before the snap has been installed, or that the charm thinks that the snap is installed but snapd is still setting it up?

[Paul Collins (pjdc)]

Unclear. This deployment was a bit fraught because we didn't add the route to the tenant network until quite a few attempts by the charm to install the snap. Here's the (rather large) unit agent log: https://private-fileshare.canonical.com/~pjdc/jaas-candid.unit-prometheus-0.log.xz

For some reason /var/snap/prometheus/common also did not get created, which caused further problems. I was able to cleanly install the snap on bionic and the charm in a xenial LXD, so things are not fundamentally broken.

[Xav Paice (xavpaice)]

If something sets prometheus.do-reconfig-yml or prometheus.do-reconfig-def while prometheus.do-install is still set, this could trigger if render() makes the missing dirs.