'make microk8s-operator-update' fails with docker snap

[John A Meinel]

If you 'snap install docker' that 'docker' binary cannot read /tmp because it is a confined snap. I'm not sure what the exact permissions are, but it probably is allowed to see $HOME, but not /tmp.

I also ran into the problem that 'docker build' needs access to /var/run/docker.sock, but the default permissions are:
srw-rw---- 1 root root 0 Mar 9 18:12 /var/run/docker.sock=

So it has to be run as 'sudo docker build'. However, you can "chown :adm /var/run/docker.sock" since the admin group should probably have access to docker.sock.

The actual diff to not require /tmp is pretty trivial:
diff --git a/Makefile b/Makefile
index 4b27f3e88f..0b01cd2783 100644
--- a/Makefile
+++ b/Makefile
@@ -189,7 +189,7 @@ check-deps:

 # CAAS related targets
 DOCKER_USERNAME ?= jujusolutions
-JUJUD_STAGING_DIR ?= /tmp/jujud-operator
+JUJUD_STAGING_DIR ?= ${GOPATH}/tmp/jujud-operator
 JUJUD_BIN_DIR ?= ${GOPATH}/bin
 OPERATOR_IMAGE_BUILD_SRC ?= true
 # By default the image tag is the full version number, including the build number.
@@ -208,7 +208,7 @@ endif

 operator-image: operator-check-build
        rm -rf ${JUJUD_STAGING_DIR}
- mkdir ${JUJUD_STAGING_DIR}
+ mkdir -p ${JUJUD_STAGING_DIR}
        cp ${JUJUD_BIN_DIR}/jujud ${JUJUD_STAGING_DIR}
        cp caas/jujud-operator-dockerfile ${JUJUD_STAGING_DIR}
        cp caas/jujud-operator-requirements.txt ${JUJUD_STAGING_DIR}

Note that there are still places in the Makefile that use /tmp (and not $TMP), though those use:
docker save ${OPERATOR_IMAGE_PATH} | gzip > /tmp/jujud-operator-image.tar.gz

Which doesn't ask *docker* to access those paths, but only your shell.
That said, if we are going to build in JUJUD_STAGING_DIR, it seems odd to use yet-another temp directory for the compressed image. So we might want something more like:

diff --git a/Makefile b/Makefile
index 4b27f3e88f..acf3600ace 100644
--- a/Makefile
+++ b/Makefile
@@ -189,7 +189,8 @@ check-deps:

 # CAAS related targets
 DOCKER_USERNAME ?= jujusolutions
-JUJUD_STAGING_DIR ?= /tmp/jujud-operator
+DOCKER_STAGING_DIR ?= ${GOPATH}/tmp
+JUJUD_STAGING_DIR ?= ${DOCKER_STAGING_DIR}/jujud-operator
 JUJUD_BIN_DIR ?= ${GOPATH}/bin
 OPERATOR_IMAGE_BUILD_SRC ?= true
 # By default the image tag is the full version number, including the build number.
@@ -208,7 +209,7 @@ endif

 operator-image: operator-check-build
        rm -rf ${JUJUD_STAGING_DIR}
- mkdir ${JUJUD_STAGING_DIR}
+ mkdir -p ${JUJUD_STAGING_DIR}
        cp ${JUJUD_BIN_DIR}/jujud ${JUJUD_STAGING_DIR}
        cp caas/jujud-operator-dockerfile ${JUJUD_STAGING_DIR}
        cp caas/jujud-operator-requirements.txt ${JUJUD_STAGING_DIR}
@@ -233,8 +234,8 @@ check-k8s-model:

 local-operator-update: check-k8s-model operator-image
        $(eval kubeworkers != juju status -m ${JUJU_K8S_MODEL} kubernetes-worker --format json | jq -c '.machines | keys' | tr -c '[:digit:]' ' ' 2>&1)
- docker save ${OPERATOR_IMAGE_PATH} | gzip > /tmp/jujud-operator-image.tar.gz
- $(foreach wm,$(kubeworkers), juju scp -m ${JUJU_K8S_MODEL} /tmp/jujud-operator-image.tar.gz $(wm):/tmp/jujud-operator-image.tar.gz ; )
+ docker save ${OPERATOR_IMAGE_PATH} | gzip > ${DOCKER_STAGING_DIR}/jujud-operator-image.tar.gz
+ $(foreach wm,$(kubeworkers), juju scp -m ${JUJU_K8S_MODEL} ${DOCKER_STAGING_DIR}/jujud-operator-image.tar.gz $(wm):/tmp/jujud-operator-image.tar.gz ; )
        $(foreach wm,$(kubeworkers), juju ssh -m ${JUJU_K8S_MODEL} $(wm) -- "zcat /tmp/jujud-operator-image.tar.gz | docker load" ; )

 STATIC_ANALYSIS_JOB ?=

(the other references to /tmp above are on the target nodes, so less likely to be a problem.)

Oddly, we don't 'rm /tmp/jujud-operator-image.tar.gz' even though we *do* 'rm -rf /tmp/jujud-operator' if the build completes successfully.

[John A Meinel (jameinel)]

I should state this differently. snaps *do* have access to a '/tmp' directory. but it is intentionally a *different* directory than the host machines' /tmp

See places like:
https://forum.snapcraft.io/t/temporarydirectory-and-how-to-manage-access-to-tmp/10176
https://forum.snapcraft.io/t/sharing-files-via-tmp/1613/20

Regardless the fix is still that you cannot use '/tmp' for files that you want to share between snaps and things-that-aren't-that-snap.