Deployment fails if /tmp and /var are on different file systems.

[Tim Van Steenburgh]

Opened by Karimerto on 2018-02-23 11:58:04+00:00 at https://github.com/juju-solutions/charm-flannel/issues/42

------------------------------------------------------------

I am manually trying to deploy a simple kubernetes system following this guide: Kubernetes on Ubuntu VMs
Everything goes well and everything seems to be in place, except for flannel. Looking at the logs I quickly found out why:
2018-02-23 11:47:55 INFO juju-log Reactive main running for hook update-status
2018-02-23 11:47:55 INFO juju-log Invoking reactive handler: reactive/flannel.py:24:install_flannel_binaries
2018-02-23 11:47:56 DEBUG update-status ERROR could not download resource: could not replace existing resource directory: rename /tmp/juju-resource-595581070/flannel-amd64 /var/lib/juju/agents/unit-flannel-0/resources/flannel-amd64: invalid cross-device link
2018-02-23 11:47:56 INFO juju-log Missing flannel resource.

Renaming a file across file system boundaries is obviously not possible, so the solution here would be to either use /var/tmp as the temporary folder, or actually copy the file in question. I have not yet had a chance to look at any of the code, but this does not sound too difficult to fix.

====================== COMMENTS ============================

Comment created by Cynerva on 2018-02-23 14:39:58+00:00

Thanks for the report.

It looks like the call to `resource_get` here returns empty: https://github.com/juju-solutions/charm-flannel/blob/master/reactive/flannel.py#L30

`resource_get` makes a subprocess call to `resource-get`, catches the error, and returns `False`: https://github.com/juju/charm-helpers/blob/master/charmhelpers/core/hookenv.py#L1030-L1034

So I'm pretty sure the root error:
```
ERROR could not download resource: could not replace existing resource directory: rename /tmp/juju-resource-595581070/flannel-amd64 /var/lib/juju/agents/unit-flannel-0/resources/flannel-amd64: invalid cross-device link
```

is coming from `resource-get`, which is a Juju primitive. I've opened an issue to get that looked at: https://bugs.launchpad.net/juju/+bug/1751291

@Karimerto Can you share the output of `juju version`, either here or in the linked Juju bug? Thanks.

------------------------------------------------------------

Comment created by Karimerto on 2018-02-23 21:49:36+00:00

Certainly, I am using the latest(?) juju that was available via snap a few days ago.
```
$ juju version
2.3.3-xenial-amd64
```

I was looking at the code of [coreos/flannel](https://github.com/coreos/flannel) and thought, apparently somewhat prematurely, that the error might come from there. They even have a TODO for it, see [here](https://github.com/coreos/flannel/blob/master/main.go#L553). So I'm just wondering whether flannel itself will still fail once that juju issue is fixed, since they have a very similar thing happening there.

------------------------------------------------------------

Comment created by Karimerto on 2018-02-25 19:37:12+00:00

It seems simply mounting `/tmp` elsewhere (`mount --bind /var/newtmp /tmp`) and restarting the flannel service (`systemctl restart jujud-unit-flannel-1.service`) did not do the trick.

I think the temp file itself (`/tmp/juju-resource-595581070/flannel-amd64` in this case) needs to be placed next to the actual target file `/var/lib/juju/agents/unit-flannel-0/resources/flannel-amd64`. Perhaps append a temporary suffix to the target and rename afterwards.