EOF not properly retried under some circumstances

[Paweł Stołowski]

From time to time retry doesn't kick in, and the tests on Travis fail after just one attempt:

Mar 30 09:55:06 localhost.localdomain /usr/lib/snapd/snapd[4769]: retry.go:58: DEBUG: The retry loop for https://search.apps.ubuntu.com/api/v1/snaps/metadata finished after 1 retries, elapsed time=7.467119ms, status: Post https://search.apps.ubuntu.com/api/v1/snaps/metadata: EOF
Mar 30 09:55:06 localhost.localdomain snapd[4769]: 2017/03/30 09:55:06.317380 retry.go:58: DEBUG: The retry loop for https://search.apps.ubuntu.com/api/v1/snaps/metadata finished after 1 retries, elapsed time=7.467119ms, status: Post https://search.apps.ubuntu.com/api/v1/snaps/metadata: EOF

[Matt Bruzek (mbruzek)]

I encountered this EOF problem with installing the core snap. It failed with an unexpected EOF error. That error broke the charm's install hook which made the deployment of Kubernetes break.

2017-04-06 21:28:25 INFO install core 76.86 MB / 78.35 MB 98.10% 4.26 MB/s 0
2017-04-06 21:28:25 INFO install core 77.80 MB / 78.35 MB 99.30% 4.27 MB/s 0
2017-04-06 21:28:25 INFO install core 78.31 MB / 78.35 MB 99.96% 4.25 MB/s 0
2017-04-06 21:28:25 INFO install �[Kerror: cannot perform the following tasks:
2017-04-06 21:28:25 INFO install - Download snap "core" (1577) from channel "stable" (unexpected EOF)
2017-04-06 21:28:25 INFO install
2017-04-06 21:28:25 INFO install Traceback (most recent call last):
2017-04-06 21:28:25 INFO install File "/var/lib/juju/agents/unit-kubernetes-worker-0/charm/hooks/install", line 19, in <module>
2017-04-06 21:28:25 INFO install main()
2017-04-06 21:28:25 INFO install File "/usr/local/lib/python3.5/dist-packages/charms/reactive/__init__.py", line 78, in main
2017-04-06 21:28:25 INFO install bus.dispatch()
2017-04-06 21:28:25 INFO install File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 434, in dispatch
2017-04-06 21:28:25 INFO install _invoke(other_handlers)
2017-04-06 21:28:25 INFO install File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 417, in _invoke
2017-04-06 21:28:25 INFO install handler.invoke()
2017-04-06 21:28:25 INFO install File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 291, in invoke
2017-04-06 21:28:25 INFO install self._action(*args)
2017-04-06 21:28:25 INFO install File "/var/lib/juju/agents/unit-kubernetes-worker-0/charm/reactive/kubernetes_worker.py", line 137, in install_snaps
2017-04-06 21:28:25 INFO install snap.install('kubectl', channel=channel, classic=True)
2017-04-06 21:28:25 INFO install File "lib/charms/layer/snap.py", line 44, in install
2017-04-06 21:28:25 INFO install _install_store(snapname, **kw)
2017-04-06 21:28:25 INFO install File "lib/charms/layer/snap.py", line 142, in _install_store
2017-04-06 21:28:25 INFO install stderr=subprocess.STDOUT)
2017-04-06 21:28:25 INFO install File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
2017-04-06 21:28:25 INFO install **kwargs).stdout
2017-04-06 21:28:25 INFO install File "/usr/lib/python3.5/subprocess.py", line 708, in run
2017-04-06 21:28:25 INFO install output=stdout, stderr=stderr)
2017-04-06 21:28:25 INFO install subprocess.CalledProcessError: Command '['snap', 'install', '--classic', 'kubectl']' returned non-zero exit status 1
2017-04-06 21:28:25 ERROR juju.worker.uniter.operation runhook.go:107 hook "install" failed: exit status 1
2017-04-06 21:28:25 DEBUG juju.worker.uniter.operation executor.go:84 lock released

Would it be possible to retry snaps that get unexpected EOF so our code does not have to work around this problem?

[Paweł Stołowski (stolowski)]

Yes, we will fix that, and hopefully soon. Apologies for inconvienience caused by this.

The general problem of retrying (on various operations with the store) turned out to be more tricky than anticipated a few months ago (when a portion of fixes for retrying landed). There is one more fix that just landed and will be included in next release (but it won't cover this particular case you just experienced).

[Paweł Stołowski (stolowski)]

Actually, upon further discussion with the team it turns out this 'unexpected EOF' issue may be covered by the fix I mentioned in my earlier comment.