Installation should retry when failing to connect due to network transient issue

[Larry Michel]

Occasionally, there are network issues preventing connection to archives. The charm install will automatically fail. These network issues preventing contact with archive will sometimes persist for a short period of time and may be intermittent. If they persist beyond X minutes or so, then the charm should fail. But regardless of timeout, charm should retry.

Here's an example:

From console.txt:

2015-05-07 04:24:37,453 [WARNING] deployer.import: Config specifies num units for subordinate: neutron-calico
2015-05-07 04:24:37,454 [INFO] deployer.import: Adding 2 more units to swift-storage
2015-05-07 04:47:33,208 [ERROR] deployer.env: The following units had errors:
   unit: mysql/0: machine: 3/lxc/0 agent-state: error details: hook failed: "install"
2015-05-07 04:47:33,213 [ERROR] oil_ci.deploy.oil_deployer: Deployment failed:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/oil_ci/deploy/oil_deployer.py", line 140, in deploy
    debug=True, verbose=True)
  File "/usr/lib/python2.7/dist-packages/oil_ci/juju/juju_deployer.py", line 85, in run_deployer
    importer.Importer(env, deploy, options=opts).run()
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 206, in run
    self.wait_for_units(ignore_errors=ignore_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/action/importer.py", line 181, in wait_for_units
    services=self.deployment.get_service_names(), on_errors=on_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/env/go.py", line 227, in wait_for_units
    services=services, on_errors=on_errors).run(callback)
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 1923, in run
    if self.complete() is True:
  File "/usr/lib/python2.7/dist-packages/deployer/env/watchers.py", line 77, in complete
    on_errors(new_errors)
  File "/usr/lib/python2.7/dist-packages/deployer/env/watchers.py", line 98, in callback
    raise ErrorExit()

From unit log:

unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 Err http://archive.ubuntu.com/ubuntu/ trusty/main python-jinja2 all 2.7.2-2
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 Connection failed [IP: 91.189.91.14 80]
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 Fetched 14.3 kB in 2min 13s (106 B/s)
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/j/jinja2/python-jinja2_2.7.2-2_all.deb Connection failed [IP: 91.189.91.14 80]
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 Traceback (most recent call last):
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/install", line 6, in <module>
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 import lib.utils as utils
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/lib/utils.py", line 48, in <module>
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 install('python-jinja2')
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/lib/utils.py", line 41, in install
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 subprocess.check_call(cmd)
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 raise CalledProcessError(retcode, cmd)
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO unit.mysql/0.install logger.go:40 subprocess.CalledProcessError: Command '['apt-get', '-y', 'install', 'python-jinja2']' returned non-zero exit status 100
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO juju.worker.uniter.context context.go:381 handling reboot
unit-mysql-0[1981]: 2015-05-07 04:47:29 ERROR juju.worker.uniter.operation runhook.go:89 hook "install" failed: exit status 1
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO juju.worker.uniter modes.go:452 ModeContinue exiting
unit-mysql-0[1981]: 2015-05-07 04:47:29 INFO juju.worker.uniter modes.go:450 ModeHookError starting
unit-mysql-0[1981]: 2015-05-07 04:47:29 DEBUG juju.worker.uniter.filter filter.go:597 want resolved event
unit-mysql-0[1981]: 2015-05-07 04:47:29 DEBUG juju.worker.uniter.filter filter.go:591 want forced upgrade true
unit-mysql-0[1981]: 2015-05-07 04:47:29 DEBUG juju.worker.uniter.filter filter.go:727 no new charm event

[Michał Sawicz (saviq)]

The same applies to the Git fetcher.