wait() timeout is not sufficiently propagated

[Stuart Bishop]

sentry.wait() will time out after about 10 minutes despite being passed a very large timeout value. Something in the guts of all this has an implicit timeout that is not being overridden.

A simple fix would be to catch exceptions and retry the wait until the requested timeout is actually reached.

Traceback (most recent call last):
  File "/home/stub/charms/cassandra/spike/tests/test_integration.py", line 328, in test_z_add_and_drop_node
    self.wait()
  File "/home/stub/charms/cassandra/spike/tests/test_integration.py", line 167, in wait
    self.deployment.sentry.wait(timeout=WAIT_TIMEOUT)
  File "/usr/lib/python3/dist-packages/amulet/sentry.py", line 259, in wait
    status = self.unit[unit].juju_agent()
  File "/usr/lib/python3/dist-packages/amulet/sentry.py", line 117, in juju_agent
    return self._run_unit_script("juju_agent.py")
  File "/usr/lib/python3/dist-packages/amulet/sentry.py", line 114, in _run_unit_script
    raise IOError(output)
OSError: ERROR command timed out

[Stuart Bishop (stub)]

        # Work around Bug #1421195 by retrying failed waits.
        # self.sentry.wait(timeout=timeout)
        until = time.time() + timeout
        while True:
            timeout = int(min(max(until - time.time(), 0), 300))
            try:
                self.sentry.wait(timeout=timeout)
                break
            except (OSError, amulet.helpers.TimeoutError):
                if time.time() > until:
                    raise