amulet openstack tests have race - some tests start before relations/hooks have settled

[Ryan Beisner]

Although most observable in the cinder-ceph amulet tests (by way of relation data sometimes existing, sometimes not existing at the point of checking), this potentially impacts all openstack charm amulet tests.

Currently, those tests use an arbitrary sleep after amulet / deployer has returned "complete." Sometimes that arbitrary wait time is not enough, depending on timing and load of other activities in the test rig.

So I implemented the same logic used in mojo-openstack specs (juju run against all units, twice before proceeding). That reduced the occurrence of race, but did not eliminate it. The juju run approach appears to no longer be effective.

Amulet helpers need a reliable method to know when hooks have settled, relations are made, and relation data is in a consistent state.

[Ryan Beisner (1chb1n)]

Example of impact: http://paste.ubuntu.com/11812413/

Where the broker_rsp relation data is not always present at the time the tests fire.

[Stuart Bishop (stub)]

For Juju 1.23 and earlier you need to install the juju-wait plugin from ppa:stub/juju, or implement its algorithm yourself (lp:juju-wait). Every other approach has been tried before and fails.

For Juju 1.24, you should be able to use the newer agent statuses.