adding unit under load times out perf test

[Christopher Lee]

Using an exploratory performance test that compares 1.25.6 against 2.0 we find that 1.25.6 passes while 2.0 times out when adding units.

The test in a nutshell does:
  - Bootstrap
  - Deploy peer-xplode charm [2]
  - In a loop:
     - Add a unit, wait for it to be 'started' and wait for the workload to be set (start hook sets this)
     - Do this for 20 units.
     (There is 2 flavors of this, add a single unit at a time until we have 20, and adding 2, 4, 6, then 8 units (again 20 total)).

This is the script used[1] and it's been run using lxd as the provider.

The charm used[2] is a slightly modified charm from John M that continuously fires off hooks.

With 1.25.6 we see the test complete (add 20 units in total).
With 2.0 we get a timeout (1200 seconds) while waiting for the workload to be set when we get to 6+ units.

The script is setup just like a CI test assess script so can be run like this:
  JUJU_HOME=~/cloud-city JUJU_REPOSITORY=../repository/ ./assess_perf_testing_xplod.py parallel-lxd /path/to/juju/binary /path/to/log-result/dir <some identifier name, i.e. perf-test>

[1] https://code.launchpad.net/~veebers/juju-ci-tools/exploratory-testing-peer-xplod
[2] https://code.launchpad.net/~veebers/+junk/updated-peer-explod-charm/

[Joseph Phillips (manadart)]

Deployment and provisioning logic has changed substantially since this report.