juju-test plugin should provide option to leave env alone

[David Britton]

The default behavior of destroying and creating environments between test runs is spot on, but for debugging, it's slow. Please provide an option to leave the environment alone both for quick setup, and for examining the environment after the failure.

[Marco Ceppi (marcoceppi)]

There's already a flag to do this, --set-e which will break the test immediately after a test fails but prior to destroying the environment.

[Marco Ceppi (marcoceppi)]

If this isn't working for you, or you need additional details, please let me know!

[David Britton (dpb)]

I don't believe --set-e will use an existing environment, will it?

[Marco Ceppi (marcoceppi)]

Sorry, no, that won't persist a bootstrap between runs, but it will stop break the test run if a test fails.

As for the don't tear down and bootstrap over and over again, there's a few different models for this and people from both camps asking. One is where a bootstrap is provided once and every test uses that. The other is like the first, only the test runner force destroys all machines in the deployment after each test (saving time for each bootstrap). I'm uncertain how requiring postmortem inspection of a failed test is not satisfied by --set-e and how having a common bootstrap addresses that

[David Britton (dpb)]

Also, --set-e doesn't allow inspection of the environment if things work.

In short, I'm trying to speed up the development cycle. As it is right now, I don't run juju-test, but run the test script directly, which has drawbacks (namely it's not the same test flow exactly).

[Sean Feole (sfeole)]

This is also a rather high priority bug with the hyperscale team. Like David, we would also like to utilize juju test. But on a manual provider ( LP #1284743) .

I see pros and cons to each of your examples in Comment #4. But if I had to choose one I would say the latter is fine with us.

"The other is like the first, only the test runner force destroys all machines in the deployment after each test (saving time for each bootstrap). "