juju-deployer model teardown skips machine 0

[James Page]

UOSCI makes use of bundletester, which uses juju-deployer to tear down services and units on a model; as part of this process it force terminates all machines *apart* from machine 0, which in a juju 2.0 model is no longer the controller.

This results in odd model recycle behaviour - if the application on machine 0 has a hook error as the rest of the model tears down, it will block the test execution.

[James Page (james-page)]

Example of this:

  https://review.openstack.org/#/c/497660

the hook is probably erroring due to the force termination of the other machines in the model; I'm not sure we need to fix anything in the charm per-say, but we do need to fixup juju-deploy to be more intelligent when tearing down models.

[James Page (james-page)]

Fairly easy to reproduce; using a fresh model deploy the attached bundle (based on the hacluster amulet tests).

Use the following code to reset the environment:

from deployer.env.go import GoEnvironment
environment = GoEnvironment('serverstack-serverstack:admin/keystone-teardown')
environment.reset(terminate_machines=True, terminate_delay=60, force_terminate=True)

environment will have all machines apart from machine 0 force terminated; machine 0 takes a while to clear down as the subordinate charm stops, and then the leader-settings-changed hook fails due to the forced teardown of machines.

I think the right fix is to have the deployer code also force terminate machine 0 for juju versions >= 2.0