Environment destroy can miss manual machines

[Jesse Meek]

We need to be able to assert the absence of these as part of the destroy txn in state/environ.go, but in order to do this manual machines need to add refcounts to their environments - such that they can be handled in a similar fashion to hosted environments.

[Andrew Wilkins (axwalk)]

I think we'll change storage so that it doesn't need to be handled here. Persistent storage will be destroyed via the storage provisioner as per usual. There was a gap that has just been filled that meant persistent storage would be leaked by destroy-env; this is no longer true, so we can relax the rules of env destruction.

[Andrew Wilkins (axwalk)]

We've been refcounting machines for a little while now, so we can now automatically destroy manual machines.

[Curtis Hovey (sinzui)]

Sorry, issue http://reports.vapour.ws/releases/issue/57b1c1f9749a567693457040 shows that the situation is now worse. We see upstart and systemd hosts have a juju2 left running. The console logs now show which more information about which processes and directories were left behind.

[Andrew Wilkins (axwalk)]

I've done a bunch of tests, and the agents always uninstall. It looks like there's a delay, though. I'll dig in and see if we can cut that out.

[Andrew Wilkins (axwalk)]

Actually there's not even that much of a delay - only 10 seconds between the signal being delivered, and the agent being uninstalled...

[Andrew Wilkins (axwalk)]

Finally was able to repro, using the same CI test. It looks like there might be a race which is causing the agent to skip the uninstall logic sometimes.

[Andrew Wilkins (axwalk)]

https://github.com/juju/juju/pull/6034

[Aaron Bentley (abentley)]

We are still seeing this as of 95d0fd04

[Aaron Bentley (abentley)]

e.g. http://reports.vapour.ws/releases/4292/job/manual-deploy-precise-amd64/attempt/3996

[Aaron Bentley (abentley)]

AIUI, this is won't-fix for 1.25

[Tim Penhey (thumper)]

The tests were failing due to stale CI lxd images.

They were running pre-release LXD.

update and upgrade fixed amd64 tests.

All other long standing CI lxd containers need to be updated.

[Curtis Hovey (sinzui)]

All manual tests are updating and upgrading the manual container before bootstrapping Juju. any other failures are now Juju's fault.