environs/openstack: destroy-environment leaks machines in hpcloud

[Dave Cheney]

Discovered during load testing.

We're having trouble launching more than ~200 machines in hp cloud. After running into random failures, and destroying the environment, there remain 7 machines.

ubuntu@juju-hpgoctrl2-machine-0:~$ juju destroy-environment
ubuntu@juju-hpgoctrl2-machine-0:~$ echo $?

ubuntu@juju-hpgoctrl2-machine-0:~$ nova list
+---------+---------------------------+--------+--------------------------------------+
| ID | Name | Status | Networks |
+---------+---------------------------+--------+--------------------------------------+
| 1465097 | juju-hpgoctrl2-machine-0 | ACTIVE | private=10.7.194.166, 15.185.162.247 |
| 1520731 | juju-goscale2-machine-191 | ACTIVE | private=10.6.126.214, 15.185.170.227 |
| 1520733 | juju-goscale2-machine-192 | ACTIVE | private=10.6.126.220, 15.185.171.141 |
| 1520737 | juju-goscale2-machine-193 | ACTIVE | private=10.6.127.23, 15.185.173.21 |
| 1520743 | juju-goscale2-machine-194 | ACTIVE | private=10.6.127.31, 15.185.163.140 |
| 1520755 | juju-goscale2-machine-195 | ACTIVE | private=10.6.126.210, 15.185.172.230 |
| 1520763 | juju-goscale2-machine-196 | ACTIVE | private=10.6.126.252, 15.185.172.125 |
| 1520769 | juju-goscale2-machine-197 | ACTIVE | private=10.6.127.70, 15.185.170.77 |
+---------+---------------------------+--------+--------------------------------------+

Running destroy-environment again removed the remaining machines.

[Dave Cheney (dave-cheney)]

Yup, happened agian during load testing. This time the botostrap machine was still up an creating new machines as the destroy-environment was happening.

Repeatedly running the destroy-environment command eventually kills all the instances, although there is no feedback when the environment is totally destroyed.

[John A Meinel (jameinel)]

I believe we also have seen this when doing Live testing. If you run the test suite repeatedly, you'll often find you have a couple of instances still running when it is all done.

[Curtis Hovey (sinzui)]

I have not seen this problem in months. Could this be fixed? I changed the bug to incomplete to review the situation. If this has been observed, the bug can go back to triaged.

[Launchpad Janitor (janitor)]

[Expired for juju-core because there has been no activity for 60 days.]