Deletion of cluster could move it to 'Error' state

[Andrew Lazarev]

I was trying to create cluster and it hanged in 'Waiting' state because one of nodes didn't get floating IP (this is separate issue and it is not in savanna's scope).
So, I decided to delete cluster. And it finished in 'Error' state. I think that cluster must be deleted in any case if user requested that.

Logs:

2014-02-28 15:19:22.856 81472 INFO savanna.service.api [-] Cluster status has been changed: id=5ec469df-480b-4b50-945e-3e6ab5c0e3a7, New status=Deleting
2014-02-28 15:19:29.473 81472 WARNING savanna.service.engine [-] Can't start cluster 'al-hive' (reason: Instance id '728a5b95-8546-4813-bb00-d41430645e58' not found!)
2014-02-28 15:19:29.524 81472 INFO savanna.service.direct_engine [-] Cluster status has been changed: id=5ec469df-480b-4b50-945e-3e6ab5c0e3a7, New status=Error
2014-02-28 15:19:29.525 81472 INFO savanna.service.direct_engine [-] Cluster 'al-hive' creation rollback (reason: Instance id '728a5b95-8546-4813-bb00-d41430645e58' not found!)
2014-02-28 15:19:31.440 81472 ERROR savanna.context [-] Thread 'cluster-creating-5ec469df-480b-4b50-945e-3e6ab5c0e3a7' fails with exception: 'Instance id '728a5b95-8546-4813-bb00-d41430645e58' not found!'
2014-02-28 15:19:31.440 81472 TRACE savanna.context Traceback (most recent call last):
2014-02-28 15:19:31.440 81472 TRACE savanna.context File "/Users/andrew/Savanna/savanna/savanna/context.py", line 124, in _wrapper
2014-02-28 15:19:31.440 81472 TRACE savanna.context func(*args, **kwargs)
2014-02-28 15:19:31.440 81472 TRACE savanna.context File "/Users/andrew/Savanna/savanna/savanna/service/api.py", line 202, in _provision_cluster
2014-02-28 15:19:31.440 81472 TRACE savanna.context INFRA.create_cluster(cluster)
2014-02-28 15:19:31.440 81472 TRACE savanna.context File "/Users/andrew/Savanna/savanna/savanna/service/direct_engine.py", line 81, in create_cluster
2014-02-28 15:19:31.440 81472 TRACE savanna.context self._rollback_cluster_creation(cluster, ex)
2014-02-28 15:19:31.440 81472 TRACE savanna.context File "/Users/andrew/Savanna/savanna/savanna/openstack/common/excutils.py", line 68, in __exit__
2014-02-28 15:19:31.440 81472 TRACE savanna.context six.reraise(self.type_, self.value, self.tb)
2014-02-28 15:19:31.440 81472 TRACE savanna.context File "/Users/andrew/Savanna/savanna/savanna/service/direct_engine.py", line 59, in create_cluster
2014-02-28 15:19:31.440 81472 TRACE savanna.context self._await_networks(cluster, instances)
2014-02-28 15:19:31.440 81472 TRACE savanna.context File "/Users/andrew/Savanna/savanna/savanna/service/engine.py", line 62, in _await_networks
2014-02-28 15:19:31.440 81472 TRACE savanna.context if networks.init_instances_ips(instance):
2014-02-28 15:19:31.440 81472 TRACE savanna.context File "/Users/andrew/Savanna/savanna/savanna/service/networks.py", line 53, in init_instances_ips
2014-02-28 15:19:31.440 81472 TRACE savanna.context
2014-02-28 15:19:31.440 81472 TRACE savanna.context File "/Users/andrew/Savanna/savanna/savanna/conductor/api.py", line 128, in instance_update
2014-02-28 15:19:31.440 81472 TRACE savanna.context self._manager.instance_update(context, _get_id(instance), values)
2014-02-28 15:19:31.440 81472 TRACE savanna.context File "/Users/andrew/Savanna/savanna/savanna/conductor/manager.py", line 201, in instance_update
2014-02-28 15:19:31.440 81472 TRACE savanna.context self.db.instance_update(context, instance, values)
2014-02-28 15:19:31.440 81472 TRACE savanna.context File "/Users/andrew/Savanna/savanna/savanna/db/api.py", line 166, in instance_update
2014-02-28 15:19:31.440 81472 TRACE savanna.context IMPL.instance_update(context, instance, values)
2014-02-28 15:19:31.440 81472 TRACE savanna.context File "/Users/andrew/Savanna/savanna/savanna/db/sqlalchemy/api.py", line 281, in instance_update
2014-02-28 15:19:31.440 81472 TRACE savanna.context "Instance id '%s' not found!")
2014-02-28 15:19:31.440 81472 TRACE savanna.context NotFoundException: Instance id '728a5b95-8546-4813-bb00-d41430645e58' not found!
2014-02-28 15:19:31.440 81472 TRACE savanna.context
2014-02-28 15:19:31.454 81472 DEBUG savanna.utils.api [-] Validation Error occurred: error_code=400, error_message=Instance id '762796f6-6cbc-442e-b9f5-b43bcfca8239' not found!, error_name=NOT_FOUND bad_request /Users/andrew/Savanna/savanna/savanna/utils/api.py:239

[Andrew Lazarev (alazarev)]

https://review.openstack.org/#/c/98556/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/98556
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=d429c22a85a8637d805a6f66716688bc3e969180
Submitter: Jenkins
Branch: master

commit d429c22a85a8637d805a6f66716688bc3e969180
Author: Andrew Lazarev <email address hidden>
Date: Fri Jun 6 16:22:02 2014 -0700

    Added several checks on deleted cluster to prevent error logs

    Change-Id: I8f867d7cc89abbdf407a170344be062f00ed50a2
    Closes-Bug: #1274739
    Closes-Bug: #1286391