Interrupted stack-update leaves undeletable stack

[Steven Hardy]

After doing a series of updates on a tripleo overcloud, I interrupted one by restarting the heat-engine, which as expected set the stack to UPDATE_FAILED.

I then attempted to delete the stack, which proved impossible, due to a persistent error deleting the backup stack:

 [-] Unexpected exception in delete: The Resource Type (OS::TripleO::Network::Ports::NetVipMap) could not be found.
 [-] Stack DELETE FAILED (overcloud): The Resource Type (OS::TripleO::Network::Ports::NetVipMap) could not be found.
 [-] Unhandled error in asynchronous task
 Traceback (most recent call last):
.py", line 127, in log_exceptions
     gt.wait()
   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
     return self._exit_event.wait()
   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
     current.throw(*self._exc)
   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
     result = function(*args, **kwargs)
.py", line 118, in _start_with_trace
     return func(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
     return f(*args, **kwargs)
   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 86, in handle_exceptions
     {'func': func.__name__, 'msg': errmsg})
   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
     six.reraise(self.type_, self.value, self.tb)
   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 81, in handle_exceptions
     return func(stack, *args, **kwargs)
   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 1540, in delete
     if backup_stack:
   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 580, in __len__
     return len(self.resources)
   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 266, in resources
     self.t.resource_definitions(self).items())
   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 265, in <genexpr>
     for (name, data) in
   File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 138, in __new__
     resource_name=name)
   File "/usr/lib/python2.7/site-packages/heat/engine/environment.py", line 475, in get_class_to_instantiate
     raise exception.StackValidationFailed(message=six.text_type(exc))
 StackValidationFailed: The Resource Type (OS::TripleO::Network::Ports::NetVipMap) could not be found.

I'm not sure if this is due to the recent changes to the environment re TemplateResource class generation, but the root cause appears to be an inconsistent stored template/environment for the backup stack.

In this case, it seems like we need to tolerate the StackValidationFailed and just delete the resource that exist, rather than failing and leaving a stack in an undeletable state.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/264275

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steven Hardy (<email address hidden>) on branch: master
Review: https://review.openstack.org/264275
Reason: I've not managed to reproduce this, will revisit if it happens again