| 2015-01-21 01:07:21 |
Patrick Crews |
description |
Created a test stack.
Created snapshot of the stack.
Deleted the snapshot image via horizon / nova / glance vs. heat api.
On heat stack-delete, saw the error message below in the logs.
If the stack-delete action's goal is to eliminate all snapshots, then not finding it should perhaps throw a warning and inform the user, but this seems to not be handling this situation very well (even though it could be considered a win in terms of what the api call is trying to achieve)
2015-01-20 09:34:25.385 DEBUG heat.engine.scheduler [-] Task delete_snapshot from Server "wordpress_instance" [28aa750c-3c50-48f4-a9ab-e64a73051b68] Stack "test-stack-0" [e67708a9-ae6f-4335-bf41-ac1682319548] running from (pid=13179) step /opt/stack/heat/heat/engine/scheduler.py:219
2015-01-20 09:34:25.429 DEBUG heat.engine.stack_lock [-] Engine 43614879-f616-42f8-84fb-fd42a5d03ed7 released lock on stack e67708a9-ae6f-4335-bf41-ac1682319548 from (pid=13179) release /opt/stack/heat/heat/engine/stack_lock.py:123
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 455, in fire_timers
timer()
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
cb(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 212, in main
result = function(*args, **kwargs)
File "/opt/stack/heat/heat/engine/service.py", line 113, in _start_with_trace
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
return f(*args, **kwargs)
File "/opt/stack/heat/heat/engine/stack.py", line 982, in delete
self.delete_snapshot(snapshot)
File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
return f(*args, **kwargs)
File "/opt/stack/heat/heat/engine/stack.py", line 1087, in delete_snapshot
scheduler.TaskRunner(rsrc.delete_snapshot, data)()
File "/opt/stack/heat/heat/engine/scheduler.py", line 174, in __call__
self.start(timeout=timeout)
File "/opt/stack/heat/heat/engine/scheduler.py", line 199, in start
self.step()
File "/opt/stack/heat/heat/engine/scheduler.py", line 222, in step
next(self._runner)
File "/opt/stack/heat/heat/engine/scheduler.py", line 321, in wrapper
subtask = parent.throw(*sys.exc_info())
File "/opt/stack/heat/heat/engine/resource.py", line 798, in delete_snapshot
yield self.action_handler_task('delete_snapshot', args=[data])
File "/opt/stack/heat/heat/engine/scheduler.py", line 295, in wrapper
step = next(subtask)
File "/opt/stack/heat/heat/engine/resource.py", line 506, in action_handler_task
handler_data = handler(*args)
File "/opt/stack/heat/heat/engine/resources/server.py", line 1179, in handle_delete_snapshot
image_id = snapshot['resource_data']['snapshot_image_id']
KeyError: 'snapshot_image_id' |
The following error occurs when heat hits the end of a list of stack snapshots. It is appearing to cause stack-delete operations to hang indefinitely, though it could also be tied to another bug - https://bugs.launchpad.net/heat/+bug/1412961
Originally thought this was due to having deleted the snapshot images via other means. However, still seeing this error without touching snapshot images via nova / horizon / glance.
In the output below, tried deleting a stack with several snapshots.
All were handled well until the end of the list.
2015-01-20 15:49:42.789 DEBUG heat.engine.scheduler [-] Task delete_snapshot from Server "wordpress_instance" [106a764d-09a5-4b7a-b2a4-dbbb229154c0] Stack "test-stack-0" [33f1cda8-e3cd-4aa1-95fe-04eefdbb7ca9] running from (pid=11340) step /opt/stack/heat/heat/engine/scheduler.py:219
2015-01-20 15:49:42.838 DEBUG heat.engine.scheduler [-] Task delete_snapshot from Server "wordpress_instance" [106a764d-09a5-4b7a-b2a4-dbbb229154c0] Stack "test-stack-0" [33f1cda8-e3cd-4aa1-95fe-04eefdbb7ca9] sleeping from (pid=11340) _sleep /opt/stack/heat/heat/engine/scheduler.py:164
2015-01-20 15:49:43.839 DEBUG heat.engine.scheduler [-] Task delete_snapshot from Server "wordpress_instance" [106a764d-09a5-4b7a-b2a4-dbbb229154c0] Stack "test-stack-0" [33f1cda8-e3cd-4aa1-95fe-04eefdbb7ca9] running from (pid=11340) step /opt/stack/heat/heat/engine/scheduler.py:219
2015-01-20 15:49:43.839 DEBUG heat.engine.scheduler [-] Task delete_snapshot from Server "wordpress_instance" [106a764d-09a5-4b7a-b2a4-dbbb229154c0] Stack "test-stack-0" [33f1cda8-e3cd-4aa1-95fe-04eefdbb7ca9] complete from (pid=11340) step /opt/stack/heat/heat/engine/scheduler.py:225
2015-01-20 15:49:43.840 DEBUG heat.engine.scheduler [-] Task delete_snapshot from Server "wordpress_instance" [106a764d-09a5-4b7a-b2a4-dbbb229154c0] Stack "test-stack-0" [33f1cda8-e3cd-4aa1-95fe-04eefdbb7ca9] starting from (pid=11340) start /opt/stack/heat/heat/engine/scheduler.py:191
2015-01-20 15:49:43.840 DEBUG heat.engine.scheduler [-] Task delete_snapshot from Server "wordpress_instance" [106a764d-09a5-4b7a-b2a4-dbbb229154c0] Stack "test-stack-0" [33f1cda8-e3cd-4aa1-95fe-04eefdbb7ca9] running from (pid=11340) step /opt/stack/heat/heat/engine/scheduler.py:219
2015-01-20 15:49:43.890 DEBUG heat.engine.stack_lock [-] Engine b24c2aa7-9cc1-49a9-825a-98d0bb3dac19 released lock on stack 33f1cda8-e3cd-4aa1-95fe-04eefdbb7ca9 from (pid=11340) release /opt/stack/heat/heat/engine/stack_lock.py:123
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 455, in fire_timers
timer()
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
cb(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 212, in main
result = function(*args, **kwargs)
File "/opt/stack/heat/heat/engine/service.py", line 113, in _start_with_trace
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
return f(*args, **kwargs)
File "/opt/stack/heat/heat/engine/stack.py", line 982, in delete
self.delete_snapshot(snapshot)
File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
return f(*args, **kwargs)
File "/opt/stack/heat/heat/engine/stack.py", line 1087, in delete_snapshot
scheduler.TaskRunner(rsrc.delete_snapshot, data)()
File "/opt/stack/heat/heat/engine/scheduler.py", line 174, in __call__
self.start(timeout=timeout)
File "/opt/stack/heat/heat/engine/scheduler.py", line 199, in start
self.step()
File "/opt/stack/heat/heat/engine/scheduler.py", line 222, in step
next(self._runner)
File "/opt/stack/heat/heat/engine/scheduler.py", line 321, in wrapper
subtask = parent.throw(*sys.exc_info())
File "/opt/stack/heat/heat/engine/resource.py", line 798, in delete_snapshot
yield self.action_handler_task('delete_snapshot', args=[data])
File "/opt/stack/heat/heat/engine/scheduler.py", line 295, in wrapper
step = next(subtask)
File "/opt/stack/heat/heat/engine/resource.py", line 506, in action_handler_task
handler_data = handler(*args)
File "/opt/stack/heat/heat/engine/resources/server.py", line 1179, in handle_delete_snapshot
image_id = snapshot['resource_data']['snapshot_image_id']
KeyError: 'snapshot_image_id' |
|
