| 2015-09-25 10:19:33 |
Oleksii Chuprykov |
description |
Steps to reproduce:
rg.yaml:
heat_template_version: 2013-05-23
resources:
rg:
type: OS::Heat::ResourceGroup
properties:
count: 125
resource_def:
type: rand_str.yaml
rand_str.yaml
heat_template_version: 2013-05-23
resources:
(yep, without any resource:) )
Run:
heat stack-create abc --template-file rg.yaml
wait about 20-30 sec and run:
heat stack-delete abc
Heat stucks in DELETE_IN_PROGRESS
Found this in logs:
2015-09-25 12:59:15.450 ERROR heat.engine.resource [-] DB error Not found
or
2015-09-25 13:04:52.109 ERROR heat.engine.resource [-] DB error This result object does not return rows. It has been closed automatically.
2015-09-25 13:04:52.110 ERROR sqlalchemy.pool.QueuePool [-] Exception during reset or similar
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool Traceback (most recent call last):
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 636, in _finalize_fairy
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool fairy._reset(pool)
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 776, in _reset
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool pool._dialect.do_rollback(self)
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/dialects/mysql/base.py", line 2519, in do_rollback
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool dbapi_connection.rollback()
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 711, in rollback
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool self._read_ok_packet()
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 687, in _read_ok_packet
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool raise err.OperationalError(2014, "Command Out of Sync")
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool OperationalError: (2014, 'Command Out of Sync')
or:
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
cb(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
result = function(*args, **kwargs)
File "/opt/stack/heat/heat/engine/service.py", line 117, in _start_with_trace
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
return f(*args, **kwargs)
File "/opt/stack/heat/heat/engine/stack.py", line 1449, in delete
self.state_set(action, stack_status, reason)
File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
return f(*args, **kwargs)
File "/opt/stack/heat/heat/engine/stack.py", line 723, in state_set
stack = stack_object.Stack.get_by_id(self.context, self.id)
File "/opt/stack/heat/heat/objects/stack.py", line 90, in get_by_id
db_stack = db_api.stack_get(context, stack_id, **kwargs)
File "/opt/stack/heat/heat/db/api.py", line 134, in stack_get
eager_load=eager_load)
File "/opt/stack/heat/heat/db/sqlalchemy/api.py", line 344, in stack_get
result = query.get(stack_id)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 819, in get
return self._get_impl(ident, loading.load_on_ident)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 852, in _get_impl
return fallback_fn(self, key)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 219, in load_on_ident
return q.one()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2473, in one
ret = list(self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2516, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2529, in _execute_and_instances
close_with_result=True)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2520, in _connection_from_session
**kw)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 882, in connection
execution_options=execution_options)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 889, in _connection_for_bind
conn = engine.contextual_connect(**kw)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2041, in contextual_connect
**kwargs)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 92, in __init__
self.dispatch.engine_connect(self, self.__branch)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/event/attr.py", line 256, in __call__
fn(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 72, in _connect_ping_listener
connection.scalar(select([1]))
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 844, in scalar
return self.execute(object, *multiparams, **params).scalar()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/result.py", line 1064, in scalar
row = self.first()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/result.py", line 1038, in first
return self._non_result(None)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/result.py", line 920, in _non_result
"This result object does not return rows. "
ResourceClosedError: This result object does not return rows. It has been closed automatically.
All other time heat spamming into log something about:
c89]: {}}) running from (pid=32479) step /opt/stack/heat/heat/engine/scheduler.py:214
2015-09-25 13:13:32.333 DEBUG heat.engine.scheduler [-] Task destroy_resource running from (pid=32479) step /opt/stack/heat/heat/engine/scheduler.py:214
2015-09-25 13:13:32.359 INFO heat.engine.environment [-] Registering file:///home/oleksii/rand_str.yaml -> file:///home/oleksii/rand_str.yaml
2015-09-25 13:13:32.360 DEBUG heat.engine.scheduler [-] Task DependencyTaskGroup((destroy_resource) {ResourceGroup "rg" [6c19089b-9ed9-4d0f-93ca-9c76ba9fa401] Stack "abc" [b5d8388e-68fc-4279-bbc7-9c8ae3637c89]: {}}) sleeping from (pid=32479) _sleep /opt/stack/heat/heat/engine/scheduler.py:160
Not 100% reproducible, but you could experiment with different time intervals 20-30-40-50 secs.
Run heat stack-delete abc again, and stack will be deleted as usual. |
Steps to reproduce:
rg.yaml:
heat_template_version: 2013-05-23
resources:
rg:
type: OS::Heat::ResourceGroup
properties:
count: 125
resource_def:
type: rand_str.yaml
rand_str.yaml
heat_template_version: 2013-05-23
resources:
(yep, without any resource:) )
Run:
heat stack-create abc --template-file rg.yaml
wait about 20-30 sec and run:
heat stack-delete abc
(before stack will be in CREATE_COMPLETE)
Heat stucks in DELETE_IN_PROGRESS
Found this in logs:
2015-09-25 12:59:15.450 ERROR heat.engine.resource [-] DB error Not found
or
2015-09-25 13:04:52.109 ERROR heat.engine.resource [-] DB error This result object does not return rows. It has been closed automatically.
2015-09-25 13:04:52.110 ERROR sqlalchemy.pool.QueuePool [-] Exception during reset or similar
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool Traceback (most recent call last):
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 636, in _finalize_fairy
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool fairy._reset(pool)
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 776, in _reset
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool pool._dialect.do_rollback(self)
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/dialects/mysql/base.py", line 2519, in do_rollback
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool dbapi_connection.rollback()
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 711, in rollback
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool self._read_ok_packet()
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 687, in _read_ok_packet
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool raise err.OperationalError(2014, "Command Out of Sync")
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool OperationalError: (2014, 'Command Out of Sync')
or:
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
cb(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
result = function(*args, **kwargs)
File "/opt/stack/heat/heat/engine/service.py", line 117, in _start_with_trace
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
return f(*args, **kwargs)
File "/opt/stack/heat/heat/engine/stack.py", line 1449, in delete
self.state_set(action, stack_status, reason)
File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
return f(*args, **kwargs)
File "/opt/stack/heat/heat/engine/stack.py", line 723, in state_set
stack = stack_object.Stack.get_by_id(self.context, self.id)
File "/opt/stack/heat/heat/objects/stack.py", line 90, in get_by_id
db_stack = db_api.stack_get(context, stack_id, **kwargs)
File "/opt/stack/heat/heat/db/api.py", line 134, in stack_get
eager_load=eager_load)
File "/opt/stack/heat/heat/db/sqlalchemy/api.py", line 344, in stack_get
result = query.get(stack_id)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 819, in get
return self._get_impl(ident, loading.load_on_ident)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 852, in _get_impl
return fallback_fn(self, key)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 219, in load_on_ident
return q.one()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2473, in one
ret = list(self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2516, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2529, in _execute_and_instances
close_with_result=True)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2520, in _connection_from_session
**kw)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 882, in connection
execution_options=execution_options)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 889, in _connection_for_bind
conn = engine.contextual_connect(**kw)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2041, in contextual_connect
**kwargs)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 92, in __init__
self.dispatch.engine_connect(self, self.__branch)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/event/attr.py", line 256, in __call__
fn(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 72, in _connect_ping_listener
connection.scalar(select([1]))
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 844, in scalar
return self.execute(object, *multiparams, **params).scalar()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/result.py", line 1064, in scalar
row = self.first()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/result.py", line 1038, in first
return self._non_result(None)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/result.py", line 920, in _non_result
"This result object does not return rows. "
ResourceClosedError: This result object does not return rows. It has been closed automatically.
All other time heat spamming into log something about:
c89]: {}}) running from (pid=32479) step /opt/stack/heat/heat/engine/scheduler.py:214
2015-09-25 13:13:32.333 DEBUG heat.engine.scheduler [-] Task destroy_resource running from (pid=32479) step /opt/stack/heat/heat/engine/scheduler.py:214
2015-09-25 13:13:32.359 INFO heat.engine.environment [-] Registering file:///home/oleksii/rand_str.yaml -> file:///home/oleksii/rand_str.yaml
2015-09-25 13:13:32.360 DEBUG heat.engine.scheduler [-] Task DependencyTaskGroup((destroy_resource) {ResourceGroup "rg" [6c19089b-9ed9-4d0f-93ca-9c76ba9fa401] Stack "abc" [b5d8388e-68fc-4279-bbc7-9c8ae3637c89]: {}}) sleeping from (pid=32479) _sleep /opt/stack/heat/heat/engine/scheduler.py:160
Not 100% reproducible, but you could experiment with different time intervals 20-30-40-50 secs.
Run heat stack-delete abc again, and stack will be deleted as usual. |
|
