Run:
heat stack-create abc --template-file rg.yaml
wait about 20-30 sec and run:
heat stack-delete abc
Heat stucks in DELETE_IN_PROGRESS
Found this in logs:
2015-09-25 12:59:15.450 ERROR heat.engine.resource [-] DB error Not found
or
2015-09-25 13:04:52.109 ERROR heat.engine.resource [-] DB error This result object does not return rows. It has been closed automatically.
2015-09-25 13:04:52.110 ERROR sqlalchemy.pool.QueuePool [-] Exception during reset or similar
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool Traceback (most recent call last):
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 636, in _finalize_fairy
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool fairy._reset(pool)
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 776, in _reset
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool pool._dialect.do_rollback(self)
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/dialects/mysql/base.py", line 2519, in do_rollback
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool dbapi_connection.rollback()
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 711, in rollback
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool self._read_ok_packet()
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 687, in _read_ok_packet
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool raise err.OperationalError(2014, "Command Out of Sync")
2015-09-25 13:04:52.110 TRACE sqlalchemy.pool.QueuePool OperationalError: (2014, 'Command Out of Sync')
or:
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
cb(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
result = function(*args, **kwargs)
File "/opt/stack/heat/heat/engine/service.py", line 117, in _start_with_trace
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
return f(*args, **kwargs)
File "/opt/stack/heat/heat/engine/stack.py", line 1449, in delete
self.state_set(action, stack_status, reason)
File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
return f(*args, **kwargs)
File "/opt/stack/heat/heat/engine/stack.py", line 723, in state_set
stack = stack_object.Stack.get_by_id(self.context, self.id)
File "/opt/stack/heat/heat/objects/stack.py", line 90, in get_by_id
db_stack = db_api.stack_get(context, stack_id, **kwargs)
File "/opt/stack/heat/heat/db/api.py", line 134, in stack_get
eager_load=eager_load)
File "/opt/stack/heat/heat/db/sqlalchemy/api.py", line 344, in stack_get
result = query.get(stack_id)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 819, in get
return self._get_impl(ident, loading.load_on_ident)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 852, in _get_impl
return fallback_fn(self, key)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 219, in load_on_ident
return q.one()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2473, in one
ret = list(self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2516, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2529, in _execute_and_instances
close_with_result=True)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2520, in _connection_from_session
**kw)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 882, in connection
execution_options=execution_options)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 889, in _connection_for_bind
conn = engine.contextual_connect(**kw)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2041, in contextual_connect
**kwargs)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 92, in __init__
self.dispatch.engine_connect(self, self.__branch)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/event/attr.py", line 256, in __call__
fn(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 72, in _connect_ping_listener
connection.scalar(select([1]))
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 844, in scalar
return self.execute(object, *multiparams, **params).scalar()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/result.py", line 1064, in scalar
row = self.first()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/result.py", line 1038, in first
return self._non_result(None)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/result.py", line 920, in _non_result
"This result object does not return rows. "
ResourceClosedError: This result object does not return rows. It has been closed automatically.
All other time heat spamming into log something about:
Steps to reproduce:
rg.yaml: version: 2013-05-23 :ResourceGroup
resource_ def:
type: rand_str.yaml
heat_template_
resources:
rg:
type: OS::Heat:
properties:
count: 125
rand_str.yaml version: 2013-05-23
heat_template_
resources:
(yep, without any resource:) )
Run:
heat stack-create abc --template-file rg.yaml
wait about 20-30 sec and run:
heat stack-delete abc
Heat stucks in DELETE_IN_PROGRESS
Found this in logs:
2015-09-25 12:59:15.450 ERROR heat.engine. resource [-] DB error Not found
or
2015-09-25 13:04:52.109 ERROR heat.engine. resource [-] DB error This result object does not return rows. It has been closed automatically. pool.QueuePool [-] Exception during reset or similar pool.QueuePool Traceback (most recent call last): pool.QueuePool File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ pool.py" , line 636, in _finalize_fairy pool.QueuePool fairy._reset(pool) pool.QueuePool File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ pool.py" , line 776, in _reset pool.QueuePool pool._dialect. do_rollback( self) pool.QueuePool File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ dialects/ mysql/base. py", line 2519, in do_rollback pool.QueuePool dbapi_connectio n.rollback( ) pool.QueuePool File "/usr/local/ lib/python2. 7/dist- packages/ pymysql/ connections. py", line 711, in rollback pool.QueuePool self._read_ ok_packet( ) pool.QueuePool File "/usr/local/ lib/python2. 7/dist- packages/ pymysql/ connections. py", line 687, in _read_ok_packet pool.QueuePool raise err.Operational Error(2014, "Command Out of Sync") pool.QueuePool OperationalError: (2014, 'Command Out of Sync')
2015-09-25 13:04:52.110 ERROR sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
2015-09-25 13:04:52.110 TRACE sqlalchemy.
or:
File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ hubs/timer. py", line 58, in __call__ lib/python2. 7/dist- packages/ eventlet/ greenthread. py", line 214, in main heat/heat/ engine/ service. py", line 117, in _start_with_trace lib/python2. 7/dist- packages/ osprofiler/ profiler. py", line 105, in wrapper heat/heat/ engine/ stack.py" , line 1449, in delete state_set( action, stack_status, reason) lib/python2. 7/dist- packages/ osprofiler/ profiler. py", line 105, in wrapper heat/heat/ engine/ stack.py" , line 723, in state_set Stack.get_ by_id(self. context, self.id) heat/heat/ objects/ stack.py" , line 90, in get_by_id stack_get( context, stack_id, **kwargs) heat/heat/ db/api. py", line 134, in stack_get load=eager_ load) heat/heat/ db/sqlalchemy/ api.py" , line 344, in stack_get lib/python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 819, in get impl(ident, loading. load_on_ ident) lib/python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 852, in _get_impl lib/python2. 7/dist- packages/ sqlalchemy/ orm/loading. py", line 219, in load_on_ident lib/python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2473, in one lib/python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2516, in __iter__ and_instances( context) lib/python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2529, in _execute_ and_instances with_result= True) lib/python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2520, in _connection_ from_session lib/python2. 7/dist- packages/ sqlalchemy/ orm/session. py", line 882, in connection options= execution_ options) lib/python2. 7/dist- packages/ sqlalchemy/ orm/session. py", line 889, in _connection_ for_bind contextual_ connect( **kw) lib/python2. 7/dist- packages/ sqlalchemy/ engine/ base.py" , line 2041, in contextual_connect lib/python2. 7/dist- packages/ sqlalchemy/ engine/ base.py" , line 92, in __init__ dispatch. engine_ connect( self, self.__branch) lib/python2. 7/dist- packages/ sqlalchemy/ event/attr. py", line 256, in __call__ lib/python2. 7/dist- packages/ oslo_db/ sqlalchemy/ engines. py", line 72, in _connect_ ping_listener scalar( select( [1])) lib/python2. 7/dist- packages/ sqlalchemy/ engine/ base.py" , line 844, in scalar object, *multiparams, **params).scalar() lib/python2. 7/dist- packages/ sqlalchemy/ engine/ result. py", line 1064, in scalar lib/python2. 7/dist- packages/ sqlalchemy/ engine/ result. py", line 1038, in first result( None) lib/python2. 7/dist- packages/ sqlalchemy/ engine/ result. py", line 920, in _non_result rror: This result object does not return rows. It has been closed automatically.
cb(*args, **kw)
File "/usr/local/
result = function(*args, **kwargs)
File "/opt/stack/
return func(*args, **kwargs)
File "/usr/local/
return f(*args, **kwargs)
File "/opt/stack/
self.
File "/usr/local/
return f(*args, **kwargs)
File "/opt/stack/
stack = stack_object.
File "/opt/stack/
db_stack = db_api.
File "/opt/stack/
eager_
File "/opt/stack/
result = query.get(stack_id)
File "/usr/local/
return self._get_
File "/usr/local/
return fallback_fn(self, key)
File "/usr/local/
return q.one()
File "/usr/local/
ret = list(self)
File "/usr/local/
return self._execute_
File "/usr/local/
close_
File "/usr/local/
**kw)
File "/usr/local/
execution_
File "/usr/local/
conn = engine.
File "/usr/local/
**kwargs)
File "/usr/local/
self.
File "/usr/local/
fn(*args, **kw)
File "/usr/local/
connection.
File "/usr/local/
return self.execute(
File "/usr/local/
row = self.first()
File "/usr/local/
return self._non_
File "/usr/local/
"This result object does not return rows. "
ResourceClosedE
All other time heat spamming into log something about:
c89]: {}}) running from (pid=32479) step /opt/stack/ heat/heat/ engine/ scheduler. py:214 scheduler [-] Task destroy_resource running from (pid=32479) step /opt/stack/ heat/heat/ engine/ scheduler. py:214 environment [-] Registering file:// /home/oleksii/ rand_str. yaml -> file:// /home/oleksii/ rand_str. yaml scheduler [-] Task DependencyTaskG roup((destroy_ resource) {ResourceGroup "rg" [6c19089b- 9ed9-4d0f- 93ca-9c76ba9fa4 01] Stack "abc" [b5d8388e- 68fc-4279- bbc7-9c8ae3637c 89]: {}}) sleeping from (pid=32479) _sleep /opt/stack/ heat/heat/ engine/ scheduler. py:160
2015-09-25 13:13:32.333 DEBUG heat.engine.
2015-09-25 13:13:32.359 INFO heat.engine.
2015-09-25 13:13:32.360 DEBUG heat.engine.
Not 100% reproducible, but you could experiment with different time intervals 20-30-40-50 secs.
Run heat stack-delete abc again, and stack will be deleted as usual.