OpenStack Heat

test_template_resource.TemplateResourceSuspendResumeTest.test_suspend_resume fails on Mitaka gate

Bug #1523903 reported by Sergey Kraynev on 2015-12-08

This bug report is a duplicate of: Bug #1499669: Heat stucks in DELETE_IN_PROGRESS for some input data. Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Heat	New	Medium	Unassigned

Bug Description

Heat integration job [1] failed with error:

heat_integrationtests.functional.test_template_resource.TemplateResourceSuspendResumeTest.test_suspend_resume
| -------------------------------------------------------------------------------------------------------------
|
| Captured traceback:
| ~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "heat_integrationtests/functional/test_template_resource.py", line 760, in test_suspend_resume
self.stack_suspend(stack_identifier=stack_identifier)
File "heat_integrationtests/common/test.py", line 545, in stack_suspend
self._handle_in_progress(self.client.actions.suspend, stack_name)
File "heat_integrationtests/common/test.py", line 348, in _handle_in_progress
fn(*args, **kwargs)
File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/v1/actions.py", line 39, in suspend
self.client.post('/stacks/%s/actions' % stack_id, data=body)
File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 287, in post
return self.client_request("POST", url, **kwargs)
File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 277, in client_request
resp, body = self.json_request(method, url, **kwargs)
File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 266, in json_request
resp = self._http_request(url, method, **kwargs)
File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 228, in _http_request
resp = self._http_request(path, method, **kwargs)
File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 221, in _http_request
raise exc.from_response(resp)
heatclient.exc.HTTPInternalServerError: ERROR: Remote error: ResourceClosedError This result object does not return rows. It has been closed automatically.
[u'

http://logs.openstack.org/53/254653/1/check/gate-heat-dsvm-functional-orig-mysql/5e08ded/console.html

Tags:

Revision history for this message

Steven Hardy (shardy) wrote on 2015-12-08:

Download full text (9.3 KiB)

Looking at the heat log from the CI run mentioned:

Looking at the heat log from the CI run mentioned:

2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2575, in _connection_from_session
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher     **kw)
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher     execution_options=execution_options)
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 900, in _connection_for_bind
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher     conn = engine.contextual_connect(**kw)
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2041, in contextual_connect
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher     **kwargs)
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 92, in __init__
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher     self.dispatch.engine_connect(self, self.__branch)
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/event/attr.py", line 256, in __call__
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher     fn(*args, **kw)
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 72, in _connect_ping_listener
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher     connection.scalar(select([1]))
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 844, in scalar
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher     return self.execute(object, *multiparams, **params).scalar()
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/result.py", line 1064, in scalar
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher     row = self.first()
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/result.py", line 1038, in first
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher     return self._non_result(None)
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/result.py", line 920, in _non_result
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher     "This result object does not return rows. "
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher ResourceClosedError: This result object does not return rows. It has been closed automatically.
2015-12-08 11:49:58.612 31410 ERROR oslo_messaging.rpc.dispatcher 
2015-12-08 11:49:59.034 31410 ERROR oslo_messaging._drivers.common [req-4c688a53-9cb6-4415-aaac-06c887dd5f84 - demo - - -] Returning exception This result object does not return rows. It has been closed automatically. to caller
2015-12-08 11:49:59.034 31410 ERROR oslo_messaging._drivers.common [req-4c688a53-9cb6-4415-aaac-06c887dd5f84 - demo - - -] ['Traceback (most recent call last):\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper\n    return f(*args, **kwargs)\n', '  File "/opt/stack/new/heat/heat/common/context.py", line 296, in wrapped\n    return func(self, ctx, *args, **kwargs)\n', '  File "/opt/stack/new/heat/heat/engine/service.py", line 1530, in stack_suspend\n    s = self._get_stack(cnxt, stack_identity)\n', '  File "/opt/stack/new/heat/heat/engine/service.py", line 451, in _get_stack\n    eager_load=True)\n', '  File "/opt/stack/new/heat/heat/objects/stack.py", line 91, in get_by_id\n    stack = cls._from_db_object(context, cls(context), db_stack)\n', '  File "/opt/stack/new/heat/heat/objects/stack.py", line 73, in _from_db_object\n    context, db_stack[\'id\'])\n', '  File "/opt/stack/new/heat/heat/objects/stack_tag.py", line 65, in get\n    db_tags = db_api.stack_tags_get(context, stack_id)\n', '  File "/opt/stack/new/heat/heat/db/api.py", line 91, in stack_tags_get\n    return IMPL.stack_tags_get(context, stack_id)\n', '  File "/opt/stack/new/heat/heat/db/sqlalchemy/api.py", line 245, in stack_tags_get\n    .filter_by(stack_id=stack_id)\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2423, in all\n    return list(self)\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2571, in __iter__\n    return self._execute_and_instances(context)\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2584, in _execute_and_instances\n    close_with_result=True)\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2575, in _connection_from_session\n    **kw)\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection\n    execution_options=execution_options)\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 900, in _connection_for_bind\n    conn = engine.contextual_connect(**kw)\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2041, in contextual_connect\n    **kwargs)\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 92, in __init__\n    self.dispatch.engine_connect(self, self.__branch)\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/event/attr.py", line 256, in __call__\n    fn(*args, **kw)\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 72, in _connect_ping_listener\n    connection.scalar(select([1]))\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 844, in scalar\n    return self.execute(object, *multiparams, **params).scalar()\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/result.py", line 1064, in scalar\n    row = self.first()\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/result.py", line 1038, in first\n    return self._non_result(None)\n', '  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/result.py", line 920, in _non_result\n    "This result object does not return rows. "\n', 'ResourceClosedError: This result object does not return rows. It has been closed automatically.\n']
2015-12-08 11:49:59.044 31410 DEBUG oslo_messaging._drivers.amqpdriver [req-4c688a53-9cb6-4415-aaac-06c887dd5f84 - demo - - -] sending reply msg_id: c5b89fd70a9d4058941c1425263c7410 reply queue: reply_7831b918317044ed85543463cb8a8de6 _send_reply /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:79
2015-12-08 11:49:59.051 31410 ERROR sqlalchemy.pool.QueuePool [req-4c688a53-9cb6-4415-aaac-06c887dd5f84 - demo - - -] Exception during reset or similar
2015-12-08 11:49:59.051 31410 ERROR sqlalchemy.pool.QueuePool Traceback (most recent call last):
2015-12-08 11:49:59.051 31410 ERROR sqlalchemy.pool.QueuePool   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 636, in _finalize_fairy
2015-12-08 11:49:59.051 31410 ERROR sqlalchemy.pool.QueuePool     fairy._reset(pool)
2015-12-08 11:49:59.051 31410 ERROR sqlalchemy.pool.QueuePool   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 776, in _reset
2015-12-08 11:49:59.051 31410 ERROR sqlalchemy.pool.QueuePool     pool._dialect.do_rollback(self)
2015-12-08 11:49:59.051 31410 ERROR sqlalchemy.pool.QueuePool   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/dialects/mysql/base.py", line 2519, in do_rollback
2015-12-08 11:49:59.051 31410 ERROR sqlalchemy.pool.QueuePool     dbapi_connection.rollback()
2015-12-08 11:49:59.051 31410 ERROR sqlalchemy.pool.QueuePool   File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 724, in rollback
2015-12-08 11:49:59.051 31410 ERROR sqlalchemy.pool.QueuePool     self._read_ok_packet()
2015-12-08 11:49:59.051 31410 ERROR sqlalchemy.pool.QueuePool   File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 700, in _read_ok_packet
2015-12-08 11:49:59.051 31410 ERROR sqlalchemy.pool.QueuePool     raise err.OperationalError(2014, "Command Out of Sync")
2015-12-08 11:49:59.051 31410 ERROR sqlalchemy.pool.QueuePool OperationalError: (2014, 'Command Out of Sync')
2015-12-08 11:49:59.051 31410 ERROR sqlalchemy.pool.QueuePool 
2015-12-08 11:49:59.143 31410 DEBUG oslo_messaging._drivers.amqp

So, it looks more like a DB issue than a heat bug, unless we're doing something wrong in the DB layer.

Steve Baker (steve-stevebaker) on 2015-12-08

tags:

added: gate-failure

Report a bug

This report contains Public information

Everyone can see this information.

Duplicate of bug #1499669 Remove

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.