ResourceClosedError from DB in functional tests

[Thomas Herve]

As seen here: http://logs.openstack.org/19/211419/4/check/gate-heat-dsvm-functional-orig-mysql/82e3894/console.html

We get "failed to reach UPDATE_COMPLETE status within the required time".

There is an interesting error in the engine logs:

ResourceClosedError: This result object does not return rows. It has been closed automatically.

It may be a concurrency issue, but it's hard to tell what's going on.

[Steve Baker (steve-stevebaker)]

This is happening often enough to affect our merge rate.

http://logs.openstack.org/57/218957/3/check/gate-heat-dsvm-functional-orig-mysql/4e5d44e/console.html#_2015-09-07_21_27_43_693

2015-09-07 21:27:43.693 | 2015-09-07 21:27:43.658 | heat_integrationtests.functional.test_template_resource.TemplateResourceUpdateTest.test_template_resource_update_template_schema(provider_props_change)
2015-09-07 21:27:43.693 | 2015-09-07 21:27:43.660 | -------------------------------------------------------------------------------------------------------------------------------------------------------
2015-09-07 21:27:43.693 | 2015-09-07 21:27:43.662 |
2015-09-07 21:27:43.693 | 2015-09-07 21:27:43.664 | Captured traceback:
2015-09-07 21:27:43.693 | 2015-09-07 21:27:43.665 | ~~~~~~~~~~~~~~~~~~~
2015-09-07 21:27:43.694 | 2015-09-07 21:27:43.667 | Traceback (most recent call last):
2015-09-07 21:27:43.694 | 2015-09-07 21:27:43.668 | File "heat_integrationtests/functional/test_template_resource.py", line 464, in test_template_resource_update_template_schema
2015-09-07 21:27:43.694 | 2015-09-07 21:27:43.670 | files={'the.yaml': self.initial_tmpl})
2015-09-07 21:27:43.694 | 2015-09-07 21:27:43.671 | File "heat_integrationtests/common/test.py", line 431, in stack_create
2015-09-07 21:27:43.696 | 2015-09-07 21:27:43.672 | stack = self.client.stacks.get(name)
2015-09-07 21:27:43.697 | 2015-09-07 21:27:43.673 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 204, in get
2015-09-07 21:27:43.698 | 2015-09-07 21:27:43.675 | resp = self.client.get('/stacks/%s' % stack_id)
2015-09-07 21:27:43.699 | 2015-09-07 21:27:43.676 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 284, in get
2015-09-07 21:27:43.730 | 2015-09-07 21:27:43.677 | return self.client_request("GET", url, **kwargs)
2015-09-07 21:27:43.730 | 2015-09-07 21:27:43.678 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 277, in client_request
2015-09-07 21:27:43.730 | 2015-09-07 21:27:43.679 | resp, body = self.json_request(method, url, **kwargs)
2015-09-07 21:27:43.730 | 2015-09-07 21:27:43.682 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 266, in json_request
2015-09-07 21:27:43.731 | 2015-09-07 21:27:43.683 | resp = self._http_request(url, method, **kwargs)
2015-09-07 21:27:43.731 | 2015-09-07 21:27:43.684 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 228, in _http_request
2015-09-07 21:27:43.731 | 2015-09-07 21:27:43.686 | resp = self._http_request(path, method, **kwargs)
2015-09-07 21:27:43.731 | 2015-09-07 21:27:43.687 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 221, in _http_request
2015-09-07 21:27:43.731 | 2015-09-07 21:27:43.690 | raise exc.from_response(resp)
2015-09-07 21:27:43.731 | 2015-09-07 21:27:43.691 | heatclient.exc.HTTPInternalServerError: ERROR: Remote error: ResourceClosedError This result object does not return ...

[Angus Salkeld (asalkeld)]

I'd like to help but I ran "testr run --until-failure test_template_resource_update_template_schem" for 3 hours and didn't hit.

[Rabi Mishra (rabi)]

We're still seeing these issues often, though not for the same tests.

http://logs.openstack.org/97/283297/1/check/gate-heat-dsvm-functional-orig-mysql-lbaasv1/cf193d6/console.html#_2016-02-24_05_41_52_787

[Angus Salkeld (asalkeld)]

http://stackoverflow.com/questions/29030574/sqlalchemy-exc-resourceclosederror-this-connection-is-closed-when-inserting-aft

[Zane Bitter (zaneb)]

I'd bet this has the same root cause as bug 1546431.

Since, as Rabi pointed out, this can happen in any test (not just test_template_resource_update_template_schema), I updated the title to make it easier to find.

[Steve Baker (steve-stevebaker)]

In the last 7 days, all 13 ResourceClosedError errors had a root cause of "Command Out of Sync". There were 42 "Command Out of Sync" errors in the same period.

Therefore I've marked this as a duplicate of #1546431

[Zane Bitter (zaneb)]

It's a sqlalchemy bug: https://bitbucket.org/zzzeek/sqlalchemy/issues/3803/dbapi-connections-go-invalid-on