ResourceClosedError from DB in functional tests

Bug #1483670 reported by Thomas Herve on 2015-08-11
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
Triaged
High
Unassigned

Bug Description

As seen here: http://logs.openstack.org/19/211419/4/check/gate-heat-dsvm-functional-orig-mysql/82e3894/console.html

We get "failed to reach UPDATE_COMPLETE status within the required time".

There is an interesting error in the engine logs:

ResourceClosedError: This result object does not return rows. It has been closed automatically.

It may be a concurrency issue, but it's hard to tell what's going on.

Thomas Herve (therve) on 2015-08-11
description: updated
Changed in heat:
status: New → Triaged
importance: Undecided → Medium
Steve Baker (steve-stevebaker) wrote :
Download full text (3.2 KiB)

This is happening often enough to affect our merge rate.

http://logs.openstack.org/57/218957/3/check/gate-heat-dsvm-functional-orig-mysql/4e5d44e/console.html#_2015-09-07_21_27_43_693

2015-09-07 21:27:43.693 | 2015-09-07 21:27:43.658 | heat_integrationtests.functional.test_template_resource.TemplateResourceUpdateTest.test_template_resource_update_template_schema(provider_props_change)
2015-09-07 21:27:43.693 | 2015-09-07 21:27:43.660 | -------------------------------------------------------------------------------------------------------------------------------------------------------
2015-09-07 21:27:43.693 | 2015-09-07 21:27:43.662 |
2015-09-07 21:27:43.693 | 2015-09-07 21:27:43.664 | Captured traceback:
2015-09-07 21:27:43.693 | 2015-09-07 21:27:43.665 | ~~~~~~~~~~~~~~~~~~~
2015-09-07 21:27:43.694 | 2015-09-07 21:27:43.667 | Traceback (most recent call last):
2015-09-07 21:27:43.694 | 2015-09-07 21:27:43.668 | File "heat_integrationtests/functional/test_template_resource.py", line 464, in test_template_resource_update_template_schema
2015-09-07 21:27:43.694 | 2015-09-07 21:27:43.670 | files={'the.yaml': self.initial_tmpl})
2015-09-07 21:27:43.694 | 2015-09-07 21:27:43.671 | File "heat_integrationtests/common/test.py", line 431, in stack_create
2015-09-07 21:27:43.696 | 2015-09-07 21:27:43.672 | stack = self.client.stacks.get(name)
2015-09-07 21:27:43.697 | 2015-09-07 21:27:43.673 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 204, in get
2015-09-07 21:27:43.698 | 2015-09-07 21:27:43.675 | resp = self.client.get('/stacks/%s' % stack_id)
2015-09-07 21:27:43.699 | 2015-09-07 21:27:43.676 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 284, in get
2015-09-07 21:27:43.730 | 2015-09-07 21:27:43.677 | return self.client_request("GET", url, **kwargs)
2015-09-07 21:27:43.730 | 2015-09-07 21:27:43.678 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 277, in client_request
2015-09-07 21:27:43.730 | 2015-09-07 21:27:43.679 | resp, body = self.json_request(method, url, **kwargs)
2015-09-07 21:27:43.730 | 2015-09-07 21:27:43.682 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 266, in json_request
2015-09-07 21:27:43.731 | 2015-09-07 21:27:43.683 | resp = self._http_request(url, method, **kwargs)
2015-09-07 21:27:43.731 | 2015-09-07 21:27:43.684 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 228, in _http_request
2015-09-07 21:27:43.731 | 2015-09-07 21:27:43.686 | resp = self._http_request(path, method, **kwargs)
2015-09-07 21:27:43.731 | 2015-09-07 21:27:43.687 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 221, in _http_request
2015-09-07 21:27:43.731 | 2015-09-07 21:27:43.690 | raise exc.from_response(resp)
2015-09-07 21:27:43.731 | 2015-09-07 21:27:43.691 | heatclient.exc.HTTPInternalServerError: ERROR: Remote error: ResourceClosedError This result object does not return ...

Read more...

tags: added: gate-failure
Changed in heat:
importance: Medium → High
milestone: none → liberty-rc1
Changed in heat:
milestone: liberty-rc1 → ongoing
Angus Salkeld (asalkeld) wrote :

I'd like to help but I ran "testr run --until-failure test_template_resource_update_template_schem" for 3 hours and didn't hit.

Zane Bitter (zaneb) wrote :

I'd bet this has the same root cause as bug 1546431.

Since, as Rabi pointed out, this can happen in any test (not just test_template_resource_update_template_schema), I updated the title to make it easier to find.

summary: - Intermittent failure in test_template_resource_update_template_schema
+ ResourceClosedError from DB in functional tests
Changed in heat:
milestone: ongoing → newton-2
Steve Baker (steve-stevebaker) wrote :

In the last 7 days, all 13 ResourceClosedError errors had a root cause of "Command Out of Sync". There were 42 "Command Out of Sync" errors in the same period.

Therefore I've marked this as a duplicate of #1546431

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers