DBDeadlock error while updating stack in DB

Bug #1578615 reported by Anant Patil
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Undecided
Anant Patil

Bug Description

Functional test failed for https://review.openstack.org/#/c/312841/ due to ERROR: Remote error: DBDeadlock (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction')

traceback:
Traceback (most recent call last):
2016-05-05 07:44:49.604 | 2016-05-05 07:44:49.492 | File "/opt/stack/new/heat/heat_integrationtests/common/test.py", line 347, in _stack_delete
2016-05-05 07:44:49.604 | 2016-05-05 07:44:49.494 | stack_identifier)
2016-05-05 07:44:49.604 | 2016-05-05 07:44:49.496 | File "/opt/stack/new/heat/heat_integrationtests/common/test.py", line 361, in _handle_in_progress
2016-05-05 07:44:49.604 | 2016-05-05 07:44:49.498 | fn(*args, **kwargs)
2016-05-05 07:44:49.604 | 2016-05-05 07:44:49.501 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 192, in delete
2016-05-05 07:44:49.604 | 2016-05-05 07:44:49.503 | self._delete("/stacks/%s" % stack_id)
2016-05-05 07:44:49.604 | 2016-05-05 07:44:49.505 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/openstack/common/apiclient/base.py", line 228, in _delete
2016-05-05 07:44:49.605 | 2016-05-05 07:44:49.507 | return self.client.delete(url)
2016-05-05 07:44:49.605 | 2016-05-05 07:44:49.510 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 300, in delete
2016-05-05 07:44:49.605 | 2016-05-05 07:44:49.511 | return self.raw_request("DELETE", url, **kwargs)
2016-05-05 07:44:49.605 | 2016-05-05 07:44:49.513 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 281, in raw_request
2016-05-05 07:44:49.605 | 2016-05-05 07:44:49.515 | return self._http_request(url, method, **kwargs)
2016-05-05 07:44:49.606 | 2016-05-05 07:44:49.519 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 235, in _http_request
2016-05-05 07:44:49.606 | 2016-05-05 07:44:49.521 | resp = self._http_request(path, method, **kwargs)
2016-05-05 07:44:49.606 | 2016-05-05 07:44:49.523 | File "/opt/stack/new/heat/.tox/integration/local/lib/python2.7/site-packages/heatclient/common/http.py", line 228, in _http_request
2016-05-05 07:44:49.606 | 2016-05-05 07:44:49.526 | raise exc.from_response(resp)
2016-05-05 07:44:49.606 | 2016-05-05 07:44:49.528 | heatclient.exc.HTTPInternalServerError: ERROR: Remote error: DBDeadlock (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'UPDATE stack SET action=%(action)s, status=%(status)s, status_reason=%(status_reason)s, name=%(name)s, raw_template_id=%(raw_template_id)s, prev_raw_template_id=%(prev_raw_template_id)s, username=%(username)s, tenant=%(tenant)s, user_creds_id=%(user_creds_id)s, owner_id=%(owner_id)s, parent_resource_name=%(parent_resource_name)s, timeout=%(timeout)s, disable_rollback=%(disable_rollback)s, stack_user_project_id=%(stack_user_project_id)s, backup=%(backup)s, nested_depth=%(nested_depth)s, convergence=%(convergence)s, current_traversal=%(current_traversal)s, current_deps=%(current_deps)s, updated_at=%(updated_at)s WHERE stack.id = %(id_1)s AND stack.current_traversal = %(current_traversal_1)s'] [parameters: {'username': u'demo', 'updated_at': datetime.datetime(2016, 5, 5, 7, 37, 48, 98572), u'current_traversal_1': u'e5ec8254-6e2d-429c-8736-727d5b1744ef', 'prev_raw_template_id': 1341, 'current_traversal': 'c6e6d055-5185-4707-8e74-257266473a8d', 'parent_resource_name': None, 'current_deps': '{"edges": [[[736, true], [744, true]], [[743, true], [744, true]], [[739, true], [744, true]], [[740, true], [744, true]], [[737, true], [744, true]]]}', 'owner_id': None, 'status': 'IN_PROGRESS', 'disable_rollback': 1, 'stack_user_project_id': u'512e6ba7c6634a4d81a02231d8881eab', 'convergence': 1, 'raw_template_id': 1448, 'status_reason': 'Stack DELETE started', 'tenant': u'26148ef5c23e487e917337e3e34b920f', 'name': u'ParallelDeploymentsTest-1320370170', 'nested_depth': 0, u'id_1': u'4546ccca-80a6-4b36-8cdb-3a6fdc39aab8', 'user_creds_id': 119, 'timeout': None, 'action': 'DELETE', 'backup': 0}]

The stack was being deleted and while updating the DB with new action and status, this exception was raised.

Full message:
http://logs.openstack.org/41/312841/1/check/gate-heat-dsvm-functional-convg-mysql-lbaasv1/5ad6f0b/console.html

Anant Patil (ananta)
Changed in heat:
assignee: nobody → Anant Patil (ananta)
Anant Patil (ananta)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/312936

Changed in heat:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/312936
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=6f361f7da39c9567456d05523358aa6b65221798
Submitter: Jenkins
Branch: master

commit 6f361f7da39c9567456d05523358aa6b65221798
Author: Anant Patil <email address hidden>
Date: Thu May 5 18:35:38 2016 +0530

    Fix DBDeadlock error in stack update

    Fix the DBDeadlock error encounter occasionally when stack is being
    updated in DB.

    Change-Id: I24a929d88d87c5c6d6b410ad990d8d027f4c5b1b
    Closes-Bug: #1578615

Changed in heat:
status: In Progress → Fix Released
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/heat 7.0.0.0b1

This issue was fixed in the openstack/heat 7.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.