Heat Translator

Stack DELETE FAILED - Resource DELETE failed: DBDeadlock

Bug #1820970 reported by kirandevraaj on 2019-03-20

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Heat Translator	Invalid	Undecided	Unassigned

Bug Description

2019-03-19 07:43:50.054 32102 DEBUG novaclient.v2.client [req-f428f37d-a52d-44d7-878a-22ad20245ddb - c_rally_d7b3a7b5_5xD2l6QH - default default] REQ: curl -g -i -X GET http://192.168.7.7:8774/v2.1/b0cb920b56964e2eb19f24d982a0b437/servers/77d2ef5a-abb1-45fe-9c10-9f54e1bd091e -H "Accept: application/json" -H "OpenStack-API-Version: compute 2.65" -H "User-Agent: python-novaclient" -H "X-Auth-Token: {SHA1}34ca550310505bee8c4e6d8374fb59e26d304712" -H "X-OpenStack-Nova-API-Version: 2.65" _http_log_request /opt/mhos/python/lib/python2.7/site-packages/keystoneauth1/session.py:448

2019-03-19 07:43:50.115 32103 DEBUG heat.engine.scheduler [req-47a66cd2-56b7-4a29-8f47-33e1106bf0ab - c_rally_d7b3a7b5_BSyhBbWb - default default] Task delete from RouterInterface "router_interface_subnet_mgmt" [170513e9-de8b-4c0d-a40a-f9a0edb8e339:subnet_id=d9c29433-18d5-4c37-af9e-e81984d83046] Stack "s_rally_d7b3a7b5_L3tqFu1S" [6ae93dee-ffae-4a35-81ae-4cf02670802e] starting start /opt/mhos/openstack/heat/lib/python2.7/site-packages/heat/engine/scheduler.py:177

2019-03-19 07:43:50.117 32103 DEBUG heat.engine.scheduler [req-47a66cd2-56b7-4a29-8f47-33e1106bf0ab - c_rally_d7b3a7b5_BSyhBbWb - default default] Task delete from RouterInterface "router_interface_subnet_mgmt" [170513e9-de8b-4c0d-a40a-f9a0edb8e339:subnet_id=d9c29433-18d5-4c37-af9e-e81984d83046] Stack "s_rally_d7b3a7b5_L3tqFu1S" [6ae93dee-ffae-4a35-81ae-4cf02670802e] running step /opt/mhos/openstack/heat/lib/python2.7/site-packages/heat/engine/scheduler.py:209

2019-03-19 07:43:50.120 32105 DEBUG heat.engine.scheduler [req-47a66cd2-56b7-4a29-8f47-33e1106bf0ab - c_rally_d7b3a7b5_BSyhBbWb - default default] Task delete from Server "server_jump" [e3450ba9-4c4c-40a4-99f0-b8bd9cfc3d52] Stack "s_rally_d7b3a7b5_L3tqFu1S" [6ae93dee-ffae-4a35-81ae-4cf02670802e] starting start /opt/mhos/openstack/heat/lib/python2.7/site packages/heat/engine/scheduler.py:177

2019-03-19 07:43:50.121 32105 DEBUG heat.engine.scheduler [req-47a66cd2-56b7-4a29-8f47-33e1106bf0ab - c_rally_d7b3a7b5_BSyhBbWb - default default] Task delete from Server "server_jump" [e3450ba9-4c4c-40a4-99f0-b8bd9cfc3d52] Stack "s_rally_d7b3a7b5_L3tqFu1S" [6ae93dee-ffae-4a35-81ae-4cf02670802e] running step /opt/mhos/openstack/heat/lib/python2.7/site-packages/heat/engine/scheduler.py:209

2019-03-19 07:43:50.122 32105 INFO heat.engine.resource [req-47a66cd2-56b7-4a29-8f47-33e1106bf0ab - c_rally_d7b3a7b5_BSyhBbWb - default default] deleting Server "server_jump" [e3450ba9-
40a4-99f0-b8bd9cfc3d52] Stack "s_rally_d7b3a7b5_L3tqFu1S" [6ae93dee-ffae-4a35-81ae-4cf02670802e]

2019-03-19 07:43:50.119 32103 INFO heat.engine.resource [req-47a66cd2-56b7-4a29-8f47-33e1106bf0ab - c_rally_d7b3a7b5_BSyhBbWb - default default] deleting RouterInterface
bnet_mgmt" [170513e9-de8b-4c0d-a40a-f9a0edb8e339:subnet_id=d9c29433-18d5-4c37-af9e-e81984d83046] Stack "s_rally_d7b3a7b5_L3tqFu1S" [6ae93dee-ffae-4a35-81ae-4cf02670802e]

2019-03-19 07:43:50.139 32103 INFO heat.engine.resource [req-47a66cd2-56b7-4a29-8f47-33e1106bf0ab - c_rally_d7b3a7b5_BSyhBbWb - default default] delete RouterInterface "router_interface_subnet_mgmt" [170513e9-de8b-4c0d-a40a-f9a0edb8e339:subnet_id=d9c29433-18d5-4c37-af9e-e81984d83046] Stack "s_rally_d7b3a7b5_L3tqFu1S" [6ae93dee-ffae-4a35-81ae-4cf02670802e] attempt 1
2019-03-19 07:43:50.141 32105 INFO heat.engine.resource [req-47a66cd2-56b7-4a29-8f47-33e1106bf0ab - c_rally_d7b3a7b5_BSyhBbWb - default default] DELETE: Server "server_jump" [e3450ba9-4c4c-40a4-99f0-b8bd9cfc3d52] Stack "s_rally_d7b3a7b5_L3tqFu1S" [6ae93dee-ffae-4a35-81ae-4cf02670802e]
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource Traceback (most recent call last):
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/heat/engine/resource.py", line 923, in _action_recorder
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource set_in_progress()
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/tenacity/__init__.py", line 241, in wrapped_f
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource return self.call(f, *args, **kw)
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/tenacity/__init__.py", line 330, in call
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource start_time=start_time)
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/tenacity/__init__.py", line 279, in iter
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource return fut.result()
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/python/lib/python2.7/site-packages/concurrent/futures/_base.py", line 455, in result
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource return self.__get_result()
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/tenacity/__init__.py", line 333, in call
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource result = fn(*args, **kwargs)
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/heat/engine/resource.py", line 920, in set_in_progress
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource self.state_set(action, self.IN_PROGRESS, lock=lock_acquire)
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/heat/engine/resource.py", line 2316, in state_set
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource self._add_event(action, status, reason)
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/heat/engine/resource.py", line 2149, in _add_event
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource ev.store()
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/heat/engine/event.py", line 67, in store
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource new_ev = event_object.Event.create(self.context, ev)
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/heat/objects/event.py", line 111, in create
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource dict(db_api.event_create(context, values)))
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 1137, in event_create
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource event_ref.save(context.session)
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py", line 50, in save
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource session.flush()
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 572, in __exit__
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource self.rollback()
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource compat.reraise(exc_type, exc_value, exc_tb)
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 569, in __exit__
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource self.commit()
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 471, in commit
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource t[1].commit()
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1643, in commit
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource self._do_commit()
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1674, in _do_commit
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource self.connection._commit_impl()
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 726, in _commit_impl
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource self._handle_dbapi_exception(e, None, None, None, None)
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource util.raise_from_cause(newraise, exc_info)
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 724, in _commit_impl
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource self.engine.dialect.do_commit(self.connection)
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/openstack/heat/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 1765, in do_commit
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource dbapi_connection.commit()
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/python/lib/python2.7/site-packages/pymysql/connections.py", line 422, in commit
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource self._read_ok_packet()
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/python/lib/python2.7/site-packages/pymysql/connections.py", line 396, in _read_ok_packet
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource pkt = self._read_packet()
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/python/lib/python2.7/site-packages/pymysql/connections.py", line 683, in _read_packet
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource packet.check_error()
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/python/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in check_error
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource err.raise_mysql_exception(self._data)
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource File "/opt/mhos/python/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource raise errorclass(errno, errval)
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource DBDeadlock: (pymysql.err.OperationalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') (Background on this error at: http://sqlalche.me/e/e3q8)
2019-03-19 07:43:50.141 32105 ERROR heat.engine.resource
2019-03-19 07:43:50.223 32105 INFO heat.engine.stack [req-47a66cd2-56b7-4a29-8f47-33e1106bf0ab - c_rally_d7b3a7b5_BSyhBbWb - default default] Stack DELETE FAILED (s_rally_d7b3a7b5_L3tqFu1S): Resource DELETE failed: DBDeadlock: resources.server_jump: (pymysql.err.OperationalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') (Background on this error at: http://sqlalche.me/e/e3q8)

2019-03-19 07:43:50.230 32105 DEBUG heat.engine.stack [req-47a66cd2-56b7-4a29-8f47-33e1106bf0ab - c_rally_d7b3a7b5_BSyhBbWb - default default] Persisting stack s_rally_d7b3a7b5_L3tqFu1S status DELETE FAILED _send_notification_and_add_event /opt/mhos/openstack/heat/lib/python2.7/site-packages/heat/engine/stack.py:1040

Revision history for this message

kirandevraaj (kirandevraaj) wrote on 2019-03-20:

Issue was seen with stable/rocky branch. Heat stack create & delete test was executed with 30 times/10 concurrency.

Revision history for this message

Bob Haddleton (bob-haddleton) wrote on 2019-03-20:

This belongs in the heat project, probably in Storyboard - https://storyboard.openstack.org/#!/project/openstack/heat - heat-translator does not have anything to do with stack create/delete.

Changed in heat-translator:
status:	New → Invalid

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.