Deadlock in reservation commit

[Matthew Booth]

Details in http://logs.openstack.org/46/104146/15/check/check-tempest-dsvm-full/d235389/, specifically in n-cond logs:

2014-08-12 14:58:57.099 ERROR nova.quota [req-7efe48be-f5b4-4343-898a-5b4b32694530 AggregatesAdminTestJSON-719157131 AggregatesAdminTestJSON-1908648657] Failed to commit reservations [u'5bdde344-b26f-4e0a-9aa7-d91d775b6df0', u'5f757426-8f4e-454f-aedb-1186771f85fd', u'819aeaf6-9faf-4da5-a16d-ce1c571c4975']
2014-08-12 14:58:57.099 21994 TRACE nova.quota Traceback (most recent call last):
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/opt/stack/new/nova/nova/quota.py", line 1326, in commit
2014-08-12 14:58:57.099 21994 TRACE nova.quota user_id=user_id)
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/opt/stack/new/nova/nova/quota.py", line 569, in commit
2014-08-12 14:58:57.099 21994 TRACE nova.quota user_id=user_id)
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/opt/stack/new/nova/nova/db/api.py", line 1148, in reservation_commit
2014-08-12 14:58:57.099 21994 TRACE nova.quota user_id=user_id)
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 167, in wrapper
2014-08-12 14:58:57.099 21994 TRACE nova.quota return f(*args, **kwargs)
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 205, in wrapped
2014-08-12 14:58:57.099 21994 TRACE nova.quota return f(*args, **kwargs)
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 3302, in reservation_commit
2014-08-12 14:58:57.099 21994 TRACE nova.quota for reservation in reservation_query.all():
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2241, in all
2014-08-12 14:58:57.099 21994 TRACE nova.quota return list(self)
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-08-12 14:58:57.099 21994 TRACE nova.quota return self._execute_and_instances(context)
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2368, in _execute_and_instances
2014-08-12 14:58:57.099 21994 TRACE nova.quota result = conn.execute(querycontext.statement, self._params)
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 662, in execute
2014-08-12 14:58:57.099 21994 TRACE nova.quota params)
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
2014-08-12 14:58:57.099 21994 TRACE nova.quota compiled_sql, distilled_params
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
2014-08-12 14:58:57.099 21994 TRACE nova.quota context)
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1024, in _handle_dbapi_exception
2014-08-12 14:58:57.099 21994 TRACE nova.quota exc_info
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 196, in raise_from_cause
2014-08-12 14:58:57.099 21994 TRACE nova.quota reraise(type(exception), exception, tb=exc_tb)
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 867, in _execute_context
2014-08-12 14:58:57.099 21994 TRACE nova.quota context)
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 324, in do_execute
2014-08-12 14:58:57.099 21994 TRACE nova.quota cursor.execute(statement, parameters)
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
2014-08-12 14:58:57.099 21994 TRACE nova.quota self.errorhandler(self, exc, value)
2014-08-12 14:58:57.099 21994 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2014-08-12 14:58:57.099 21994 TRACE nova.quota raise errorclass, errorvalue
2014-08-12 14:58:57.099 21994 TRACE nova.quota OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'SELECT reservations.created_at AS reservations_created_at, reservations.updated_at AS reservations_updated_at, reservations.deleted_at AS reservations_deleted_at, reservations.deleted AS reservations_deleted, reservations.id AS reservations_id, reservations.uuid AS reservations_uuid, reservations.usage_id AS reservations_usage_id, reservations.project_id AS reservations_project_id, reservations.user_id AS reservations_user_id, reservations.resource AS reservations_resource, reservations.delta AS reservations_delta, reservations.expire AS reservations_expire \nFROM reservations \nWHERE reservations.deleted = %s AND reservations.uuid IN (%s, %s, %s) FOR UPDATE' (0, '5bdde344-b26f-4e0a-9aa7-d91d775b6df0', '5f757426-8f4e-454f-aedb-1186771f85fd', '819aeaf6-9faf-4da5-a16d-ce1c571c4975')
2014-08-12 14:58:57.099 21994 TRACE nova.quota