reservation_commit can lead to deadlock

[Vish Ishaya]

reservation_commit can deadlock under load:

2014-01-29 18:51:00.470 ERROR nova.quota [req-659db2df-6124-4e6a-b86b-700aae1de805 183137d95c584efb84e773a21f2ef7a1 d8d5b06deffc45e7b258eb65ea04017c] Failed to commit reservations ['5bea6421-1648-4fe1-9d21-4232f536e031', '8b27edda-f40e-476c-a9b3-d007aa3f6aac', '58e357df-b45d-4008-9ef3-0da3d8daebbb']
2014-01-29 18:51:00.470 4380 TRACE nova.quota Traceback (most recent call last):
2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/nova/quota.py", line 982, in commit
2014-01-29 18:51:00.470 4380 TRACE nova.quota self._driver.commit(context, reservations, project_id=project_id)
2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/nova/quota.py", line 370, in commit
2014-01-29 18:51:00.470 4380 TRACE nova.quota db.reservation_commit(context, reservations, project_id=project_id)
2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 970, in reservation_commit
2014-01-29 18:51:00.470 4380 TRACE nova.quota project_id=project_id)
2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 114, in wrapper
2014-01-29 18:51:00.470 4380 TRACE nova.quota return f(*args, **kwargs)
2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 2786, in reservation_commit
2014-01-29 18:51:00.470 4380 TRACE nova.quota for reservation in reservation_query.all():
2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2115, in all
2014-01-29 18:51:00.470 4380 TRACE nova.quota return list(self)
2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
2014-01-29 18:51:00.470 4380 TRACE nova.quota return self._execute_and_instances(context)
2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances
2014-01-29 18:51:00.470 4380 TRACE nova.quota result = conn.execute(querycontext.statement, self._params)
2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1449, in execute
2014-01-29 18:51:00.470 4380 TRACE nova.quota params)
2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
2014-01-29 18:51:00.470 4380 TRACE nova.quota compiled_sql, distilled_params
2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
2014-01-29 18:51:00.470 4380 TRACE nova.quota context)
2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
2014-01-29 18:51:00.470 4380 TRACE nova.quota context)
2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 331, in do_execute
2014-01-29 18:51:00.470 4380 TRACE nova.quota cursor.execute(statement, parameters)
2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
2014-01-29 18:51:00.470 4380 TRACE nova.quota self.errorhandler(self, exc, value)
2014-01-29 18:51:00.470 4380 TRACE nova.quota File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2014-01-29 18:51:00.470 4380 TRACE nova.quota raise errorclass, errorvalue
2014-01-29 18:51:00.470 4380 TRACE nova.quota OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'SELECT reservations.created_at AS reservations_created_at, reservations.updated_at AS reservations_updated_at, reservations.deleted_at AS reservations_deleted_at, reservations.deleted AS reservations_deleted, reservations.id AS reservations_id, reservations.uuid AS reservations_uuid, reservations.usage_id AS reservations_usage_id, reservations.project_id AS reservations_project_id, reservations.resource AS reservations_resource, reservations.delta AS reservations_delta, reservations.expire AS reservations_expire \nFROM reservations \nWHERE reservations.deleted = %s AND reservations.uuid IN (%s, %s, %s) FOR UPDATE' (0, '5bea6421-1648-4fe1-9d21-4232f536e031', '8b27edda-f40e-476c-a9b3-d007aa3f6aac', '58e357df-b45d-4008-9ef3-0da3d8daebbb')

This can be fixed with our @_retry_on_deadlock wrapper

[Liyingjun (liyingjun)]

https://review.openstack.org/#/c/70027/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/70027
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9c556d1557a7056d6056b25d3b91ff4abcb97971
Submitter: Jenkins
Branch: master

commit 9c556d1557a7056d6056b25d3b91ff4abcb97971
Author: Vishvananda Ishaya <email address hidden>
Date: Wed Jan 29 16:40:26 2014 -0800

    Retry reservation commit and rollback on deadlock

    Reservations can deadlock when concurrent commits are run from
    multiple hosts on the same project. This fixes the issue by using
    our convienient retry on deadlock feature.

    Change-Id: I31d74f4267eaa33e798de926069d1a0f63006a59
    Closes-bug: #1274341

[Joe Gordon (jogo)]

Is this a duplicate of bug 1250173?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/72546

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/72546
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0d3d7c94b34468980469f412739c5a2aaf633c79
Submitter: Jenkins
Branch: stable/havana

commit 0d3d7c94b34468980469f412739c5a2aaf633c79
Author: Vishvananda Ishaya <email address hidden>
Date: Wed Jan 29 16:40:26 2014 -0800

    Retry reservation commit and rollback on deadlock

    Reservations can deadlock when concurrent commits are run from
    multiple hosts on the same project. This fixes the issue by using
    our convienient retry on deadlock feature.

    Change-Id: I31d74f4267eaa33e798de926069d1a0f63006a59
    Closes-bug: #1274341
    (cherry picked from commit 9c556d1557a7056d6056b25d3b91ff4abcb97971)