quota reservations can raise DBDeadlock

[Chris Behrens]

Due to how innodb works and that we do a select+insert... a couple of parallel select+inserts can raise DBDeadlock when using thread pooling:

2013-06-21 03:37:36.763 4585 TRACE nova DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'INSERT INTO quota_usages (created_at, updated_at, deleted_at, deleted, project_id, resource, in_use, reserved, until_refresh) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)' (datetime.datetime(2013, 6, 21, 3, 37, 36, 758605), None, None, 0, 'fake', 'cores', 0, 0, None)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/33915

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/33915
Committed: http://github.com/openstack/nova/commit/c0ca847286b88f37e3b4cb6b394fced4b60ed06f
Submitter: Jenkins
Branch: master

commit c0ca847286b88f37e3b4cb6b394fced4b60ed06f
Author: Chris Behrens <email address hidden>
Date: Fri Jun 21 03:41:54 2013 +0000

    Retry quota_reserve on DBDeadlock

    quota_reserve() can raise DBDeadlock... so this adds the decorator that
    retries the transaction.

    Fixes bug 1193199

    Change-Id: Id632360372b21f3e75a2af50153eb18cc2025390

[Matt Riedemann (mriedem)]

Seeing a similar deadlock on the old side of grenade which would be stable/icehouse now:

http://logs.openstack.org/60/109660/7/check/check-grenade-dsvm-partial-ncpu/deb4b82/logs/old/screen-n-api.txt.gz#_2014-07-28_19_59_01_933

This is in a security group test but I'm not sure that matters.

Bug 1297590 seems related.

[Matt Riedemann (mriedem)]

Opened bug 1350064 for the grenade race.