Quota Deadlock when creating >200 servers at once under Psql

[wingwj]

When you create >200 servers at once under psql,
  it's easier to meet a deadlock exception in quota process(reserve/commit).

Like this:
http://paste.openstack.org/show/61534/

The '@retry_on_dead_lock' function, is not available for matching psql.

[Rui Chen (kiwik-chenrui)]

If error message is matched, raw DB error will be transformed to nova DBDeadlock.

there is only mysql regex string, no postgresql.

_DEADLOCK_RE_DB = {
    "mysql": re.compile(r"^.*\(1213, 'Deadlock.*")
}

look like a bug.

[wingwj (wingwj)]

I'll add the psql support in @retry_on_deadlock, and test the issue again. Thanks~

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/74743

[wingwj (wingwj)]

Link to oslo patch

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/74917

[Matt Riedemann (mriedem)]

We'd also need this for DB2:

http://pic.dhe.ibm.com/infocenter/db2luw/v10r5/topic/com.ibm.db2.luw.messages.sql.doc/doc/msql00911n.html?resultof=%22%64%65%61%64%6c%6f%63%6b%22%20%22%65%72%72%6f%72%22%20

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/91979

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/91983

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/92001

[sean mooney (sean-k-mooney)]

i belive this bug also impacts neutron and cinder

see bugs below

https://bugs.launchpad.net/neutron/+bug/1333084
https://bugs.launchpad.net/cinder/+bug/1294855

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/92001
Committed: https://git.openstack.org/cgit/openstack/oslo.db/commit/?id=3a6a5b01a72e0e20787f3d6a47f6f0c0262dbbeb
Submitter: Jenkins
Branch: master

commit 3a6a5b01a72e0e20787f3d6a47f6f0c0262dbbeb
Author: wingwj <email address hidden>
Date: Sun May 4 15:43:47 2014 +0800

    Add _wrap_db_error support for postgresql

    The original "_wrap_db_error" function is only available for mysql.
    We need to add support for postgresql.

    The deadlock exception thrown from postgresql is TransactionRollbackError,
    it's not the same with "OperationalError" thrown from mysql,
    and isn't included in sqlalchemy. So, the base exception need to be
    catched.

    Change-Id: I439a4e0c431c9446ffabd1c93f11b8f509b1ebd7
    Closes-Bug: #1270725

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/107488

[Jay Pipes (jaypipes)]

wingwj,

Pavel Kholkin has been working on refactoring the quota DB code to remove the use of SELECT FOR UPDATE (which triggers this behaviour behind the scenes in both MySQL and PostgreSQL).

If you are not currently working on this bug, can I transfer ownership of the bug (and bug 1283987) to Pavel?

Please let me know at your earliest!
-jay

[Sean Dague (sdague)]

Does not appear to be in progress in Nova

[Joe Gordon (jogo)]

Looks like this should be fixed in nova since the fix landed in oslo