Deadlock in unit tests in Nailgun

[Nikolay Markov]

test_deletion_during_deployment (nailgun.test.integration.test_charset_issues.TestCharsetIssues) ... Exception in thread REMOVE_NODES:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/task/fake.py", line 358, in run
    resp_method(**kwargs)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/rpc/receiver.py", line 131, in remove_cluster_resp
    db().commit()
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 703, in commit
    self.transaction.commit()
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 361, in commit
    self._prepare_impl()
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 340, in _prepare_impl
    self.session.flush()
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 1718, in flush
    self._flush(objects)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 1789, in _flush
    flush_context.execute()
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 326, in execute
    n.execute_aggregate(self, set_)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 413, in execute_aggregate
    self.execute(uow)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 498, in execute
    uow
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 115, in delete_obj
    cached_connections, mapper, table, delete)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 671, in _emit_delete_statements
    connection.execute(statement, del_objects)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1449, in execute
    params)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context
    context)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context
    context)
  File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/engine/default.py", line 331, in do_execute
    cursor.execute(statement, parameters)
DBAPIError: (TransactionRollbackError) deadlock detected
DETAIL: Process 17008 waits for ShareLock on transaction 21657474; blocked by process 16547.
Process 16547 waits for ShareLock on transaction 21657472; blocked by process 17008.
HINT: See server log for query details.
 'DELETE FROM clusters WHERE clusters.id = %(id)s' {'id': 19}

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/89786

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/89791

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/89791
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=0da9e80d5179b739c3832e49d6b0baf04ec8bc8d
Submitter: Jenkins
Branch: master

commit 0da9e80d5179b739c3832e49d6b0baf04ec8bc8d
Author: Kamil Sambor <email address hidden>
Date: Wed Apr 23 12:15:09 2014 +0200

    Added function to deal with deadlock in tests

    Added new method used in fake threads during test. Method catch deadlock
    and retry operation in FakeDeletionThread. Removed skipping of
    problematic test.

    Change-Id: I134cd92f2f9dcf8f1e954ba5857a827ef1fec342
    Closes-Bug: #1308115

[Nikolay Markov (nmarkov)]

Fix proposed: https://review.openstack.org/#/c/93548/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/93548
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=518ccf2a36ff6a278766a45571e1eac90bd70231
Submitter: Jenkins
Branch: master

commit 518ccf2a36ff6a278766a45571e1eac90bd70231
Author: Nikolay Markov <email address hidden>
Date: Wed May 14 15:05:57 2014 +0400

    Added SELECT FOR UPDATE to prevent deadlocks in tests

    Closes-Bug: #1308115

    Change-Id: Ib72bb7a054a13c6fadfa548bfc4640b198c47cfe