dead-lock error during cloud deployment

[Alexey Khivin]

Triyng to deploy configuration on VirtualBox

Multihost with HA, Neutron + gre
3 Controllers, 1 cinder, 1 Compute

All nodes deployed successfuly except one controller

Rest Api log contains

[7fedb3f16740] (logger) Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 420, in _delegate
    return handle_class(cls)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 396, in handle_class
    return tocall(*args)
  File "<string>", line 2, in PUT
  File "/usr/lib/python2.6/site-packages/nailgun/api/v1/handlers/base.py", line 90, in content_json
    data = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nailgun/api/v1/handlers/base.py", line 362, in PUT
    task = task_manager.execute()
  File "/usr/lib/python2.6/site-packages/nailgun/task/manager.py", line 145, in execute
    self._remove_obsolete_tasks()
  File "/usr/lib/python2.6/site-packages/nailgun/task/manager.py", line 120, in _remove_obsolete_tasks
    db().flush()
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1919, in flush
    self._flush(objects)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 2037, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 2001, in _flush
    flush_context.execute()
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 367, in execute
    n.execute_aggregate(self, set_)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 644, in execute_aggregate
    uow)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 117, in delete_obj
    cached_connections, mapper, table, delete)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 694, in _emit_delete_statements
    c = connection.execute(statement, del_objects)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 729, in execute
    return meth(self, multiparams, params)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
    context)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception
    exc_info
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 928, in _execute_context
    context)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 433, in do_executemany
    cursor.executemany(statement, parameters)
DBAPIError: (TransactionRollbackError) deadlock detected
DETAIL: Process 682 waits for ShareLock on transaction 7599; blocked by process 681.
Process 681 waits for ShareLock on transaction 7600; blocked by process 682.
HINT: See server log for query details.
 'DELETE FROM tasks WHERE tasks.id = %(id)s' ({'id': 11}, {'id': 12})

[root@fuel ~]# fuel --f
api: '1.0'
astute_sha: f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0
auth_required: true
build_id: 2015-02-18_08-17-51
build_number: '96'
feature_groups:
- mirantis
fuellib_sha: 9715623a0f215c5070ffe68c81dc9f47db361b88
fuelmain_sha: c799e3a6d88289e58db764a6be7910aab7da3149
nailgun_sha: 6967b24adc4d74e36d69b59973ff79d6ab2389e5
ostf_sha: 3b57985d4d2155510894a1f6d03b478b201f7780
production: docker
release: 6.0.1
release_versions:
  2014.2-6.0.1:
    VERSION:
      api: '1.0'
      astute_sha: f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0
      build_id: 2015-02-18_08-17-51
      build_number: '96'
      feature_groups:
      - mirantis
      fuellib_sha: 9715623a0f215c5070ffe68c81dc9f47db361b88
      fuelmain_sha: c799e3a6d88289e58db764a6be7910aab7da3149
      nailgun_sha: 6967b24adc4d74e36d69b59973ff79d6ab2389e5
      ostf_sha: 3b57985d4d2155510894a1f6d03b478b201f7780
      production: docker
      release: 6.0.1

[Ryan Moe (rmoe)]

This doesn't seem like a duplicate. 1401879 is about deadlocks in Neutron while this bug report is about a deadlock in nailgun. This also means we're dealing with PostgreSQL here as opposed to MySQL in 1401879.

[Dima Shulyak (dshulyak)]

Afaik it doesnt reproduce often enough (there was no single error of such kind during system tests), so i am marking this as medium

[Alexey Khivin (akhivin)]

It seems it is easier to reproduce this bug on a slow environment.

[Ryan Moe (rmoe)]

Medium priority bugs are not backported to stable releases.

[Alexander Kislitsky (akislitsky)]

Seems to be fixed in https://bugs.launchpad.net/fuel/+bug/1433528 (https://review.openstack.org/#/c/165355/1/nailgun/nailgun/task/manager.py - exactly in _remove_obsolete_tasks function).

Now this deadlock is unable to reproduce.

[Alexander Kislitsky (akislitsky)]

Moving bug into incomplete. Please reopen the bug, if it will be reproduced.

[Dmitry Pyzhov (dpyzhov)]

Please reopen this bug if you have more information