dead-lock error during cloud deployment

Bug #1423503 reported by Alexey Khivin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
Medium
Alexander Kislitsky
6.0.x
Won't Fix
Medium
Fuel Python (Deprecated)

Bug Description

Triyng to deploy configuration on VirtualBox

Multihost with HA, Neutron + gre
3 Controllers, 1 cinder, 1 Compute

All nodes deployed successfuly except one controller

Rest Api log contains

[7fedb3f16740] (logger) Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 420, in _delegate
    return handle_class(cls)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 396, in handle_class
    return tocall(*args)
  File "<string>", line 2, in PUT
  File "/usr/lib/python2.6/site-packages/nailgun/api/v1/handlers/base.py", line 90, in content_json
    data = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nailgun/api/v1/handlers/base.py", line 362, in PUT
    task = task_manager.execute()
  File "/usr/lib/python2.6/site-packages/nailgun/task/manager.py", line 145, in execute
    self._remove_obsolete_tasks()
  File "/usr/lib/python2.6/site-packages/nailgun/task/manager.py", line 120, in _remove_obsolete_tasks
    db().flush()
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1919, in flush
    self._flush(objects)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 2037, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 2001, in _flush
    flush_context.execute()
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 367, in execute
    n.execute_aggregate(self, set_)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 644, in execute_aggregate
    uow)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 117, in delete_obj
    cached_connections, mapper, table, delete)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 694, in _emit_delete_statements
    c = connection.execute(statement, del_objects)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 729, in execute
    return meth(self, multiparams, params)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
    context)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception
    exc_info
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 928, in _execute_context
    context)
  File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 433, in do_executemany
    cursor.executemany(statement, parameters)
DBAPIError: (TransactionRollbackError) deadlock detected
DETAIL: Process 682 waits for ShareLock on transaction 7599; blocked by process 681.
Process 681 waits for ShareLock on transaction 7600; blocked by process 682.
HINT: See server log for query details.
 'DELETE FROM tasks WHERE tasks.id = %(id)s' ({'id': 11}, {'id': 12})

[root@fuel ~]# fuel --f
api: '1.0'
astute_sha: f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0
auth_required: true
build_id: 2015-02-18_08-17-51
build_number: '96'
feature_groups:
- mirantis
fuellib_sha: 9715623a0f215c5070ffe68c81dc9f47db361b88
fuelmain_sha: c799e3a6d88289e58db764a6be7910aab7da3149
nailgun_sha: 6967b24adc4d74e36d69b59973ff79d6ab2389e5
ostf_sha: 3b57985d4d2155510894a1f6d03b478b201f7780
production: docker
release: 6.0.1
release_versions:
  2014.2-6.0.1:
    VERSION:
      api: '1.0'
      astute_sha: f7cda2171b0b677dfaeb59693d980a2d3ee4c3e0
      build_id: 2015-02-18_08-17-51
      build_number: '96'
      feature_groups:
      - mirantis
      fuellib_sha: 9715623a0f215c5070ffe68c81dc9f47db361b88
      fuelmain_sha: c799e3a6d88289e58db764a6be7910aab7da3149
      nailgun_sha: 6967b24adc4d74e36d69b59973ff79d6ab2389e5
      ostf_sha: 3b57985d4d2155510894a1f6d03b478b201f7780
      production: docker
      release: 6.0.1

Revision history for this message
Alexey Khivin (akhivin) wrote :
Revision history for this message
Ryan Moe (rmoe) wrote :

This doesn't seem like a duplicate. 1401879 is about deadlocks in Neutron while this bug report is about a deadlock in nailgun. This also means we're dealing with PostgreSQL here as opposed to MySQL in 1401879.

Changed in fuel:
milestone: 6.0.1 → 6.1
assignee: Fuel Library Team (fuel-library) → Fuel Python Team (fuel-python)
Revision history for this message
Dima Shulyak (dshulyak) wrote :

Afaik it doesnt reproduce often enough (there was no single error of such kind during system tests), so i am marking this as medium

Changed in fuel:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Alexey Khivin (akhivin) wrote :

It seems it is easier to reproduce this bug on a slow environment.

Revision history for this message
Ryan Moe (rmoe) wrote :

Medium priority bugs are not backported to stable releases.

Dmitry Pyzhov (dpyzhov)
tags: added: feature-deadlocks
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Alexander Kislitsky (akislitsky)
Revision history for this message
Alexander Kislitsky (akislitsky) wrote :

Seems to be fixed in https://bugs.launchpad.net/fuel/+bug/1433528 (https://review.openstack.org/#/c/165355/1/nailgun/nailgun/task/manager.py - exactly in _remove_obsolete_tasks function).

Now this deadlock is unable to reproduce.

Revision history for this message
Alexander Kislitsky (akislitsky) wrote :

Moving bug into incomplete. Please reopen the bug, if it will be reproduced.

Changed in fuel:
status: Confirmed → Incomplete
Revision history for this message
Dmitry Pyzhov (dpyzhov) wrote :

Please reopen this bug if you have more information

Changed in fuel:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.