These tests are failed time to time:
nailgun.test.integration.test_task_managers.TestTaskManagers.test_deletion_during_deployment
nailgun.test.integration.test_charset_issues.TestCharsetIssues.test_deletion_during_deployment
Fail reason is deadlock as it's shown in logs:
test_deletion_during_deployment (nailgun.test.integration.test_task_managers.TestTaskManagers) ... Exception in thread REMOVE_NODES:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/task/fake.py", line 386, in run
handling_db_errors(resp_method, **kwargs)
File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/task/fake.py", line 60, in handling_db_errors
handling_db_errors(func, retries=retries, **kwargs)
File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/task/fake.py", line 46, in handling_db_errors
func(**kwargs)
File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/rpc/receiver.py", line 115, in remove_cluster_resp
cls.remove_nodes_resp(**kwargs)
File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/rpc/receiver.py", line 55, in remove_nodes_resp
node_db = db().query(Node).get(node['uid'])
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 775, in get
return self._load_on_ident(key)
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2512, in _load_on_ident
return q.one()
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2184, in one
ret = list(self)
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
return self._execute_and_instances(context)
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2240, in _execute_and_instances
close_with_result=True)
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2231, in _connection_from_session
**kw)
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 777, in connection
close_with_result=close_with_result)
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 781, in _connection_for_bind
return self.transaction._connection_for_bind(engine)
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 289, in _connection_for_bind
self._assert_is_active()
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 213, in _assert_is_active
% self._rollback_exception
InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (TransactionRollbackError) deadlock detected
DETAIL: Process 23546 waits for ShareLock on transaction 24210489; blocked by process 23157.
Process 23157 waits for ShareLock on transaction 24210487; blocked by process 23546.
HINT: See server log for query details.
'DELETE FROM clusters WHERE clusters.id = %(id)s' {'id': 274}
ERROR
test_deletion_during_deployment (nailgun.test.integration.test_charset_issues.TestCharsetIssues) ... Exception in thread REMOVE_NODES:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/task/fake.py", line 386, in run
handling_db_errors(resp_method, **kwargs)
File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/task/fake.py", line 60, in handling_db_errors
handling_db_errors(func, retries=retries, **kwargs)
File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/task/fake.py", line 46, in handling_db_errors
func(**kwargs)
File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/rpc/receiver.py", line 115, in remove_cluster_resp
cls.remove_nodes_resp(**kwargs)
File "/home/jenkins/workspace/stackforge-verify-fuel-web/nailgun/nailgun/rpc/receiver.py", line 55, in remove_nodes_resp
node_db = db().query(Node).get(node['uid'])
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 775, in get
return self._load_on_ident(key)
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2512, in _load_on_ident
return q.one()
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2184, in one
ret = list(self)
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
return self._execute_and_instances(context)
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2240, in _execute_and_instances
close_with_result=True)
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/query.py", line 2231, in _connection_from_session
**kw)
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 777, in connection
close_with_result=close_with_result)
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 781, in _connection_for_bind
return self.transaction._connection_for_bind(engine)
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 289, in _connection_for_bind
self._assert_is_active()
File "/home/jenkins/workspace/stackforge-verify-fuel-web_VENV/lib/python2.6/site-packages/sqlalchemy/orm/session.py", line 213, in _assert_is_active
% self._rollback_exception
InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (TransactionRollbackError) deadlock detected
DETAIL: Process 10497 waits for ShareLock on transaction 24280499; blocked by process 10482.
Process 10482 waits for ShareLock on transaction 24280497; blocked by process 10497.
HINT: See server log for query details.
'DELETE FROM clusters WHERE clusters.id = %(id)s' {'id': 19}
ERROR
It's very annoying thing as tests fail in about 1/4 of CI test runs.
Please look at /fuel-jenkins. mirantis. com/job/ stackforge- verify- fuel-web/ 3416/consoleFul l /fuel-jenkins. mirantis. com/job/ stackforge- verify- fuel-web/ 3426/consoleFul l
https:/
https:/
for examples.
All last 10 nailgun test run fails are connected with this problem.