DB Deadlock during boot_command_delete rally test

[Eugene Nikanorov]

MOS 7.0, Rally test boot_runcommand_delete:

The following can be found in logs:

DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'UPDATE floatingips SET status=%s WHERE floatingips.id = %s' ('DOWN', '2b9d909f-6773-45ea-8a38-21f3a18a0340')\n"

TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
TRACE oslo_messaging.rpc.dispatcher executor_callback))
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
TRACE oslo_messaging.rpc.dispatcher executor_callback)
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/l3_rpc.py", line 196, in update_floatingip_statuses
TRACE oslo_messaging.rpc.dispatcher context, fip_id, constants.FLOATINGIP_STATUS_DOWN)
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/neutron/db/l3_db.py", line 993, in update_floatingip_status
TRACE oslo_messaging.rpc.dispatcher fip_query.update({'status': status}, synchronize_session=False)
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2757, in update
TRACE oslo_messaging.rpc.dispatcher update_op.exec_()
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 897, in exec_
TRACE oslo_messaging.rpc.dispatcher self._do_exec()
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 995, in _do_exec
TRACE oslo_messaging.rpc.dispatcher update_stmt, params=self.query._params)
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 991, in execute
TRACE oslo_messaging.rpc.dispatcher bind, close_with_result=True).execute(clause, params or {})
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 729, in execute
TRACE oslo_messaging.rpc.dispatcher return meth(self, multiparams, params)
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
TRACE oslo_messaging.rpc.dispatcher return connection._execute_clauseelement(self, multiparams, params)
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
TRACE oslo_messaging.rpc.dispatcher compiled_sql, distilled_params
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
TRACE oslo_messaging.rpc.dispatcher context)
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 261, in _handle_dbapi_exception
TRACE oslo_messaging.rpc.dispatcher e, statement, parameters, cursor, context)
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1156, in _handle_dbapi_exception
TRACE oslo_messaging.rpc.dispatcher util.raise_from_cause(newraise, exc_info)
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
TRACE oslo_messaging.rpc.dispatcher reraise(type(exception), exception, tb=exc_tb)
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
TRACE oslo_messaging.rpc.dispatcher context)
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 436, in do_execute
TRACE oslo_messaging.rpc.dispatcher cursor.execute(statement, parameters)
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 205, in execute
TRACE oslo_messaging.rpc.dispatcher self.errorhandler(self, exc, value)
TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
TRACE oslo_messaging.rpc.dispatcher raise errorclass, errorvalue
TRACE oslo_messaging.rpc.dispatcher DBDeadlock: (OperationalError) (1213, 'Deadlock found when trying to get lock; try restarting transaction') 'UPDATE floatingips SET status=%s WHERE floatingips.id = %s' ('DOWN', '2b9d909f-6773-45ea-8a38-21f3a18a0340')
TRACE oslo_messaging.rpc.dispatcher
ERROR oslo_messaging._drivers.common [req-8a0e2bf8-fb3c-4b35-9071-44c682be9fa6 ] Returning exception

Should not have much impact end-user experience since the request is executed by RPC workers.

[Dina Belova (dbelova)]

This bug needs to be rechecked for 9.0 - if that's still reproducible, etc.

[Oleg Bondarev (obondarev)]

Filed upstream https://bugs.launchpad.net/neutron/+bug/1564144
Fix upstream: https://review.openstack.org/#/c/299658

[Dina Belova (dbelova)]

Adding 9.0 milestone back due to the wait-for-stable tag -> fix will arrive as a part of sync with stable/mitaka code.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/19866
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: ed1ca7dcfcc89fc19384dbbf2174ae0f6795289b
Author: Jenkins <email address hidden>
Date: Wed Apr 20 08:40:40 2016

Merge the tip of origin/stable/mitaka into origin/9.0/mitaka

643b443 Imported Translations from Zanata
1ffea42 Updated from global requirements
b970ed5 Clear DVR MAC on last agent deletion from host
eee9e58 Add an option for WSGI pool size
93795a4 Fix deprecation warning for external_network_bridge
36305c0 Add ALLOCATING state to routers
07fa372 ADDRESS_SCOPE_MARK_IDS should not be global for L3 agent
9c58ae6 Wrap all update/delete l3_rpc handlers with retries
ece192b Use new DB context when checking if agent is online during rescheduling
2e2d75c ovsfw: Load vlan tag from other_config
5853af9 Iptables firewall prevent IP spoofed DHCP requests
9679285 Return oslo_config Opts to config generator
e2676ae DVR: rebind port if ofport changes

Closes-Bug: #1566689
Closes-Bug: #1496723
Closes-Bug: #1523479
Closes-Bug: #1561509

Change-Id: Id18fd3ba2fa15369748828c462e8e888ccecc0de

[Timur Nurlygayanov (tnurlygayanov)]

Verified on MOS 9.0 #495.

[Alexander Ignatov (aignatov)]

This bug is fixed in Newton upstream so closing it as Invalid.