neutron

Mysql/Eventlet deadlock when creating a port

Bug #1337690 reported by Magesh GV
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Expired
Medium
Unassigned

Bug Description

Another instance of the Mysql/Eventlet deadlock which is occuring while trying to bind the network to a DHCP agent during port create

2014-07-02 22:17:43.322 31258 ERROR neutron.api.v2.resource [-] create failed
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource Traceback (most recent call last):
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 84, in resource
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource result = method(request=request, **args)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 407, in create
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource obj)})
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 386, in notify
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource notifier_method)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 268, in _send_dhcp_notification
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource self._dhcp_agent_notifier.notify(context, data, methodname)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py", line 153, in notify
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource self._notification(context, methodname, data, network_id)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/agentnotifiers/dhcp_rpc_agent_api.py", line 71, in _notification
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource chosen_agents = plugin.schedule_network(adminContext, network)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/db/agentschedulers_db.py", line 207, in schedule_network
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource self, context, created_network)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/scheduler/dhcp_agent_scheduler.py", line 83, in schedule
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource self._schedule_bind_network(context, agent, network['id'])
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 447, in __exit__
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource self.rollback()
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource compat.reraise(exc_type, exc_value, exc_tb)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 444, in __exit__
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource self.commit()
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 354, in commit
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource self._prepare_impl()
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _prepare_impl
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource self.session.flush()
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/db/sqlalchemy/session.py", line 524, in _wrap
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource return f(*args, **kwargs)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/openstack/common/db/sqlalchemy/session.py", line 718, in flush
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource return super(Session, self).flush(*args, **kwargs)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1818, in flush
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource self._flush(objects)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1936, in _flush
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource transaction.rollback(_capture_exception=True)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 58, in __exit__
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource compat.reraise(exc_type, exc_value, exc_tb)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1900, in _flush
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource flush_context.execute()
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource rec.execute(self)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 525, in execute
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource uow
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 64, in save_obj
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource table, insert)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 541, in _emit_insert_statements
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource execute(statement, multiparams)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 662, in execute
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource params)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource compiled_sql, distilled_params
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 874, in _execute_context
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource context)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1024, in _handle_dbapi_exception
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource exc_info
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 195, in raise_from_cause
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource reraise(type(exception), exception, tb=exc_tb)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 867, in _execute_context
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource context)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 324, in do_execute
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource cursor.execute(statement, parameters)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource self.errorhandler(self, exc, value)
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource raise errorclass, errorvalue
2014-07-02 22:17:43.322 31258 TRACE neutron.api.v2.resource OperationalError: (OperationalError) (1205, 'Lock wait timeout exceeded; try restarting transaction') 'INSERT INTO networkdhcpagentbindings (network_id, dhcp_agent_id) VALUES (%s, %s)' ('20a4c19b-7931-4815-8eb8-337ab158785f', '130e492d-8e74-46bf-a6b2-47e857f592c4')

Tags: db
Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

Hi, could you please post a link to a jenkins job logs where this happened?

tags: added: db
Changed in neutron:
importance: Undecided → High
Revision history for this message
Magesh GV (magesh-gv) wrote :

This was not detected during a Jenkins job, but rather on a test deployment.

Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

This issue might have been fixed in the scope of a few commits that removed rpc calls inside transactions.

Changed in neutron:
importance: High → Medium
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for neutron because there has been no activity for 60 days.]

Changed in neutron:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.