neutron

Missing retry decorator on dhcp_ready_on_ports

Bug #1602833 reported by Henry Gessau on 2016-07-13

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	Undecided	Henry Gessau

Bug Description

2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server [req-30a8fac4-91ee-45e1-a259-19410b3a86f1 - - - - -] Exception during message handling
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 239, in dhcp_ready_on_ports
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server provisioning_blocks.DHCP_ENTITY)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/neutron/db/provisioning_blocks.py", line 141, in provisioning_complete
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server standard_attr_id):
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/neutron/db/provisioning_blocks.py", line 114, in remove_provisioning_component
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server return False
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 490, in __exit__
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server self.rollback()
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server compat.reraise(exc_type, exc_value, exc_tb)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 487, in __exit__
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server self.commit()
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 392, in commit
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server self._prepare_impl()
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server self.session.flush()
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2019, in flush
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server self._flush(objects)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2137, in _flush
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server transaction.rollback(_capture_exception=True)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server compat.reraise(exc_type, exc_value, exc_tb)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2101, in _flush
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server flush_context.execute()
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server rec.execute(self)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 561, in execute
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server uow
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 249, in delete_obj
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server cached_connections, mapper, table, delete)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 892, in _emit_delete_statements
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server c = connection.execute(statement, del_objects)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server return meth(self, multiparams, params)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server return connection._execute_clauseelement(self, multiparams, params)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server compiled_sql, distilled_params
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server context)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server util.raise_from_cause(newraise, exc_info)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server reraise(type(exception), exception, tb=exc_tb, cause=cause)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server context)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server cursor.execute(statement, parameters)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/cursors.py", line 161, in execute
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server result = self._query(query)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/cursors.py", line 317, in _query
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server conn.query(q)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/connections.py", line 837, in query
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/connections.py", line 1021, in _read_query_result
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server result.read()
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/connections.py", line 1304, in read
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server first_packet = self.connection._read_packet()
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/connections.py", line 983, in _read_packet
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server packet.check_error()
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/connections.py", line 395, in check_error
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server err.raise_mysql_exception(self._data)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server _check_mysql_exception(errinfo)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server File "/opt/bbc/openstack-11.0-master/neutron/local/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server raise InternalError(errno, errorvalue)
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server DBDeadlock: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'DELETE FROM provisioningblocks WHERE provisioningblocks.standard_attr_id = %(standard_attr_id)s AND provisioningblocks.entity = %(entity)s'] [parameters: {'standard_attr_id': 14843, 'entity': u'DHCP'}]
2016-07-12 17:03:45.124 55439 ERROR oslo_messaging.rpc.server

Tags:

Henry Gessau (gessau) on 2016-07-13

tags:

added: db l3-ipam-dhcp

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-07-13: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/341795

Changed in neutron:
assignee:	nobody → Henry Gessau (gessau)
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-07-13: Fix merged to neutron (master)

Reviewed: https://review.openstack.org/341795
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=890c30fc52eb1b16cb7f671206b1a0edeb7fb58f
Submitter: Jenkins
Branch: master

commit 890c30fc52eb1b16cb7f671206b1a0edeb7fb58f
Author: Henry Gessau <email address hidden>
Date: Wed Jul 13 16:14:23 2016 -0400

Add retry decorator to dhcp_ready_on_ports

Enable DB retries for this provisioning blocks phase.

Closes-Bug: #1602833

Change-Id: Ie6444d6552b4c1e73c00d5624e3866417204c9e8

Changed in neutron:
status:	In Progress → Fix Released

Ihar Hrachyshka (ihar-hrachyshka) on 2016-07-25

tags:

added: neutron-proactive-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-09-01: Fix included in openstack/neutron 9.0.0.0b3

This issue was fixed in the openstack/neutron 9.0.0.0b3 development milestone.

Ihar Hrachyshka (ihar-hrachyshka) on 2016-10-07

tags:

removed: neutron-proactive-backport-potential

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.