After applying fix [1], still seeing error where original exception is replaced with another one in rollback block [2]. This is where original analysis(from ticket description) applies. So fix [3] is still needed. 2016-04-19 00:50:30.716 18062 ERROR neutron.db.ipam_pluggable_backend [req-0ed5f0d8-42a0-47e4-89e2-bdf6fef3e1dc tempest-PortsTestJSON-447574898 -] IP deallocation failed on external system for 10.100.0.5, 10.100.0.20 2016-04-19 00:50:30.726 18062 ERROR root [req-0ed5f0d8-42a0-47e4-89e2-bdf6fef3e1dc tempest-PortsTestJSON-447574898 -] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 310, in update_port_with_ips\n ip[\'subnet_id\'], ip[\'ip_address\'])\n', ' File "/opt/stack/new/neutron/neutron/db/db_base_plugin_common.py", line 92, in _delete_ip_allocation\n subnet_id=subnet_id).delete()\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3048, in delete\n delete_op.exec_()\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 1125, in exec_\n self._do_pre()\n', ' File "", line 1, in \n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 1163, in _do_pre\n session._autoflush()\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1293, in _autoflush\n self.flush()\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2019, in flush\n self._flush(objects)\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2137, in _flush\n transaction.rollback(_capture_exception=True)\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2101, in _flush\n flush_context.execute()\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute\n rec.execute(self)\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute\n uow\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 174, in save_obj\n mapper, table, insert)\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 767, in _emit_insert_statements\n execute(statement, multiparams)\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute\n return meth(self, multiparams, params)\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection\n return connection._execute_clauseelement(self, multiparams, params)\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement\n compiled_sql, distilled_params\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context\n context)\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception\n util.raise_from_cause(newraise, exc_info)\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause\n reraise(type(exception), exception, tb=exc_tb, cause=cause)\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context\n context)\n', ' File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute\n cursor.execute(statement, parameters)\n', ' File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 158, in execute\n result = self._query(query)\n', ' File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 308, in _query\n conn.query(q)\n', ' File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 820, in query\n self._affected_rows = self._read_query_result(unbuffered=unbuffered)\n', ' File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1002, in _read_query_result\n result.read()\n', ' File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1285, in read\n first_packet = self.connection._read_packet()\n', ' File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 966, in _read_packet\n packet.check_error()\n', ' File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 394, in check_error\n err.raise_mysql_exception(self._data)\n', ' File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception\n _check_mysql_exception(errinfo)\n', ' File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 115, in _check_mysql_exception\n raise InternalError(errno, errorvalue)\n', "DBDeadlock: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'INSERT INTO ipamallocations (ip_address, status, ipam_subnet_id) VALUES (%(ip_address)s, %(status)s, %(ipam_subnet_id)s)'] [parameters: {'status': 'ALLOCATED', 'ip_address': u'10.100.0.20', 'ipam_subnet_id': u'51995705-2d27-436d-9e93-f666484cb302'}]\n"] 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource [req-0ed5f0d8-42a0-47e4-89e2-bdf6fef3e1dc tempest-PortsTestJSON-447574898 -] update failed 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource Traceback (most recent call last): 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 84, in resource 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource result = method(request=request, **args) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 579, in update 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource return self._update(request, id, body, **kwargs) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource self.force_reraise() 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource return f(*args, **kwargs) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 624, in _update 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource obj = obj_updater(request.context, id, **kwargs) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1215, in update_port 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource port) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1296, in update_port 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource new_port, new_mac) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 322, in update_port_with_ips 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource revert_on_fail=False) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 75, in _ipam_deallocate_ips 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource "external system for %s"), addresses) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource self.force_reraise() 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 58, in _ipam_deallocate_ips 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource ipam_subnet = ipam_driver.get_subnet(ip['subnet_id']) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 401, in get_subnet 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource return NeutronDbSubnet.load(subnet_id, self._context) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 94, in load 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource ctx.session, neutron_subnet_id) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 31, in load_by_neutron_subnet_id 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource neutron_subnet_id=neutron_subnet_id).first() 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2634, in first 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource ret = list(self[0:1]) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__ 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource return list(res) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__ 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource return self._execute_and_instances(context) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource close_with_result=True) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource **kw) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource execution_options=execution_options) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource engine, execution_options) 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 313, in _connection_for_bind 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource self._assert_active() 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 214, in _assert_active 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource % self._rollback_exception 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (pymysql.err.InternalError) (1213, u'Deadlock found when trying to get lock; try restarting transaction') [SQL: u'INSERT INTO ipamallocations (ip_address, status, ipam_subnet_id) VALUES (%(ip_address)s, %(status)s, %(ipam_subnet_id)s)'] [parameters: {'status': 'ALLOCATED', 'ip_address': u'10.100.0.20', 'ipam_subnet_id': u'51995705-2d27-436d-9e93-f666484cb302'}] 2016-04-19 00:50:30.727 18062 ERROR neutron.api.v2.resource [1] https://review.openstack.org/307329 [2] http://logs.openstack.org/23/181023/67/check/gate-tempest-dsvm-neutron-full/a146aeb/logs/screen-q-svc.txt.gz#_2016-04-19_00_50_30_716 [3] https://review.openstack.org/#/c/306519/