Traceback from comment 11 is not the cause for network delete failure, it rather shows that db retry works. Logs from snapshot show that port failed to be deleted first time (DBDeadlock), but then was deleted on a second attempt (thanks to db retry).
Network deletion failure was caused by ObjectDeletedError. So this is not a regression, I'd suggest to open another bug for this.
There is a race on network deletion where dhcp ports of the network may be deleted concurrently: by dhcp agent issuing port deletion on server side and by neutron server itself when deleting network. This may lead to ObjectDeletedError on one of neutron servers. Traceback:
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin Traceback (most recent call last):
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 681, in _delete_ports
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin self.delete_port(context, port.id)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 131, in wrapper
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin return f(*args, **kwargs)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 1288, in delete_port
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin super(Ml2Plugin, self).delete_port(context, id)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 1915, in delete_port
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin self._delete_port(context, id)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 1938, in _delete_port
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin query.delete()
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2670, in delete
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin delete_op.exec_()
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 896, in exec_
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin self._do_pre_synchronize()
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 958, in _do_pre_synchronize
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin eval_condition(obj)]
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/evaluator.py", line 115, in evaluate
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin left_val = eval_left(obj)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/evaluator.py", line 72, in <lambda>
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin return lambda obj: get_corresponding_attr(obj)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 239, in __get__
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin return self.impl.get(instance_state(instance), dict_)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 589, in get
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin value = callable_(state, passive)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/state.py", line 424, in __call__
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin self.manager.deferred_scalar_loader(self, toload)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 614, in load_scalar_attributes
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin raise orm_exc.ObjectDeletedError(state)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin ObjectDeletedError: Instance '<Port at 0x7f4ed51ceb10>' has been deleted, or its row is otherwise not present.
Traceback from comment 11 is not the cause for network delete failure, it rather shows that db retry works. Logs from snapshot show that port failed to be deleted first time (DBDeadlock), but then was deleted on a second attempt (thanks to db retry).
Network deletion failure was caused by ObjectDeletedError. So this is not a regression, I'd suggest to open another bug for this.
There is a race on network deletion where dhcp ports of the network may be deleted concurrently: by dhcp agent issuing port deletion on server side and by neutron server itself when deleting network. This may lead to ObjectDeletedError on one of neutron servers. Traceback:
2015-07-30 14:07:59.476 20498 TRACE neutron. plugins. ml2.plugin Traceback (most recent call last): plugins. ml2.plugin File "/usr/lib/ python2. 7/dist- packages/ neutron/ plugins/ ml2/plugin. py", line 681, in _delete_ports plugins. ml2.plugin self.delete_ port(context, port.id) plugins. ml2.plugin File "/usr/lib/ python2. 7/dist- packages/ oslo_db/ api.py" , line 131, in wrapper plugins. ml2.plugin return f(*args, **kwargs) plugins. ml2.plugin File "/usr/lib/ python2. 7/dist- packages/ neutron/ plugins/ ml2/plugin. py", line 1288, in delete_port plugins. ml2.plugin super(Ml2Plugin, self).delete_ port(context, id) plugins. ml2.plugin File "/usr/lib/ python2. 7/dist- packages/ neutron/ db/db_base_ plugin_ v2.py", line 1915, in delete_port plugins. ml2.plugin self._delete_ port(context, id) plugins. ml2.plugin File "/usr/lib/ python2. 7/dist- packages/ neutron/ db/db_base_ plugin_ v2.py", line 1938, in _delete_port plugins. ml2.plugin query.delete() plugins. ml2.plugin File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2670, in delete plugins. ml2.plugin delete_op.exec_() plugins. ml2.plugin File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ orm/persistence .py", line 896, in exec_ plugins. ml2.plugin self._do_ pre_synchronize () plugins. ml2.plugin File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ orm/persistence .py", line 958, in _do_pre_synchronize plugins. ml2.plugin eval_condition( obj)] plugins. ml2.plugin File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ orm/evaluator. py", line 115, in evaluate plugins. ml2.plugin left_val = eval_left(obj) plugins. ml2.plugin File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ orm/evaluator. py", line 72, in <lambda> plugins. ml2.plugin return lambda obj: get_correspondi ng_attr( obj) plugins. ml2.plugin File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ orm/attributes. py", line 239, in __get__ plugins. ml2.plugin return self.impl. get(instance_ state(instance) , dict_) plugins. ml2.plugin File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ orm/attributes. py", line 589, in get plugins. ml2.plugin value = callable_(state, passive) plugins. ml2.plugin File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ orm/state. py", line 424, in __call__ plugins. ml2.plugin self.manager. deferred_ scalar_ loader( self, toload) plugins. ml2.plugin File "/usr/lib/ python2. 7/dist- packages/ sqlalchemy/ orm/loading. py", line 614, in load_scalar_ attributes plugins. ml2.plugin raise orm_exc. ObjectDeletedEr ror(state) plugins. ml2.plugin ObjectDeletedError: Instance '<Port at 0x7f4ed51ceb10>' has been deleted, or its row is otherwise not present.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.
2015-07-30 14:07:59.476 20498 TRACE neutron.