Comment 14 for bug 1392386

Revision history for this message
Oleg Bondarev (obondarev) wrote :

Traceback from comment 11 is not the cause for network delete failure, it rather shows that db retry works. Logs from snapshot show that port failed to be deleted first time (DBDeadlock), but then was deleted on a second attempt (thanks to db retry).

Network deletion failure was caused by ObjectDeletedError. So this is not a regression, I'd suggest to open another bug for this.

There is a race on network deletion where dhcp ports of the network may be deleted concurrently: by dhcp agent issuing port deletion on server side and by neutron server itself when deleting network. This may lead to ObjectDeletedError on one of neutron servers. Traceback:

2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin Traceback (most recent call last):
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 681, in _delete_ports
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin self.delete_port(context, port.id)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 131, in wrapper
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin return f(*args, **kwargs)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 1288, in delete_port
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin super(Ml2Plugin, self).delete_port(context, id)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 1915, in delete_port
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin self._delete_port(context, id)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 1938, in _delete_port
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin query.delete()
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2670, in delete
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin delete_op.exec_()
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 896, in exec_
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin self._do_pre_synchronize()
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 958, in _do_pre_synchronize
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin eval_condition(obj)]
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/evaluator.py", line 115, in evaluate
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin left_val = eval_left(obj)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/evaluator.py", line 72, in <lambda>
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin return lambda obj: get_corresponding_attr(obj)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 239, in __get__
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin return self.impl.get(instance_state(instance), dict_)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 589, in get
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin value = callable_(state, passive)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/state.py", line 424, in __call__
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin self.manager.deferred_scalar_loader(self, toload)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/loading.py", line 614, in load_scalar_attributes
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin raise orm_exc.ObjectDeletedError(state)
2015-07-30 14:07:59.476 20498 TRACE neutron.plugins.ml2.plugin ObjectDeletedError: Instance '<Port at 0x7f4ed51ceb10>' has been deleted, or its row is otherwise not present.