neutron

  1. Bug #1866068
  2. Comment #2

Comment 2 for bug 1866068

Revision history for this message
Bence Romsics (bence-romsics) wrote :

I managed to reproduce this with a single q-svc having api_workers=2 config.

Added a time.sleep(5) to just before pg_add() here:

https://opendev.org/openstack/neutron/src/commit/77616fd1773ebca4a6261eec91b2fc5c4ece8e77/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L2170

Rebuilt my ovn devstack with NEUTRON_CREATE_INITIAL_NETWORKS=False to prevent the creation of port group 'neutron_pg_drop' during stack.sh.

This triggered the error:

openstack network create net0
openstack port create port0 --network net0 &
openstack port create port1 --network net0 &

One port create succeeded, the other failed with:

HttpException: 500: Server Error for url: http://192.168.122.169:9696/v2.0/ports, Request Failed: internal server error while processing your request.

In the logs:

márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-6c61fbbe-b81d-4d3d-9e16-2782c64e2128 admin admin] Traceback (most recent call last):
márc 05 16:22:10 devstack0 neutron-server[11115]: File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
márc 05 16:22:10 devstack0 neutron-server[11115]: txn.results.put(txn.do_commit())
márc 05 16:22:10 devstack0 neutron-server[11115]: File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
márc 05 16:22:10 devstack0 neutron-server[11115]: self.post_commit(txn)
márc 05 16:22:10 devstack0 neutron-server[11115]: File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 70, in post_commit
márc 05 16:22:10 devstack0 neutron-server[11115]: command.post_commit(txn)
márc 05 16:22:10 devstack0 neutron-server[11115]: File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 79, in post_commit
márc 05 16:22:10 devstack0 neutron-server[11115]: row = self.api.tables[self.table_name].rows[real_uuid]
márc 05 16:22:10 devstack0 neutron-server[11115]: File "/usr/lib/python3.6/collections/__init__.py", line 991, in __getitem__
márc 05 16:22:10 devstack0 neutron-server[11115]: raise KeyError(key)
márc 05 16:22:10 devstack0 neutron-server[11115]: KeyError: <ovsdbapp.backend.ovs_idl.rowview.RowView object at 0x7ff6eae66860>
márc 05 16:22:10 devstack0 neutron-server[11115]:
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers [None req-6c61fbbe-b81d-4d3d-9e16-2782c64e2128 admin admin] Mechanism driver 'ovn' failed in create_port_postcommit: KeyError: <
ovsdbapp.backend.ovs_idl.rowview.RowView object at 0x7ff6eae66860>
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers File "/opt/stack/neutron/neutron/plugins/ml2/managers.py", line 477, in _call_on_drivers
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context)
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 530, in create_port_postcommit
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers self._ovn_client.create_port(port)
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 420, in create_port
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers txn.add(self._nb_idl.qos_add(**qos_rule_column))
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers next(self.gen)
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 184, in transaction
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers yield t
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers next(self.gen)
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/api.py", line 119, in transaction
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers del self._nested_txns_map[cur_thread_id]
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/api.py", line 69, in __exit__
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers self.result = self.commit()
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers raise result.ex
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers txn.results.put(txn.do_commit())
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers self.post_commit(txn)
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 70, in post_commit
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers command.post_commit(txn)
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python3.6/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 79, in post_commit
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers row = self.api.tables[self.table_name].rows[real_uuid]
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.6/collections/__init__.py", line 991, in __getitem__
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers raise KeyError(key)
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers KeyError: <ovsdbapp.backend.ovs_idl.rowview.RowView object at 0x7ff6eae66860>
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.managers
márc 05 16:22:10 devstack0 neutron-server[11115]: ERROR neutron.plugins.ml2.plugin [None req-6c61fbbe-b81d-4d3d-9e16-2782c64e2128 admin admin] mechanism_manager.create_port_postcommit failed, deleting port 'a60f17ac-d134-4ec3-99fd-bb9249620b1f': neutron.plugins.ml2.common.exceptions.MechanismDriverError