neutron

[master][stable/2023.2][functional] test_maintenance.TestMaintenance tests fails randomly

Bug #2039417 reported by yatin
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
yatin

Bug Description

The functional test fails randomly as:-
ft1.3: neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_maintenance.TestMaintenance.test_porttesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_maintenance.py", line 306, in test_port
    neutron_net = self._create_network('network1')
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_maintenance.py", line 68, in _create_network
    return self.deserialize(self.fmt, res)['network']
KeyError: 'network'

neutron server log when it fails:-
2023-10-06 08:16:33.966 37713 DEBUG neutron.db.ovn_revision_numbers_db [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] create_initial_revision uuid=6ab7bd53-277b-4133-ac32-52b1d0c90f78, type=security_groups, rev=-1 create_initial_revision /home/zuul/src/opendev.org/openstack/neutron/neutron/db/ovn_revision_numbers_db.py:108
2023-10-06 08:16:33.973 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:33.974 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
    txn.results.put(txn.do_commit())
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
    raise RuntimeError(msg)
RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it

2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Error during notification for neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver._create_security_group-11049860 security_group, after_create: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, payload=payload)
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 409, in _create_security_group
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager self._ovn_client.create_security_group(context,
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2328, in create_security_group
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager with self._nb_idl.transaction(check_error=True) as txn:
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager next(self.gen)
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager next(self.gen)
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager with self.create_transaction(
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager self.result = self.commit()
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager raise result.ex
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager txn.results.put(txn.do_commit())
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager raise RuntimeError(msg)
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:33.975 37713 ERROR neutron_lib.callbacks.manager
...
2023-10-06 08:16:34.047 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:34.048 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
    txn.results.put(txn.do_commit())
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
    raise RuntimeError(msg)
RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it

2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Mechanism driver 'ovn' failed in create_network_postcommit: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 497, in _call_on_drivers
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context)
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 579, in create_network_postcommit
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers self._ovn_client.create_network(context.plugin_context, network)
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1927, in create_network
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers with self._nb_idl.transaction(check_error=True) as txn:
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers next(self.gen)
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers next(self.gen)
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers with self.create_transaction(
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers self.result = self.commit()
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers raise result.ex
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers txn.results.put(txn.do_commit())
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers raise RuntimeError(msg)
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:34.048 37713 ERROR neutron.plugins.ml2.managers
2023-10-06 08:16:34.052 37713 ERROR neutron.plugins.ml2.plugin [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] mechanism_manager.create_network_postcommit failed, deleting network '767ee503-3645-4762-b0d6-e673b3c14170': neutron.plugins.ml2.common.exceptions.MechanismDriverError
...
2023-10-06 08:16:34.153 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:34.154 37713 ERROR ovsdbapp.backend.ovs_idl.transaction [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
    txn.results.put(txn.do_commit())
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
    raise RuntimeError(msg)
RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
: neutron.plugins.ml2.common.exceptions.MechanismDriverError
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] Mechanism driver 'ovn' failed in delete_network_postcommit: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1219, in _after_create_network
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers self.mechanism_manager.create_network_postcommit(mech_context)
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 547, in create_network_postcommit
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers self._call_on_drivers("create_network_postcommit", context)
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 513, in _call_on_drivers
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers raise ml2_exc.MechanismDriverError(
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers neutron.plugins.ml2.common.exceptions.MechanismDriverError
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers During handling of the above exception, another exception occurred:
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 497, in _call_on_drivers
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context)
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 631, in delete_network_postcommit
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers self._ovn_client.delete_network(
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1938, in delete_network
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers with self._nb_idl.transaction(check_error=True) as txn:
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers next(self.gen)
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers with super(OvsdbNbOvnIdl, self).transaction(*args, **kwargs) as t:
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers next(self.gen)
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 114, in transaction
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers with self.create_transaction(
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/api.py", line 71, in __exit__
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers self.result = self.commit()
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers raise result.ex
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 118, in run
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers txn.results.put(txn.do_commit())
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers raise RuntimeError(msg)
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it
2023-10-06 08:16:34.154 37713 ERROR neutron.plugins.ml2.managers
2023-10-06 08:16:34.162 37713 ERROR neutron.plugins.ml2.plugin [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] mechanism_manager.delete_network_postcommit failed: neutron.plugins.ml2.common.exceptions.MechanismDriverError
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] POST failed.: neutron.plugins.ml2.common.exceptions.MechanismDriverError
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 693, in __call__
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/pecan/core.py", line 584, in invoke_controller
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception():
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception() as ectxt:
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception():
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 65, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 163, in post
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return self.create(resources)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 181, in create
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return {key: creator(*creator_args, **creator_kwargs)}
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 731, in inner
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 223, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f_with_retry(*args, **kwargs,
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception():
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception() as ectxt:
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception():
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1211, in create_network
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation return self._after_create_network(context, result, mech_context)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1221, in _after_create_network
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception():
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise self.value
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1219, in _after_create_network
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self.mechanism_manager.create_network_postcommit(mech_context)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 547, in create_network_postcommit
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation self._call_on_drivers("create_network_postcommit", context)
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 513, in _call_on_drivers
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation raise ml2_exc.MechanismDriverError(
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation neutron.plugins.ml2.common.exceptions.MechanismDriverError
2023-10-06 08:16:34.163 37713 ERROR neutron.pecan_wsgi.hooks.translation
2023-10-06 08:16:34.166 37713 DEBUG neutron.pecan_wsgi.hooks.notifier [None req-34e699fe-0fdc-4373-8b91-5b7dd1ac60cb - 46f70361-ba71-4bd0-9769-3573fd227c4b - - - -] No notification will be sent due to unsuccessful status code: 500 after /home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/hooks/notifier.py:78

^ From https://9ce43a75e3387ceb8909-2b4f2fa211fea8445ec0f4a568f6056b.ssl.cf2.rackcdn.com/897438/1/gate/neutron-functional-with-uwsgi/7147770/controller/logs/dsvm-functional-logs.tar.gz

Other example failures:-
- https://5a1da1856f577d094ffa-09419825a71508c658bc83d968b5572f.ssl.cf2.rackcdn.com/890162/3/check/neutron-functional-with-uwsgi/4ad32c0/testr_results.html
- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d3b/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/d3b6e71/testr_results.html
- https://cf7800dce1b91328e4f8-616b34f6dd94893c803bfdcdd39959b5.ssl.cf5.rackcdn.com/897332/4/gate/neutron-functional-with-uwsgi/fe7d494/testr_results.html
- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d3b/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/d3b6e71/testr_results.html
- https://cf7800dce1b91328e4f8-616b34f6dd94893c803bfdcdd39959b5.ssl.cf5.rackcdn.com/897332/4/gate/neutron-functional-with-uwsgi/fe7d494/testr_results.html

Brian Haley (brian-haley)
tags: added: functional-tests
Changed in neutron:
importance: Undecided → High
yatin (yatinkarel)
Changed in neutron:
assignee: nobody → yatin (yatinkarel)
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/899244

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/899244
Committed: https://opendev.org/openstack/neutron/commit/39a1f1e7fd60b9b55dcb4b5ea5027e7759fc3b3b
Submitter: "Zuul (22348)"
Branch: master

commit 39a1f1e7fd60b9b55dcb4b5ea5027e7759fc3b3b
Author: yatinkarel <email address hidden>
Date: Wed Oct 25 15:10:25 2023 +0530

    [FT] Clear the idl lock in TestMaintenance tests

    One or the other test in TestMaintenance fails
    randomly with missing idl lock.

    DBInconsistenciesPeriodics set's the idl lock
    but in TestMaintenance we don't run it as a seperate
    maintenance worker and instead call check_for_inconsistencies
    when needed so not a requirement to have a dedicated lock.
    This patch clears the lock for these tests to avoid
    such random issues.

    Closes-Bug: #2039417
    Change-Id: I931997fe010295120f2400b1ec310dff82497ab6

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/899323

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/899324

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/899323
Committed: https://opendev.org/openstack/neutron/commit/54122203f4b80746aa9fb05bf1b0f66acd1927de
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 54122203f4b80746aa9fb05bf1b0f66acd1927de
Author: yatinkarel <email address hidden>
Date: Wed Oct 25 15:10:25 2023 +0530

    [FT] Clear the idl lock in TestMaintenance tests

    One or the other test in TestMaintenance fails
    randomly with missing idl lock.

    DBInconsistenciesPeriodics set's the idl lock
    but in TestMaintenance we don't run it as a seperate
    maintenance worker and instead call check_for_inconsistencies
    when needed so not a requirement to have a dedicated lock.
    This patch clears the lock for these tests to avoid
    such random issues.

    Closes-Bug: #2039417
    Change-Id: I931997fe010295120f2400b1ec310dff82497ab6
    (cherry picked from commit 39a1f1e7fd60b9b55dcb4b5ea5027e7759fc3b3b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/899324
Committed: https://opendev.org/openstack/neutron/commit/1bc5068c7eec2bd44d525070c8840aa31f444c5c
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 1bc5068c7eec2bd44d525070c8840aa31f444c5c
Author: yatinkarel <email address hidden>
Date: Wed Oct 25 15:10:25 2023 +0530

    [FT] Clear the idl lock in TestMaintenance tests

    One or the other test in TestMaintenance fails
    randomly with missing idl lock.

    DBInconsistenciesPeriodics set's the idl lock
    but in TestMaintenance we don't run it as a seperate
    maintenance worker and instead call check_for_inconsistencies
    when needed so not a requirement to have a dedicated lock.
    This patch clears the lock for these tests to avoid
    such random issues.

    Closes-Bug: #2039417
    Change-Id: I931997fe010295120f2400b1ec310dff82497ab6
    (cherry picked from commit 39a1f1e7fd60b9b55dcb4b5ea5027e7759fc3b3b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 22.1.0

This issue was fixed in the openstack/neutron 22.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 23.1.0

This issue was fixed in the openstack/neutron 23.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 24.0.0.0b1

This issue was fixed in the openstack/neutron 24.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.