Create network failed during functional test

[Slawek Kaplonski]

One of the functional, ovn related tests failed with error like:

ft1.3: neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovsdb_monitor.TestNBDbMonitorOverTcp.test_distributed_locktesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovsdb_monitor.py", line 231, in test_distributed_lock
    self.create_port()
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovsdb_monitor.py", line 77, in create_port
    net = self._make_network(self.fmt, 'net1', True)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/db/test_db_base_plugin_v2.py", line 510, in _make_network
    raise webob.exc.HTTPClientError(code=res.status_int)
webob.exc.HTTPClientError: The server could not comply with the request since it is either malformed or otherwise incorrect.

In the testrun log there is error like:
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/ovn_revision_numbers_db.py", line 93, in _get_standard_attr_id
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers row = context.session.query(STD_ATTR_MAP[resource_type]).filter_by(
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3500, in one
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers raise orm_exc.NoResultFound("No row was found for one()")
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers sqlalchemy.orm.exc.NoResultFound: No row was found for one()
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers During handling of the above exception, another exception occurred:
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 477, in _call_on_drivers
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context)
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py", line 471, in create_network_precommit
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers ovn_revision_numbers_db.create_initial_revision(
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 233, in wrapped
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers return method(*args, **kwargs)
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/ovn_revision_numbers_db.py", line 108, in create_initial_revision
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers std_attr_id = _get_standard_attr_id(
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers File "/home/zuul/src/opendev.org/openstack/neutron/neutron/db/ovn_revision_numbers_db.py", line 97, in _get_standard_attr_id
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers raise StandardAttributeIDNotFound(resource_uuid=resource_uuid)
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers neutron.db.ovn_revision_numbers_db.StandardAttributeIDNotFound: Standard attribute ID not found for 3f787ea1-0ce5-4310-9b42-b530e7dcb4a1
2020-11-04 13:34:50.442 61672 ERROR neutron.plugins.ml2.managers
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation [req-5a27b67c-a80f-4877-84cb-7b3d511d51d6 - tenid - - -] POST failed.: neutron.plugins.ml2.common.exceptions.MechanismDriverError
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/pecan/core.py", line 683, in __call__
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/pecan/core.py", line 574, in invoke_controller
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/six.py", line 703, in reraise
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation raise value
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/api.py", line 154, in wrapper
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/six.py", line 703, in reraise
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation raise value
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/api.py", line 142, in wrapper
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/six.py", line 703, in reraise
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation raise value
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/utils.py", line 76, in wrapped
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 163, in post
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation return self.create(resources)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/pecan_wsgi/controllers/resource.py", line 181, in create
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation return {key: creator(*creator_args, **creator_kwargs)}
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 687, in inner
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation return f(self, context, *args, **kwargs)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 233, in wrapped
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/six.py", line 703, in reraise
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation raise value
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/api.py", line 154, in wrapper
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/six.py", line 703, in reraise
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation raise value
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/api.py", line 142, in wrapper
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/six.py", line 703, in reraise
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation raise value
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1068, in create_network
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation result, mech_context = self._create_network_db(context, network)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/plugin.py", line 1061, in _create_network_db
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation self.mechanism_manager.create_network_precommit(mech_context)
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 512, in create_network_precommit
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation self._call_on_drivers("create_network_precommit", context,
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/managers.py", line 493, in _call_on_drivers
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation raise ml2_exc.MechanismDriverError(
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation neutron.plugins.ml2.common.exceptions.MechanismDriverError
2020-11-04 13:34:50.448 61672 ERROR neutron.pecan_wsgi.hooks.translation

Link to error logs: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_236/759673/5/check/neutron-functional-with-uwsgi/236ded2/testr_results.html

[Slawek Kaplonski (slaweq)]

Change to "High" as it affects our CI

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

The standard attribute related to the object, in this case a network, should be always present in the precommit method. This is called inside the network DB object creation transaction [1]. That guarantees the object, that is still not consolidated in the DB backend, is created in the DB transaction [2].

I would wait for [3]. If this problem is still present once we finish the DB facade migration, then I would investigate this issue further.

Regards.

[1]https://github.com/openstack/neutron/blob/033a445fbf3529ee5a5ed3488d7461fa80336f0d/neutron/plugins/ml2/plugin.py
[2]http://paste.openstack.org/show/799951/
[3]https://review.opendev.org/#/c/715315/

[Slawek Kaplonski (slaweq)]

I saw it again recently https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2c5/762582/4/check/neutron-functional-with-uwsgi/2c52a3b/testr_results.html

[Eduardo Olivares (eolivare)]

It failed again, same job (neutron-functional-with-uwsgi), but this time with TestNBDbMonitorOverSsl:
https://zuul.opendev.org/t/openstack/build/7022a256c81f429282cc8276b9dd21d9/log/controller/logs/dsvm-functional-logs/neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovsdb_monitor.TestNBDbMonitorOverSsl.test_distributed_lock/testrun.txt

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Patch: https://review.opendev.org/c/openstack/neutron/+/765874

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/788543

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/788543
Committed: https://opendev.org/openstack/neutron/commit/7cf9597570f288d27768dc5ff7be04824d09f8bc
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 7cf9597570f288d27768dc5ff7be04824d09f8bc
Author: Terry Wilson <email address hidden>
Date: Mon Dec 7 20:46:53 2020 +0000

    Rely on worker count for HashRing caching

    The current code looks at a hash ring node's created_at/updated_at
    fields and tries to determine whether the node has been updated
    based on whether updated_at - created_at > 1 second (due to the
    method that initially fills them being different by microseconds).
    Unfortunately, due to the notify() method being called which calls
    the hash ring node's touch_node(), a node can be updated in under
    a second, meaning we will prevent caching for much longer than
    we intend.

    When using sqlite in-memory db, this continually re-creating the
    Hash Ring objects for every event that is processed is exposing an
    issue where rows that should be in the db just *aren't*.

    This patch instead limits the hash ring nodes to api workers and
    prevents caching only until the number of nodes == number of api
    workers on the host. The switch from spawning hash ring nodes
    where !is_maintenance to is_api_worker is primarily because it
    seems to be difficult to get a list of *all* workers from which to
    subtract the maintenance worker so that _wait_startup_before_caching
    can wait for that specific number of workers. In practice, this
    means that RpcWorker and ServiceWorker workers would not process
    HashRing events.

    A note on bug 1903008: While this change will greatly reduce the
    likelihood of this issue taking place, we still have some work to
    do in order to fully understand why it rubs the database backend
    in the wrong way. Thus, we will make this change 'related to'
    instead of closing the bug.

    Conflicts:
            neutron/tests/unit/common/ovn/test_hash_ring_manager.py

    Related-Bug: #1894117
    Related-Bug: #1903008
    Change-Id: Ia198d45f49bddda549a0e70a3374b8339f88887b
    (cherry picked from commit c4007b0833111a25d24f597161d39ee9ccd37189)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/786097
Committed: https://opendev.org/openstack/neutron/commit/1f30f2dfff722ea65c811b4b99243fae51a2d688
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 1f30f2dfff722ea65c811b4b99243fae51a2d688
Author: Terry Wilson <email address hidden>
Date: Mon Dec 7 20:46:53 2020 +0000

    Rely on worker count for HashRing caching

    The current code looks at a hash ring node's created_at/updated_at
    fields and tries to determine whether the node has been updated
    based on whether updated_at - created_at > 1 second (due to the
    method that initially fills them being different by microseconds).
    Unfortunately, due to the notify() method being called which calls
    the hash ring node's touch_node(), a node can be updated in under
    a second, meaning we will prevent caching for much longer than
    we intend.

    When using sqlite in-memory db, this continually re-creating the
    Hash Ring objects for every event that is processed is exposing an
    issue where rows that should be in the db just *aren't*.

    This patch instead limits the hash ring nodes to api workers and
    prevents caching only until the number of nodes == number of api
    workers on the host. The switch from spawning hash ring nodes
    where !is_maintenance to is_api_worker is primarily because it
    seems to be difficult to get a list of *all* workers from which to
    subtract the maintenance worker so that _wait_startup_before_caching
    can wait for that specific number of workers. In practice, this
    means that RpcWorker and ServiceWorker workers would not process
    HashRing events.

    A note on bug 1903008: While this change will greatly reduce the
    likelihood of this issue taking place, we still have some work to
    do in order to fully understand why it rubs the database backend
    in the wrong way. Thus, we will make this change 'related to'
    instead of closing the bug.

    Related-Bug: #1894117
    Related-Bug: #1903008
    Change-Id: Ia198d45f49bddda549a0e70a3374b8339f88887b
    (cherry picked from commit c4007b0833111a25d24f597161d39ee9ccd37189)