[ovn-octavia-provider] functional job fails with "oslo_db.exception.DBReferenceError: (sqlite3.IntegrityError) FOREIGN KEY constraint failed when inserting port bindings

[Ihar Hrachyshka]

https://zuul.opendev.org/t/openstack/build/e28d2e67524b49d58398a9afa6be0ca9

==============================
Failed 1 tests - output below:
==============================

ovn_octavia_provider.tests.functional.test_agent.TestOvnOctaviaProviderAgent.test_fip_on_lb_vip
-----------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1971, in _exec_single_context
    self.dialect.do_execute(

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 919, in do_execute
    cursor.execute(statement, parameters)

    sqlite3.IntegrityError: FOREIGN KEY constraint failed

The above exception was the direct cause of the following exception:

    Traceback (most recent call last):

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/ovn_octavia_provider/tests/functional/test_agent.py", line 220, in test_fip_on_lb_vip
    self.l3_plugin.update_router(

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/db/extraroute_db.py", line 71, in update_router
    return super(ExtraRoute_dbonly_mixin, self).update_router(

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/db/l3_db.py", line 2237, in update_router
    router_dict = super(L3_NAT_db_mixin, self).update_router(context,

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/db/api.py", line 223, in wrapped
    return f_with_retry(*args, **kwargs,

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
    with excutils.save_and_reraise_exception():

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
    return f(*args, **kwargs)

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
    with excutils.save_and_reraise_exception() as ectxt:

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
    return f(*args, **kwargs)

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
    with excutils.save_and_reraise_exception():

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
    return f(*dup_args, **dup_kwargs)

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/db/l3_db.py", line 348, in update_router
    self._update_router_gw_info(context, id, gw_info, r)

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/db/l3_gateway_ip_qos.py", line 74, in _update_router_gw_info
    self)._update_router_gw_info(

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/db/l3_extra_gws_db.py", line 585, in _update_router_gw_info
    router_db = super()._update_router_gw_info(context, router_id, info,

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/db/l3_gwmode_db.py", line 74, in _update_router_gw_info
    super(L3_NAT_dbonly_mixin, self)._update_router_gw_info(

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/db/l3_db.py", line 571, in _update_router_gw_info
    self._create_gw_port(context, router_id, router, network_id,

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/db/l3_db.py", line 532, in _create_gw_port
    self._create_router_gw_port(context, router,

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/db/l3_db.py", line 378, in _create_router_gw_port
    gw_port = plugin_utils.create_port(

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/plugins/utils.py", line 338, in create_port
    return core_plugin.create_port(

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/common/utils.py", line 726, in inner
    return f(*args, **kwargs)

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/db/api.py", line 223, in wrapped
    return f_with_retry(*args, **kwargs,

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
    with excutils.save_and_reraise_exception():

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
    return f(*args, **kwargs)

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
    with excutils.save_and_reraise_exception() as ectxt:

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
    return f(*args, **kwargs)

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
    with excutils.save_and_reraise_exception():

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
    return f(*dup_args, **dup_kwargs)

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/plugins/ml2/plugin.py", line 1598, in create_port
    result, mech_context = self._create_port_db(context, port)

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/plugins/ml2/plugin.py", line 1572, in _create_port_db
    binding = db.add_port_binding(context, result['id'])

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 997, in wrapper
    with self._transaction_scope(context):

      File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1042, in _transaction_scope
    with current._produce_block(

      File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 664, in _session
    self.session.flush()

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4296, in flush
    self._flush(objects)

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4431, in _flush
    with util.safe_reraise():

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4392, in _flush
    flush_context.execute()

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
    rec.execute(self)

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
    util.preloaded.orm_persistence.save_obj(

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 93, in save_obj
    _emit_insert_statements(

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 1233, in _emit_insert_statements
    result = connection.execute(

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1422, in execute
    return meth(

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 514, in _execute_on_connection
    return connection._execute_clauseelement(

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1644, in _execute_clauseelement
    ret = self._execute_context(

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1850, in _execute_context
    return self._exec_single_context(

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1990, in _exec_single_context
    self._handle_dbapi_exception(

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2354, in _handle_dbapi_exception
    raise newraise.with_traceback(exc_info[2]) from e

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1971, in _exec_single_context
    self.dialect.do_execute(

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 919, in do_execute
    cursor.execute(statement, parameters)

    oslo_db.exception.DBReferenceError: (sqlite3.IntegrityError) FOREIGN KEY constraint failed
[SQL: INSERT INTO ml2_port_bindings (port_id, host, vnic_type, profile, vif_type, vif_details, status) VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: ('84966876-5f19-45c5-8979-234412325a2e', '', 'normal', '', 'unbound', '', 'ACTIVE')]
(Background on this error at: https://sqlalche.me/e/20/gkpj)

[Bence Romsics (bence-romsics)]

build history:
https://zuul.opendev.org/t/openstack/builds?job_name=ovn-octavia-provider-functional-release&skip=0

opensearch results:
https://opensearch.logs.openstack.org/_dashboards/app/data-explorer/discover/#?_a=(discover:(columns:!(message,log_url,build_name,project,filename),interval:auto,sort:!()),metadata:(indexPattern:'94869730-aea8-11ec-9e6a-83741af3fdcd',view:discover))&_q=(filters:!(),query:(language:lucene,query:'message:%22oslo_db.exception.DBReferenceError:%20(sqlite3.IntegrityError)%20FOREIGN%20KEY%20constraint%20failed%22%20AND%20project:%22openstack%2Fovn-octavia-provider%22'))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-10d,to:now))