neutron

[ovn-octavia-provider] functional test intermittently fail with DB error: Cursor needed to be reset because of commit/rollback and can no longer be fetched from

Bug #2020195 reported by Fernando Royo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Confirmed
High
Fernando Royo

Bug Description

Trace:

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 977, in fetchall
    rows = dbapi_cursor.fetchall()

    sqlite3.InterfaceError: Cursor needed to be reset because of commit/rollback and can no longer be fetched from.

The above exception was the direct cause of the following exception:

    Traceback (most recent call last):

      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/ovn_octavia_provider/tests/functional/test_driver.py", line 43, in test_create_lb_custom_network_retry
    self._create_load_balancer_custom_lr_ls_and_validate(
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/ovn_octavia_provider/tests/functional/base.py", line 360, in _create_load_balancer_custom_lr_ls_and_validate
    net_info.append(self._create_net("n2", "10.0.2.0/24", r_id))
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/ovn_octavia_provider/tests/functional/base.py", line 282, in _create_net
    self.l3_plugin.add_router_interface(
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/services/ovn_l3/plugin.py", line 250, in add_router_interface
    with excutils.save_and_reraise_exception():
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/services/ovn_l3/plugin.py", line 247, in add_router_interface
    self._ovn_client.create_router_port(
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1691, in create_router_port
    router = self._l3_plugin.get_router(context, router_id)
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/db/api.py", line 223, in wrapped
    return f_with_retry(*args, **kwargs,
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/db/api.py", line 137, in wrapped
    with excutils.save_and_reraise_exception():
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/db/api.py", line 135, in wrapped
    return f(*args, **kwargs)
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_db/api.py", line 144, in wrapper
    with excutils.save_and_reraise_exception() as ectxt:
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_db/api.py", line 142, in wrapper
    return f(*args, **kwargs)
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/db/api.py", line 183, in wrapped
    with excutils.save_and_reraise_exception():
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/db/api.py", line 181, in wrapped
    return f(*dup_args, **dup_kwargs)
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/db/l3_db.py", line 646, in get_router
    router = self._get_router(context, id)
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1022, in wrapper
    return fn(*args, **kwargs)
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron/db/l3_db.py", line 205, in _get_router
    router = model_query.get_by_id(
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/neutron_lib/db/model_query.py", line 169, in get_by_id
    return query.filter(model.id == object_id).one()
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/orm/query.py", line 2869, in one
    return self._iter().one()
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1476, in one
    return self._only_one_row(
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 559, in _only_one_row
    row = onerow(hard_close=True)
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1340, in _fetchone_impl
    return self._real_result._fetchone_impl(hard_close=hard_close)
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 1743, in _fetchone_impl
    row = next(self.iterator, _NO_ROW)
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/orm/loading.py", line 147, in chunks
    fetch = cursor._raw_all_rows()
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/result.py", line 392, in _raw_all_rows
    rows = self._fetchall_impl()
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 1805, in _fetchall_impl
    return self.cursor_strategy.fetchall(self, self.cursor)
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 981, in fetchall
    self.handle_exception(result, dbapi_cursor, e)
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 941, in handle_exception
    result.connection._handle_dbapi_exception(
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception
    util.raise_(newraise, with_traceback=exc_info[2], from_=e)
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 208, in raise_
    raise exception
      File "/home/zuul/src/opendev.org/openstack/ovn-octavia-provider/.tox/dsvm-functional/lib/python3.10/site-packages/sqlalchemy/engine/cursor.py", line 977, in fetchall
    rows = dbapi_cursor.fetchall()

    oslo_db.exception.DBError: (sqlite3.InterfaceError) Cursor needed to be reset because of commit/rollback and can no longer be fetched from.
(Background on this error at: https://sqlalche.me/e/14/rvf5)

It happends randomnly on functional tests, not related to any specific one.

Fernando Royo (froyoredhat)
Changed in neutron:
assignee: nobody → Fernando Royo (froyoredhat)
summary: - functional test intermittently fail with DB error: Cursor needed to be
- reset because of commit/rollback and can no longer be fetched from
+ [ovn-octavia-provider] functional test intermittently fail with DB
+ error: Cursor needed to be reset because of commit/rollback and can no
+ longer be fetched from
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to ovn-octavia-provider (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/883662

Revision history for this message
Fernando Royo (froyoredhat) wrote :

Some stable branches are also affected.
It has been evaluated using git bisect on neutron stable/xena (reduced branch comparing to master) to try to locate any patch that reproduces the bug more frequently. No relevant commits were found.

Also mixing versions of neutron, neutron-lib, oslo-db or slqalchemy without any conclusion.

As other projects also have this bug in their tests, it seems that it is a bug that has always been there, but now it is being reproduced more frequently because of some change in other libraries. Can be checked here [1]

Keep this LP bug for track and visibility.

[1] https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:build_status,negate:!f,params:(query:FAILURE),type:phrase),query:(match_phrase:(build_status:FAILURE))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:project,negate:!f,params:(query:openstack%2Fironic),type:phrase),query:(match_phrase:(project:openstack%2Fironic)))),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22Cursor%20needed%20to%20be%20reset%20because%20of%20commit%2F%22'),sort:!())

Elvira García Ruiz (elviragr)
tags: added: functional-tests
Changed in neutron:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ovn-octavia-provider (master)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/883662
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/028596705678b447797121049d542e06e1ad4355
Submitter: "Zuul (22348)"
Branch: master

commit 028596705678b447797121049d542e06e1ad4355
Author: Fernando Royo <email address hidden>
Date: Fri May 19 13:15:02 2023 +0200

    Add retry on case of sqlite3.InterfaceError on FT

    After a thorough review of the issue, it looks like that the problem
    does not originate from the base code of ovn-octavia-provider or
    neutron. Other projects are also experiencing this problem,
    indicating that it likely stems from a different source or set of
    libraries [1].

    To minimize the need for extensive rechecks on future patches, this
    patch introduces a retry mechanism, utilizing tenacity, to the
    affected methods.

    Once the root cause of the problem '(sqlite3.InterfaceError) Cursor
    needed to be reset because of commit/rollback and can no longer be
    fetched from,' is identified and resolved, this patch should be
    reverted.

    [1] https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:build_status,negate:!f,params:(query:FAILURE),type:phrase),query:(match_phrase:(build_status:FAILURE)))),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22Cursor%20needed%20to%20be%20reset%20because%20of%20commit%2F%22'),sort:!())

    Related-Bug: #2020195
    Change-Id: Ia7a9b5230f9cf56de8278b736022240a780130d6

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to ovn-octavia-provider (stable/2023.1)

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/883801

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to ovn-octavia-provider (stable/zed)

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/883802

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to ovn-octavia-provider (stable/yoga)

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/883803

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to ovn-octavia-provider (stable/xena)

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/883804

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to ovn-octavia-provider (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/883805

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ovn-octavia-provider (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/883805
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/124e13ce55da670897ffd1973d563d7dd194d636
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 124e13ce55da670897ffd1973d563d7dd194d636
Author: Fernando Royo <email address hidden>
Date: Fri May 19 13:15:02 2023 +0200

    Add retry on case of sqlite3.InterfaceError on FT

    After a thorough review of the issue, it looks like that the problem
    does not originate from the base code of ovn-octavia-provider or
    neutron. Other projects are also experiencing this problem,
    indicating that it likely stems from a different source or set of
    libraries [1].

    To minimize the need for extensive rechecks on future patches, this
    patch introduces a retry mechanism, utilizing tenacity, to the
    affected methods.

    Once the root cause of the problem '(sqlite3.InterfaceError) Cursor
    needed to be reset because of commit/rollback and can no longer be
    fetched from,' is identified and resolved, this patch should be
    reverted.

    [1] https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:build_status,negate:!f,params:(query:FAILURE),type:phrase),query:(match_phrase:(build_status:FAILURE)))),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22Cursor%20needed%20to%20be%20reset%20because%20of%20commit%2F%22'),sort:!())

    Related-Bug: #2020195
    Change-Id: Ia7a9b5230f9cf56de8278b736022240a780130d6
    (cherry picked from commit 028596705678b447797121049d542e06e1ad4355)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ovn-octavia-provider (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/883803
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/cb7e4417d2525992fbed1e45e29957899b8c0754
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit cb7e4417d2525992fbed1e45e29957899b8c0754
Author: Fernando Royo <email address hidden>
Date: Fri May 19 13:15:02 2023 +0200

    Add retry on case of sqlite3.InterfaceError on FT

    After a thorough review of the issue, it looks like that the problem
    does not originate from the base code of ovn-octavia-provider or
    neutron. Other projects are also experiencing this problem,
    indicating that it likely stems from a different source or set of
    libraries [1].

    To minimize the need for extensive rechecks on future patches, this
    patch introduces a retry mechanism, utilizing tenacity, to the
    affected methods.

    Once the root cause of the problem '(sqlite3.InterfaceError) Cursor
    needed to be reset because of commit/rollback and can no longer be
    fetched from,' is identified and resolved, this patch should be
    reverted.

    [1] https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:build_status,negate:!f,params:(query:FAILURE),type:phrase),query:(match_phrase:(build_status:FAILURE)))),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22Cursor%20needed%20to%20be%20reset%20because%20of%20commit%2F%22'),sort:!())

    Related-Bug: #2020195
    Change-Id: Ia7a9b5230f9cf56de8278b736022240a780130d6
    (cherry picked from commit 028596705678b447797121049d542e06e1ad4355)

tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ovn-octavia-provider (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/883804
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/128c92897667c7d6249d860a5fc127a0838bac26
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 128c92897667c7d6249d860a5fc127a0838bac26
Author: Fernando Royo <email address hidden>
Date: Fri May 19 13:15:02 2023 +0200

    Add retry on case of sqlite3.InterfaceError on FT

    After a thorough review of the issue, it looks like that the problem
    does not originate from the base code of ovn-octavia-provider or
    neutron. Other projects are also experiencing this problem,
    indicating that it likely stems from a different source or set of
    libraries [1].

    To minimize the need for extensive rechecks on future patches, this
    patch introduces a retry mechanism, utilizing tenacity, to the
    affected methods.

    Once the root cause of the problem '(sqlite3.InterfaceError) Cursor
    needed to be reset because of commit/rollback and can no longer be
    fetched from,' is identified and resolved, this patch should be
    reverted.

    [1] https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:build_status,negate:!f,params:(query:FAILURE),type:phrase),query:(match_phrase:(build_status:FAILURE)))),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22Cursor%20needed%20to%20be%20reset%20because%20of%20commit%2F%22'),sort:!())

    Related-Bug: #2020195
    Change-Id: Ia7a9b5230f9cf56de8278b736022240a780130d6
    (cherry picked from commit 028596705678b447797121049d542e06e1ad4355)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ovn-octavia-provider (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/883801
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/e5c91467ab55c48a86bb3888f4fcca8b5dc15229
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit e5c91467ab55c48a86bb3888f4fcca8b5dc15229
Author: Fernando Royo <email address hidden>
Date: Fri May 19 13:15:02 2023 +0200

    Add retry on case of sqlite3.InterfaceError on FT

    After a thorough review of the issue, it looks like that the problem
    does not originate from the base code of ovn-octavia-provider or
    neutron. Other projects are also experiencing this problem,
    indicating that it likely stems from a different source or set of
    libraries [1].

    To minimize the need for extensive rechecks on future patches, this
    patch introduces a retry mechanism, utilizing tenacity, to the
    affected methods.

    Once the root cause of the problem '(sqlite3.InterfaceError) Cursor
    needed to be reset because of commit/rollback and can no longer be
    fetched from,' is identified and resolved, this patch should be
    reverted.

    [1] https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:build_status,negate:!f,params:(query:FAILURE),type:phrase),query:(match_phrase:(build_status:FAILURE)))),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22Cursor%20needed%20to%20be%20reset%20because%20of%20commit%2F%22'),sort:!())

    Related-Bug: #2020195
    Change-Id: Ia7a9b5230f9cf56de8278b736022240a780130d6
    (cherry picked from commit 028596705678b447797121049d542e06e1ad4355)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ovn-octavia-provider (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/883802
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/23ea3bb6ce63460178002a8e7f7c4ab25205e8c2
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 23ea3bb6ce63460178002a8e7f7c4ab25205e8c2
Author: Fernando Royo <email address hidden>
Date: Fri May 19 13:15:02 2023 +0200

    Add retry on case of sqlite3.InterfaceError on FT

    After a thorough review of the issue, it looks like that the problem
    does not originate from the base code of ovn-octavia-provider or
    neutron. Other projects are also experiencing this problem,
    indicating that it likely stems from a different source or set of
    libraries [1].

    To minimize the need for extensive rechecks on future patches, this
    patch introduces a retry mechanism, utilizing tenacity, to the
    affected methods.

    Once the root cause of the problem '(sqlite3.InterfaceError) Cursor
    needed to be reset because of commit/rollback and can no longer be
    fetched from,' is identified and resolved, this patch should be
    reverted.

    [1] https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'94869730-aea8-11ec-9e6a-83741af3fdcd',key:build_status,negate:!f,params:(query:FAILURE),type:phrase),query:(match_phrase:(build_status:FAILURE)))),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22Cursor%20needed%20to%20be%20reset%20because%20of%20commit%2F%22'),sort:!())

    Related-Bug: #2020195
    Change-Id: Ia7a9b5230f9cf56de8278b736022240a780130d6
    (cherry picked from commit 028596705678b447797121049d542e06e1ad4355)

tags: added: in-stable-zed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to ovn-octavia-provider (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/912104

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ovn-octavia-provider (master)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/912104
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/1e45693bca42e4950a23c24ffaa3f14752aa4440
Submitter: "Zuul (22348)"
Branch: master

commit 1e45693bca42e4950a23c24ffaa3f14752aa4440
Author: Fernando Royo <email address hidden>
Date: Fri Mar 8 11:32:00 2024 +0100

    Retry on case of sqlite3.InterfaceError on FT

    Following the steps done in [1] additional retries have added after
    decompose in a soft refactor main methods that call to Neutron.

    To clarify the refactor, the following changes have been made in FT
    base methods:
    - The method _create_subnet_from_net was previously responsible for
    creating subnet, ports, and attaching to the router. Now, these three
    actions are individual methods.
    - FT calls to base methods are sorted in the following sequence:
        - Create net
        - Create subnet
        - Create port
        - Create router
        - Attach router to subnet
    - Retry mechanisms have been added to all methods that make calls to
    the L3_plugin for creation of resources.

    As this way every small method can be cover by retry in case Neutron
    reports a exception.

    NOTE: due to a cross depedency with fix done in [2], some tests have
    been skipped, but it will be restored in [2].

    [1] https://review.opendev.org/c/openstack/ovn-octavia-provider/+/883662
    [2] https://review.opendev.org/c/openstack/ovn-octavia-provider/+/911701

    Related-Bug: #2020195
    Change-Id: I98aad143556ff0ec7ecb365de99fd58a6dcb734e

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.