list object has no attribute 'acked'

[Terry Wilson]

Using python-ovs master, there are errors such as:

ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-81ffbcd9-59d1-498a-aea2-d57e0d515ff2 None None] OVS database connection to OVN_Southbound failed with error: ''list' object has no attribute 'acked''. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: AttributeError: 'list' object has no attribute 'acked'
ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last):
ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 127, in start_connection
ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.ovsdb_connection.start()
ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/local/lib/python3.10/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 83, in start
ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout)
ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 53, in wait_for_change
ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn while idl_.change_seqno == seqno and not idl_.run():
ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/opt/stack/ovs/python/ovs/db/idl.py", line 506, in run
ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.__send_monitor_request()
ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/opt/stack/ovs/python/ovs/db/idl.py", line 814, in __send_monitor_request
ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn not ConditionState.is_true(table.condition.acked)):
ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn AttributeError: 'list' object has no attribute 'acked'
ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn

ovsdbapp, neutron, and networking-ovn all directly set 'table.condition' which is kinda-sorta not really public API. The type of this variable changed with https://github.com/openvswitch/ovs/commit/46d44cf3be0dbf4a44cebea3b279b3d16a326796 and there has been some breakage.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ovsdbapp/+/834544

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/834613

[Lajos Katona (lajos-katona)]

There is another bug for new python-ovs:
https://bugs.launchpad.net/neutron/+bug/1965599

As I see there at least some overlap between them, not sure if one is duplicate of the other.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Lajos:

Yes, LP#1965599 describes the same problem. Because this bug has a patch related and is actively attended, I'll mark the other one as duplicated.

Thanks!

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovsdbapp/+/834544
Committed: https://opendev.org/openstack/ovsdbapp/commit/06207b33e3317b984868aeb67aaba4bb8423e710
Submitter: "Zuul (22348)"
Branch: master

commit 06207b33e3317b984868aeb67aaba4bb8423e710
Author: Terry Wilson <email address hidden>
Date: Mon Mar 21 19:36:42 2022 +0000

    Fix broken behavior after recent OVS update

    After the monitor_cond_since/update3 support patch in ovs
    (46d44cf3be0), directly setting table.condition is broken. This
    isn't something that was every truly supported. Ultimately, it
    would be good if we could make this code less dependent on exactly
    what happens in Idl.__init__()--maybe adding an update_tables()
    method in python-ovs.

    To avoid stomping on the state stored in ConditionState, instead
    of replacing Idl.tables, this removes deletions and adds new
    tables that were passed. Existing tables are left alone.

    Closes-Bug: 1965819
    Change-Id: Iad0d7472a7adce4a79111f94a2f33fc5b1a5c530

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/ovsdbapp/+/834727

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/ovsdbapp/+/834728

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/ovsdbapp/+/834729

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovsdbapp/+/834727
Committed: https://opendev.org/openstack/ovsdbapp/commit/de473cea9defe04a6f904fa63a8cf49746534916
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit de473cea9defe04a6f904fa63a8cf49746534916
Author: Terry Wilson <email address hidden>
Date: Mon Mar 21 19:36:42 2022 +0000

    Fix broken behavior after recent OVS update

    After the monitor_cond_since/update3 support patch in ovs
    (46d44cf3be0), directly setting table.condition is broken. This
    isn't something that was every truly supported. Ultimately, it
    would be good if we could make this code less dependent on exactly
    what happens in Idl.__init__()--maybe adding an update_tables()
    method in python-ovs.

    To avoid stomping on the state stored in ConditionState, instead
    of replacing Idl.tables, this removes deletions and adds new
    tables that were passed. Existing tables are left alone.

    Closes-Bug: 1965819
    Change-Id: Iad0d7472a7adce4a79111f94a2f33fc5b1a5c530
    (cherry picked from commit 06207b33e3317b984868aeb67aaba4bb8423e710)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovsdbapp/+/834728
Committed: https://opendev.org/openstack/ovsdbapp/commit/a7a1d5cfd3bded883c982f646758d2ab23cfb2c7
Submitter: "Zuul (22348)"
Branch: stable/xena

commit a7a1d5cfd3bded883c982f646758d2ab23cfb2c7
Author: Terry Wilson <email address hidden>
Date: Mon Mar 21 19:36:42 2022 +0000

    Fix broken behavior after recent OVS update

    After the monitor_cond_since/update3 support patch in ovs
    (46d44cf3be0), directly setting table.condition is broken. This
    isn't something that was every truly supported. Ultimately, it
    would be good if we could make this code less dependent on exactly
    what happens in Idl.__init__()--maybe adding an update_tables()
    method in python-ovs.

    To avoid stomping on the state stored in ConditionState, instead
    of replacing Idl.tables, this removes deletions and adds new
    tables that were passed. Existing tables are left alone.

    Closes-Bug: 1965819
    Change-Id: Iad0d7472a7adce4a79111f94a2f33fc5b1a5c530
    (cherry picked from commit 06207b33e3317b984868aeb67aaba4bb8423e710)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovsdbapp/+/834729
Committed: https://opendev.org/openstack/ovsdbapp/commit/99905097ceab83a07af4a7980c870da0ed416cf2
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 99905097ceab83a07af4a7980c870da0ed416cf2
Author: Terry Wilson <email address hidden>
Date: Mon Mar 21 19:36:42 2022 +0000

    Fix broken behavior after recent OVS update

    After the monitor_cond_since/update3 support patch in ovs
    (46d44cf3be0), directly setting table.condition is broken. This
    isn't something that was every truly supported. Ultimately, it
    would be good if we could make this code less dependent on exactly
    what happens in Idl.__init__()--maybe adding an update_tables()
    method in python-ovs.

    To avoid stomping on the state stored in ConditionState, instead
    of replacing Idl.tables, this removes deletions and adds new
    tables that were passed. Existing tables are left alone.

    Closes-Bug: 1965819
    Change-Id: Iad0d7472a7adce4a79111f94a2f33fc5b1a5c530
    (cherry picked from commit 06207b33e3317b984868aeb67aaba4bb8423e710)

[Frode Nordahl (fnordahl)]

There appears to be more to it. With this patch applied Neutron will still fail to start with the OVN driver when OVS 2.17 is installed.

With this instrumentation of db/idl.py:
diff --git a/python/ovs/db/idl.py b/python/ovs/db/idl.py
index 4ecdcaa19..d676a39b3 100644
--- a/python/ovs/db/idl.py
+++ b/python/ovs/db/idl.py
@@ -357,6 +357,7 @@ class Idl(object):
         """
         ack_all = self.last_id == str(uuid.UUID(int=0))
         for table in self.tables.values():
+ logging.info('HELLO ovs.db {}'.format(table.condition))
             if ack_all:
                 table.condition.request()
                 table.condition.ack()

I see:
2022-04-04 16:32:29.270 103953 INFO neutron.service [req-07913206-4206-49fc-bbfc-710c0e8f992a - - - - -] Neutron service started, listening on 0.0.0.0:9676
2022-04-04 16:32:29.296 103961 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [req-d6dc8c25-6296-43e4-9858-a93274def15d - - - - -] Getting OvsdbNbOvnIdl for WorkerService with retry
2022-04-04 16:32:29.297 103962 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [req-9d84dcb7-7b4f-4fad-8f87-c6b5238f7630 - - - - -] Getting OvsdbNbOvnIdl for WorkerService with retry
2022-04-04 16:32:29.297 103963 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [req-3fa2a398-db18-4319-9f3d-6067b55d1699 - - - - -] Getting OvsdbNbOvnIdl for WorkerService with retry
2022-04-04 16:32:29.299 103964 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [req-2821d81a-304f-491e-a053-ba845f5925cf - - - - -] Getting OvsdbNbOvnIdl for WorkerService with retry
2022-04-04 16:32:29.306 103953 INFO root [req-07913206-4206-49fc-bbfc-710c0e8f992a - - - - -] HELLO ovs.db [['name', '==', 'neutron_pg_drop']]
2022-04-04 16:32:29.306 103953 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [req-07913206-4206-49fc-bbfc-710c0e8f992a - - - - -] OVS database connection to OVN_Northbound failed with error: ''list' object has no attribute 'request''. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: AttributeError: 'list' object has no attribute 'request'
2022-04-04 16:32:29.306 103953 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last):
2022-04-04 16:32:29.306 103953 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 127, in start_connection
2022-04-04 16:32:29.306 103953 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.ovsdb_connection.start()
2022-04-04 16:32:29.306 103953 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 83, in start
2022-04-04 16:32:29.306 103953 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout)
2022-04-04 16:32:29.306 103953 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr...

[Terry Wilson (otherwiseguy)]

It takes both the ovsdbapp patch and the neutron patch.

[Terry Wilson (otherwiseguy)]

https://review.opendev.org/c/openstack/neutron/+/834613 for reference. Not sure why it didn't show up automatically.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-ovsdbapp - 1.15.1-0ubuntu2

---------------
python-ovsdbapp (1.15.1-0ubuntu2) jammy; urgency=medium

  * d/p/fix-broken-behavior-after-recent-OVS-update.patch: Cherry-picked
    from upstream to fix broken behavior after recent OVS update
    (LP: #1965819).

 -- Corey Bryant <email address hidden> Mon, 04 Apr 2022 16:30:53 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:20.0.0-0ubuntu2

---------------
neutron (2:20.0.0-0ubuntu2) jammy; urgency=medium

  * Fix broken behavior after recent OVS update (LP: #1965819):
    - d/control: Align with accompanying fix for python-ovsdbapp.
    - d/p/fix-setting-table-monitoring-conditions.patch: Cherry-picked
      from upstream.

 -- Corey Bryant <email address hidden> Mon, 04 Apr 2022 16:35:12 -0400

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ovsdbapp 1.16.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/834613
Committed: https://opendev.org/openstack/neutron/commit/a13a4cb2da03e2cdce04b0e7b4bc30e44ebdbd02
Submitter: "Zuul (22348)"
Branch: master

commit a13a4cb2da03e2cdce04b0e7b4bc30e44ebdbd02
Author: Terry Wilson <email address hidden>
Date: Tue Mar 22 04:12:08 2022 +0000

    Fix setting table monitoring conditions

    After the monitor_cond_since/update3 support patch in ovs
    (46d44cf3be0), directly setting table.condition is broken. This
    isn't something that was every truly supported. Prior to that
    patch, using Idl.cond_change() before a connection was made
    did not work, but after that patch it does.

    This patch uses the old behavior when the OVS library does not
    have the ConditionState object, and uses cond_change() otherwise.

    Related-Bug: #1965819
    Change-Id: I0503037b803a3c99fb7988bc20394c111ac456db

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/838702

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/838783

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/838784

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ovsdbapp 1.9.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ovsdbapp 1.15.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ovsdbapp 1.12.1 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/838883

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/838783
Committed: https://opendev.org/openstack/neutron/commit/af6b54a24b444d6014a4b6c7436046ccfb3aa74f
Submitter: "Zuul (22348)"
Branch: stable/xena

commit af6b54a24b444d6014a4b6c7436046ccfb3aa74f
Author: Terry Wilson <email address hidden>
Date: Tue Mar 22 04:12:08 2022 +0000

    Fix setting table monitoring conditions

    After the monitor_cond_since/update3 support patch in ovs
    (46d44cf3be0), directly setting table.condition is broken. This
    isn't something that was every truly supported. Prior to that
    patch, using Idl.cond_change() before a connection was made
    did not work, but after that patch it does.

    This patch uses the old behavior when the OVS library does not
    have the ConditionState object, and uses cond_change() otherwise.

    Related-Bug: #1965819
    Change-Id: I0503037b803a3c99fb7988bc20394c111ac456db
    (cherry picked from commit a13a4cb2da03e2cdce04b0e7b4bc30e44ebdbd02)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/838784
Committed: https://opendev.org/openstack/neutron/commit/58288d1795b1e2b30bb4df6abca4a768020bb2ff
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 58288d1795b1e2b30bb4df6abca4a768020bb2ff
Author: Terry Wilson <email address hidden>
Date: Tue Mar 22 04:12:08 2022 +0000

    Fix setting table monitoring conditions

    After the monitor_cond_since/update3 support patch in ovs
    (46d44cf3be0), directly setting table.condition is broken. This
    isn't something that was every truly supported. Prior to that
    patch, using Idl.cond_change() before a connection was made
    did not work, but after that patch it does.

    This patch uses the old behavior when the OVS library does not
    have the ConditionState object, and uses cond_change() otherwise.

    Related-Bug: #1965819
    Change-Id: I0503037b803a3c99fb7988bc20394c111ac456db
    (cherry picked from commit a13a4cb2da03e2cdce04b0e7b4bc30e44ebdbd02)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/838883
Committed: https://opendev.org/openstack/neutron/commit/37c2e6b708678d2e648f111d102b6394f4e704b0
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 37c2e6b708678d2e648f111d102b6394f4e704b0
Author: Terry Wilson <email address hidden>
Date: Tue Mar 22 04:12:08 2022 +0000

    Fix setting table monitoring conditions

    After the monitor_cond_since/update3 support patch in ovs
    (46d44cf3be0), directly setting table.condition is broken. This
    isn't something that was every truly supported. Prior to that
    patch, using Idl.cond_change() before a connection was made
    did not work, but after that patch it does.

    This patch uses the old behavior when the OVS library does not
    have the ConditionState object, and uses cond_change() otherwise.

    Related-Bug: #1965819
    Change-Id: I0503037b803a3c99fb7988bc20394c111ac456db
    (cherry picked from commit a13a4cb2da03e2cdce04b0e7b4bc30e44ebdbd02)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/838702
Committed: https://opendev.org/openstack/neutron/commit/51e4c3aad2826e401db5b37b3ab594883208dea7
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 51e4c3aad2826e401db5b37b3ab594883208dea7
Author: Terry Wilson <email address hidden>
Date: Tue Mar 22 04:12:08 2022 +0000

    Fix setting table monitoring conditions

    After the monitor_cond_since/update3 support patch in ovs
    (46d44cf3be0), directly setting table.condition is broken. This
    isn't something that was every truly supported. Prior to that
    patch, using Idl.cond_change() before a connection was made
    did not work, but after that patch it does.

    This patch uses the old behavior when the OVS library does not
    have the ConditionState object, and uses cond_change() otherwise.

    Related-Bug: #1965819
    Change-Id: I0503037b803a3c99fb7988bc20394c111ac456db
    (cherry picked from commit a13a4cb2da03e2cdce04b0e7b4bc30e44ebdbd02)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/840744

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/840744
Committed: https://opendev.org/openstack/neutron/commit/a2d65cecdee746f4b8b6e42348c4bd9206e12cfd
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit a2d65cecdee746f4b8b6e42348c4bd9206e12cfd
Author: Terry Wilson <email address hidden>
Date: Tue Mar 22 04:12:08 2022 +0000

    Fix setting table monitoring conditions

    After the monitor_cond_since/update3 support patch in ovs
    (46d44cf3be0), directly setting table.condition is broken. This
    isn't something that was every truly supported. Prior to that
    patch, using Idl.cond_change() before a connection was made
    did not work, but after that patch it does.

    This patch uses the old behavior when the OVS library does not
    have the ConditionState object, and uses cond_change() otherwise.

    Related-Bug: #1965819
    Change-Id: I0503037b803a3c99fb7988bc20394c111ac456db
    (cherry picked from commit 37c2e6b708678d2e648f111d102b6394f4e704b0)