python-ovs package 2.17 introduce table.condition attribute which is overwriten by neutron ovsdb code

[Mirek]

I've opened it up initially with openvswitch team before I dug a bit more into it. It looks like changes in 2.17 code introduce condition attribute to sync changes. However, neutron also use condition attribute, e.g. neutron/agent/ovn/metadata/ovsdb.py
class MetadataAgentOvnSbIdl(ovsdb_monitor.OvnIdl):
...
        if chassis:
            for table in set(tables).intersection({'Chassis',
                                                   'Chassis_Private'}):
                self.tables[table].condition = [['name', '==', chassis]]

So what happen is ovs idl code is falling with the following error:
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: ''list' object has no attribute 'request''. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: AttributeError: 'list' object has no attribute 'request'
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last):
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 127, in start_connection
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.ovsdb_connection.start()
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 83, in start
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout)
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 53, in wait_for_change
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn while idl_.change_seqno == seqno and not idl_.run():
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 421, in run
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.restart_fsm()
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 383, in restart_fsm
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.sync_conditions()
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 372, in sync_conditions
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn table.condition.request()
2022-03-18 21:55:11.894 24 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn AttributeError: 'list' object has no attribute 'request'

When I've added a few print lines to find out what is going on, I found out that ovs is expecting ConditionState object, but neutron code is also using condition attribute and when I've override __setattr__ func to find out who is overwrite the ovs code. It turns out ovsdb.py also use that attribute.

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 150, in publish
    errors = self._notify_loop(resource, event, trigger, payload)
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop
    callback(resource, event, trigger, payload=payload)
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/agent/ovn/metadata/server.py", line 74, in post_fork_initialize
    self.sb_idl = ovsdb.MetadataAgentOvnSbIdl(
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/neutron/agent/ovn/metadata/ovsdb.py", line 53, in __init__
    self.tables[table].condition = [['name', '==', chassis]]
  File "/usr/lib/python3/dist-packages/ovs/db/schema.py", line 190, in __setattr__
    vlog.err(''.join(traceback.format_stack()))

https://opendev.org/openstack/neutron/src/branch/master/neutron/agent/ovn/metadata/ovsdb.py
class MetadataAgentOvnSbIdl(ovsdb_monitor.OvnIdl):

    SCHEMA = 'OVN_Southbound'

    def __init__(self, chassis=None, events=None, tables=None):
        connection_string = config.get_ovn_sb_connection()
        ovsdb_monitor._check_and_set_ssl_files(self.SCHEMA)
        helper = self._get_ovsdb_helper(connection_string)
        if tables is None:
            tables = ('Chassis', 'Encap', 'Port_Binding', 'Datapath_Binding',
                      'SB_Global')
        for table in tables:
            helper.register_table(table)
        try:
            super(MetadataAgentOvnSbIdl, self).__init__(
                None, connection_string, helper, leader_only=False)
        except TypeError:
            # TODO(twilson) We can remove this when we require ovs>=2.12.0
            super(MetadataAgentOvnSbIdl, self).__init__(
                None, connection_string, helper)
        if chassis:
            for table in set(tables).intersection({'Chassis',
                                                   'Chassis_Private'}):
                self.tables[table].condition = [['name', '==', chassis]]

The workaround at the moment is to downgrade to 2.16, however, it's problematic as neutron requiments.txt are only setting lower limit for ovs package.
ovs>=2.10.0 # Apache-2.0

[Mirek (mirek186)]

Small comment, I've hit the bug when building my OpenStack using the Kolla master branch which has been set for Yoga and used the Ubuntu cloud package for Yoga which is 2.17. It's not an issue for Xena.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Mirek:

Currently, only python package ovs v2.16.0 [1] is available for downloading. Also maximum version tested in 2.16.0 [2].

In any case, since [3], as you reported, the IDL.condition member changed. The author of this commit is testing the code and will provide a solution.

Regards.

[1]https://pypi.org/project/ovs/
[2]https://github.com/openstack/requirements/blob/2f2467ad1508f4b125fc3f15c9454dc711baca01/upper-constraints.txt#L197
[3]https://github.com/openvswitch/ovs/commit/46d44cf3be0dbf4a44cebea3b279b3d16a326796

[Dr. Jens Harbott (j-harbott)]

kolla is not using the package from pypi in this case, but the version packaged by Ubuntu[1] which is backported for Focal in the Yoga Ubuntu Cloud Archive, matching the version of OVS being deployed.

Maybe Neutron could start testing against those, too. Do you know why the python pkgs hasn't been published to pypi when OVS 2.17.0 was tagged one month ago?

[1] https://launchpad.net/ubuntu/jammy/+package/python3-openvswitch

[Lajos Katona (lajos-katona)]

Please check the other bug which seems related:
https://bugs.launchpad.net/ovsdbapp/+bug/1965819

proposed fixes for it:
https://review.opendev.org/c/openstack/ovsdbapp/+/834544
https://review.opendev.org/c/openstack/neutron/+/834613