[OVN Octavia Provider] OVN provider tests using too old version of OVN

[Jake Yip]

While trying to test out ovn-octavia-provider in devstack, I copied the functional testing vars, e.g.

 OVN_BRANCH: v20.06.0
 OVS_BRANCH: v2.13.0

However, in devstack the octavia-driver-agent keeps crashing with a trace like this

ERROR octavia.cmd.driver_agent Traceback (most recent call last):
ERROR octavia.cmd.driver_agent File "/opt/stack/octavia/octavia/cmd/driver_agent.py", line 65, in _process_wrapper
ERROR octavia.cmd.driver_agent function(exit_event)
ERROR octavia.cmd.driver_agent File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/agent.py", line 42, in OvnProviderAgent
ERROR octavia.cmd.driver_agent ovn_sb_idl_for_events = impl_idl_ovn.OvnSbIdlForLb(
ERROR octavia.cmd.driver_agent File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/impl_idl_ovn.py", line 237, in __init__
ERROR octavia.cmd.driver_agent super().__init__(
ERROR octavia.cmd.driver_agent File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/ovsdb/ovsdb_monitor.py", line 42, in __init__
ERROR octavia.cmd.driver_agent super().__init__(remote, schema)
ERROR octavia.cmd.driver_agent File "/usr/local/lib/python3.8/dist-packages/ovs/db/idl.py", line 141, in __init__
ERROR octavia.cmd.driver_agent schema = schema_helper.get_idl_schema()
ERROR octavia.cmd.driver_agent File "/usr/local/lib/python3.8/dist-packages/ovs/db/idl.py", line 2058, in get_idl_schema
ERROR octavia.cmd.driver_agent self._keep_table_columns(schema, table, columns))
ERROR octavia.cmd.driver_agent File "/usr/local/lib/python3.8/dist-packages/ovs/db/idl.py", line 2066, in _keep_table_columns
ERROR octavia.cmd.driver_agent assert table_name in schema.tables
ERROR octavia.cmd.driver_agent AssertionError
ERROR octavia.cmd.driver_agent

After a bit of debugging, I found out that it is trying to get the 'Load_Balancer' table in the SB database, which is a recent commit[1] in v20.12.0

Not sure if this is a valid bug, but if the tests can be incremented to a working version of OVN that might help prevent others from tripping over this case. Thanks!

[1] https://github.com/ovn-org/ovn/commit/42e694f03c187137852c2d7349daa0541a4f5e62

[Brian Haley (brian-haley)]

There are two gate jobs in this repo, one testing "released" software, the other using the "main/master" branch. Both jobs seem to be Ok in recent patches, so I'm guessing there's something else you forgot to set in your local.conf file.

That isn't saying these settings shouldn't possibly be changed to later versions, typically they follow what the neutron repo does, which looks like it changed last November:

      OVN_BRANCH: v21.06.0
      OVS_BRANCH: v2.16.0

There's typically reasons involving the underlying testing infrastructure as to what versions are used, but you could look into a bump like this.

[Jake Yip (waipengyip)]

Thanks Brian for replying.

I checked a ovn-octavia-provider-functional-release job that it seems like OVN_BUILD_FROM_SOURCE is not set, so it may be installing from package? In which case these vars are not used.

I can't find the package that was installed, but from the job-output.txt, hints like

 OVN_CONTROLLER_SERVICE=ovn-controller.service

instead of

 <email address hidden>

leads me to believe that OVN in those tests might be from a package.

I am not familiar with ovn-octavia-provider devstack, so I think I am stuck now.

Anyway I have got a solution going forward, which is setting

 OVN_BUILD_FROM_SOURCE=True
 Q_BUILD_OVS_FROM_GIT=True
 INSTALL_OVN=True
 OVN_BRANCH=main
 OVS_BRANCH=master

for devstack testing, so you can close this bug if you want.

[Akihiro Motoki (amotoki)]

Looking at a recent result of ovn-octavia-provider-functional-release [1], it seems that local.conf is not configured properly [2]. This needs more investigation.

[1] https://zuul.opendev.org/t/openstack/build/a39b05e463db472d8d5836eb9783ba0e
[2] https://2c442655f7e6a91a0668-dd7612090ddd37717fc79d2c5d5bec61.ssl.cf2.rackcdn.com/829126/21/check/ovn-octavia-provider-functional-release/a39b05e/controller/logs/local_conf.txt

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/833798

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/833798
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/5bee600eddd872d43ba6c51400c20fa5f50bd97f
Submitter: "Zuul (22348)"
Branch: master

commit 5bee600eddd872d43ba6c51400c20fa5f50bd97f
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue Mar 15 14:30:56 2022 +0000

    Fix zuul templates for functional and tempest tests

    * The "devstack_localrc" key was missing, without it local.conf won't be
      populated.

    * Added OVN_BUILD_FROM_SOURCE to True to make sure OVN is compiled
    from code and not installed from packages.

    * Updated the version of OVN and OVS in the -released job to a newer
      version.

    Closes-Bug: #1964339
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    Change-Id: I05ed8fb053c78bd16a52b5d82a3ab51faf856d78

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/834454

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/834661

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/834454
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/3acaa772c5cfc9609501a9110294d5505c595c80
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 3acaa772c5cfc9609501a9110294d5505c595c80
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue Mar 15 14:30:56 2022 +0000

    Fix zuul templates for functional and tempest tests

    * The "devstack_localrc" key was missing, without it local.conf won't be
      populated.

    * Added OVN_BUILD_FROM_SOURCE to True to make sure OVN is compiled
    from code and not installed from packages.

    * Updated the version of OVN and OVS in the -released job to a newer
      version.

    * Configure correct db user in the functional tests job
      This was changed in the Neutron in commit [1] and should
      be updated also in the configure_functional_tests role in
      this repo.

    * Remove run-devstack role from run_functional_job playbook
      There is no need to run that role in the functional
      job and not running it can save few minutes in each job's run.
      The same change was done in Neutron in patch [2].

    [1] https://review.opendev.org/c/openstack/neutron/+/814009
    [2] https://review.opendev.org/c/openstack/neutron/+/816354

    Conflicts:
        zuul.d/base.yaml

    Closes-Bug: #1964339
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    Change-Id: I05ed8fb053c78bd16a52b5d82a3ab51faf856d78
    (cherry picked from commit 5bee600eddd872d43ba6c51400c20fa5f50bd97f)
    (cherry picked from commit 1ee61f955fa13ebffdb70b027b8768915e74fa57)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/834661
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/0fff1651a4ff414a267733ea71a3882b07215a04
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 0fff1651a4ff414a267733ea71a3882b07215a04
Author: Fernando Royo <email address hidden>
Date: Tue Mar 22 12:43:26 2022 +0100

    Fix zuul templates for functional and tempest tests

    * The "devstack_localrc" key was missing, without it local.conf won't be
      populated.

    * Added OVN_BUILD_FROM_SOURCE to True to make sure OVN is compiled
    from code and not installed from packages.

    * Updated the version of OVN and OVS in the -released job to a newer
      version, according to the referenced in Neutron stable/wallaby
      branch [1] .

    * Configure correct db user in the functional tests job
      This was changed in the Neutron in commit [2] and should
      be updated also in the configure_functional_tests role in
      this repo.

    * Remove run-devstack role from run_functional_job playbook
      There is no need to run that role in the functional
      job and not running it can save few minutes in each job's run.
      The same change was done in Neutron in patch [3].

    [1] https://review.opendev.org/c/openstack/neutron/+/751882
    [2] https://review.opendev.org/c/openstack/neutron/+/814009
    [3] https://review.opendev.org/c/openstack/neutron/+/816354

    Conflicts:
       zuul.d/base.yaml

    Closes-Bug: #1964339
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    Change-Id: I05ed8fb053c78bd16a52b5d82a3ab51faf856d78
    (cherry picked from commit 5bee600eddd872d43ba6c51400c20fa5f50bd97f)
    (cherry picked from commit 1ee61f955fa13ebffdb70b027b8768915e74fa57)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ovn-octavia-provider 1.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ovn-octavia-provider 3.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ovn-octavia-provider 1.2.0 release.