[OVN] SB connection unreliable

[Piotr Parczewski]

I believe that patch https://review.opendev.org/c/openstack/neutron/+/789255 might have broken some functionality, eg. this happens: http://paste.openstack.org/show/805541/

[Bernard Cafarelli (bcafarel)]

Some additional logs in original bugreport https://bugs.launchpad.net/bugs/1928794

This may need some additional OVN eyes, maybe some error in OVN database.

How often does this happen? On startup only, on port operations (create/delete), regularly (maintenance operations)?

https://review.opendev.org/c/openstack/neutron/+/754223 itself should not have visible change on connection DB stability

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Piotr:

I'll try to confirm that with OVN developers. I think this is happening before calling "OVNMechanismDriver.post_fork_initialize", where both IDLs are assigned. That means this error is happening when the Neutron server is starting. Can you confirm I'm right on this? Or did you saw this error before the initial transient period?

Regards.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

This patch is related to this bug: https://review.opendev.org/c/openstack/neutron/+/781555

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/781555
Committed: https://opendev.org/openstack/neutron/commit/90980f496cfa3cc5df1c93cf834a44f33d3f1f6f
Submitter: "Zuul (22348)"
Branch: master

commit 90980f496cfa3cc5df1c93cf834a44f33d3f1f6f
Author: Terry Wilson <email address hidden>
Date: Thu Mar 18 15:01:07 2021 -0500

    Add wait for the post-fork event to nb/sb objects

    It is possible for events from the nb/sb to fire before the opposite
    db connection is made. These events can call back into driver code
    which tries to access the other db before it is connected.

    Closes-Bug: #1929197
    Closes-Bug: #1928794
    Closes-Bug: #1929633
    Change-Id: If947581b90ced42981c4611c32de8f428a052c69

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/805768

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/805769

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Mark Goddard <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/805769
Reason: Seems this fix was sufficient, but not yet in RDO victoria: https://github.com/openstack/neutron/commit/d326ec6b9dec9c1f5cfbba7f7183840d3eaee226

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Mark Goddard <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/805768
Reason: Seems this fix was sufficient, but not yet in RDO victoria: https://github.com/openstack/neutron/commit/d326ec6b9dec9c1f5cfbba7f7183840d3eaee226

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/805768
Committed: https://opendev.org/openstack/neutron/commit/3139bbcbe6b3747ce432901a5a316d93f8f168fb
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 3139bbcbe6b3747ce432901a5a316d93f8f168fb
Author: Terry Wilson <email address hidden>
Date: Thu Mar 18 15:01:07 2021 -0500

    Add wait for the post-fork event to nb/sb objects

    It is possible for events from the nb/sb to fire before the opposite
    db connection is made. These events can call back into driver code
    which tries to access the other db before it is connected.

    Conflicts:
            neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py
            neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py
            neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovsdb_monitor.py
            neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovn_db_sync.py
            neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovsdb_monitor.py
            neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/test_mech_driver.py

    Closes-Bug: #1929197
    Closes-Bug: #1928794
    Closes-Bug: #1929633
    Change-Id: If947581b90ced42981c4611c32de8f428a052c69
    (cherry picked from commit 90980f496cfa3cc5df1c93cf834a44f33d3f1f6f)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.4.0 release.