[OVN] functional test failing TestNBDbMonitorOverTcp test_floatingip_mac_bindings

[Maciej Jozefczyk]

We can find random failures of the test:

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html

neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovsdb_monitor.TestNBDbMonitorOverTcp.test_floatingip_mac_bindings

ft3.1: neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovsdb_monitor.TestNBDbMonitorOverTcp.test_floatingip_mac_bindingstesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovsdb_monitor.py", line 132, in test_floatingip_mac_bindings
    macb_id = self.sb_api.db_create('MAC_Binding', datapath=dp[0]['_uuid'],
IndexError: list index out of range

[Maciej Jozefczyk (maciejjozefczyk)]

Flow of calls:
1. functional test creates network in Neutron API
2. OVN ml2 mechanism adds the required Logical_Switch
3. ovn-northd gets the data and creates datapath_binding in OVN SBDB.
4. Test reads the value of datapath_binding from OVN SBDB

The problem described in this bug its race-condition between 3 and 4.

In fact, based on the logs, I can find that the datapath_binding was created in SBDB, but functional test was *too fast* in reading it.
Found entry from SBDB:

OVSDB JSON 10510 a89109fbf52966f67327e1a51bd8619fc525413d
{"_date":1591198467205,"Datapath_Binding":{"d09c0284-afb1-441d-92c4-535ce08fadf1":{"tunnel_key":1,"external_ids":["map",[["logical-switch","f501d844-85c5-4090-8b28-850bf9f8b2dd"],["name","neutron-c3e0f8fd-ef9e-420e-bfe4-cfbcec09731d"],["name2","network1"]]]}},"Logical_Flow":{"5d68227e-8152-4532-9e42-a4bb21d50eb3":

[Maciej Jozefczyk (maciejjozefczyk)]

In functional tests we have 2 places like this, that could be affected by this problem:

stack@mjozefcz-devstack-both:~/neutron$ grep -Ri sb_api . | grep db_find
./neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovsdb_monitor.py: dp = self.sb_api.db_find(
./neutron/tests/functional/agent/ovn/metadata/test_metadata_agent.py: chassis_row = self.sb_api.db_find(
stack@mjozefcz-devstack-both:~/neutron$

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/738415

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738415
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=598e0376c6497929a5abc1aa7c7ea1dada5de589
Submitter: Zuul
Branch: master

commit 598e0376c6497929a5abc1aa7c7ea1dada5de589
Author: Maciej Jozefczyk <email address hidden>
Date: Mon Jun 29 10:55:14 2020 +0000

    [OVN] Wait for WaitForDataPathBindingCreateEvent event in functional tests

    There is a bug in OVN functional tests, where it looks for datapath binding
    of just created network.
    In rare conditions datapath binding entry could not be in place in time,
    because it is a ovn-northd responsibility to create it and we don't lock
    this process in neutron api.

    Change-Id: Ice115623491ad5b50397a0338f0a7780dc05d24c
    Closes-Bug: #1884986

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/738865

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/738865
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8341a87bd0272ddfc472138643711626f43ec58d
Submitter: Zuul
Branch: stable/ussuri

commit 8341a87bd0272ddfc472138643711626f43ec58d
Author: Maciej Jozefczyk <email address hidden>
Date: Mon Jun 29 10:55:14 2020 +0000

    [OVN] Wait for WaitForDataPathBindingCreateEvent event in functional tests

    There is a bug in OVN functional tests, where it looks for datapath binding
    of just created network.
    In rare conditions datapath binding entry could not be in place in time,
    because it is a ovn-northd responsibility to create it and we don't lock
    this process in neutron api.

    Change-Id: Ice115623491ad5b50397a0338f0a7780dc05d24c
    Closes-Bug: #1884986
    (cherry picked from commit 598e0376c6497929a5abc1aa7c7ea1dada5de589)

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

This error happened again in [1]. The datapath is not created when the DB is queried.

Snippet: http://paste.openstack.org/show/801502/

[1]https://8ee4c4cb8ca7de3283c1-5e9e014bbb914c03e8d5277ea7a5cf3c.ssl.cf2.rackcdn.com/763410/18/check/neutron-functional-with-uwsgi/58c2b44/testr_results.html