Functional test neutron.tests.functional.services.ovn_l3.test_plugin.TestRouter.test_gateway_chassis_rebalance fails intermittently

[Slawek Kaplonski]

Failure example: https://32c2553a5b1b0d58bd31-9e074609c9fda723118db1a634ce1a12.ssl.cf5.rackcdn.com/873247/3/gate/neutron-functional-with-uwsgi/6195659/testr_results.html

Stacktrace:
ft1.9: neutron.tests.functional.services.ovn_l3.test_plugin.TestRouter.test_gateway_chassis_rebalancetesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/services/ovn_l3/test_plugin.py", line 560, in test_gateway_chassis_rebalance
    self.assertTrue(self.cr_lrp_pb_event.wait(logical_port),
  File "/usr/lib/python3.10/unittest/case.py", line 687, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true : lrp cr-lrp-77e343ce-8e3c-472d-8eca-acde6196ea38 failed to bind

[yatin (yatinkarel)]

As per opensearch[1] seen 3 times in last 15 days, 1 periodic[2] and 2 open[3][4] patches:-

[1] https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22AssertionError:%20False%20is%20not%20true%20:%20lrp%22'),sort:!())
[2] https://b816c59e598aa15486ce-e6ce8e54d026b827dc95e0507d541939.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/stable/zed/neutron-functional/f2f3d57/
[3] https://11111b5d40053b2a4bc3-2ae12c77ffe7ca7f130956997762f3ed.ssl.cf5.rackcdn.com/873111/7/check/neutron-functional-with-uwsgi-20/9ae7d70
[4] https://32c2553a5b1b0d58bd31-9e074609c9fda723118db1a634ce1a12.ssl.cf5.rackcdn.com/873247/3/gate/neutron-functional-with-uwsgi/6195659

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/874765

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/874765
Committed: https://opendev.org/openstack/neutron/commit/1e41558851b7e561e1dcf99499e6f7920a91fa72
Submitter: "Zuul (22348)"
Branch: master

commit 1e41558851b7e561e1dcf99499e6f7920a91fa72
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Sat Feb 18 05:16:35 2023 +0100

    Reduce number of routers created in ``test_gateway_chassis_rebalance``

    This patch reduces the number of routers created in
    ``test_gateway_chassis_rebalance``. For the test purposes, 5 routers
    is enough, reduce the test executing time and the OVN load.

    This patch also captures the "Port_Binding" update event in the
    ``WaitForCrLrpPortBindingEvent`` class.

    Related-Bug: #2007353
    Change-Id: I66b39b1d0dccfbb18bec5646d8954eee8c687f0c