Functional test neutron.tests.functional.services.ovn_l3.test_plugin.TestRouter.test_gateway_chassis_rebalance fails intermittently

[Slawek Kaplonski]

Failure example: https://32c2553a5b1b0d58bd31-9e074609c9fda723118db1a634ce1a12.ssl.cf5.rackcdn.com/873247/3/gate/neutron-functional-with-uwsgi/6195659/testr_results.html

Stacktrace:
ft1.9: neutron.tests.functional.services.ovn_l3.test_plugin.TestRouter.test_gateway_chassis_rebalancetesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/services/ovn_l3/test_plugin.py", line 560, in test_gateway_chassis_rebalance
    self.assertTrue(self.cr_lrp_pb_event.wait(logical_port),
  File "/usr/lib/python3.10/unittest/case.py", line 687, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true : lrp cr-lrp-77e343ce-8e3c-472d-8eca-acde6196ea38 failed to bind

[yatin (yatinkarel)]

As per opensearch[1] seen 3 times in last 15 days, 1 periodic[2] and 2 open[3][4] patches:-

[1] https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22AssertionError:%20False%20is%20not%20true%20:%20lrp%22'),sort:!())
[2] https://b816c59e598aa15486ce-e6ce8e54d026b827dc95e0507d541939.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/stable/zed/neutron-functional/f2f3d57/
[3] https://11111b5d40053b2a4bc3-2ae12c77ffe7ca7f130956997762f3ed.ssl.cf5.rackcdn.com/873111/7/check/neutron-functional-with-uwsgi-20/9ae7d70
[4] https://32c2553a5b1b0d58bd31-9e074609c9fda723118db1a634ce1a12.ssl.cf5.rackcdn.com/873247/3/gate/neutron-functional-with-uwsgi/6195659

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/874765

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/874765
Committed: https://opendev.org/openstack/neutron/commit/1e41558851b7e561e1dcf99499e6f7920a91fa72
Submitter: "Zuul (22348)"
Branch: master

commit 1e41558851b7e561e1dcf99499e6f7920a91fa72
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Sat Feb 18 05:16:35 2023 +0100

    Reduce number of routers created in ``test_gateway_chassis_rebalance``

    This patch reduces the number of routers created in
    ``test_gateway_chassis_rebalance``. For the test purposes, 5 routers
    is enough, reduce the test executing time and the OVN load.

    This patch also captures the "Port_Binding" update event in the
    ``WaitForCrLrpPortBindingEvent`` class.

    Related-Bug: #2007353
    Change-Id: I66b39b1d0dccfbb18bec5646d8954eee8c687f0c

[yatin (yatinkarel)]

Seen another hit https://c364e9923423c6bcf8db-8057aaa304527df3bf7b3a2e2ca6ed85.ssl.cf2.rackcdn.com/927707/2/gate/neutron-functional-with-uwsgi/3518f33/testr_results.html

[yatin (yatinkarel)]

Another hit https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_40c/openstack/40c2c88671284a97a43510af21f2c8bf/testr_results.html

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/957679

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/957679
Committed: https://opendev.org/openstack/neutron/commit/f7e056198485b5d42bcc8a1807dab3ec710cb7e1
Submitter: "Zuul (22348)"
Branch: master

commit f7e056198485b5d42bcc8a1807dab3ec710cb7e1
Author: Yatin Karel <email address hidden>
Date: Mon Aug 18 17:21:49 2025 +0530

    Re Reduce number of routers created in test_gateway_chassis_rebalance

    Was originally reduced in [1] but got restored in [2] during rebase.
    This patch re applies it.

    [1] https://review.opendev.org/c/openstack/neutron/+/874765
    [2] https://review.opendev.org/c/openstack/neutron/+/874760

    Related-Bug: #2007353
    Signed-off-by: Yatin Karel <email address hidden>
    Change-Id: I5765b32d6d33eda88ab0206e75e57b67a4afc4b5

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2025.1
Review: https://review.opendev.org/c/openstack/neutron/+/958013

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2024.2
Review: https://review.opendev.org/c/openstack/neutron/+/958014

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/neutron/+/958015

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/958013
Committed: https://opendev.org/openstack/neutron/commit/213eedf67beaf1d5280c19f8962c933d52badc90
Submitter: "Zuul (22348)"
Branch: stable/2025.1

commit 213eedf67beaf1d5280c19f8962c933d52badc90
Author: Yatin Karel <email address hidden>
Date: Mon Aug 18 17:21:49 2025 +0530

    Re Reduce number of routers created in test_gateway_chassis_rebalance

    Was originally reduced in [1] but got restored in [2] during rebase.
    This patch re applies it.

    [1] https://review.opendev.org/c/openstack/neutron/+/874765
    [2] https://review.opendev.org/c/openstack/neutron/+/874760

    Related-Bug: #2007353
    Signed-off-by: Yatin Karel <email address hidden>
    Change-Id: I5765b32d6d33eda88ab0206e75e57b67a4afc4b5
    (cherry picked from commit f7e056198485b5d42bcc8a1807dab3ec710cb7e1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/958014
Committed: https://opendev.org/openstack/neutron/commit/b3bd1fd42f7f9da2d03dcb2dcac4dea6f7bc1523
Submitter: "Zuul (22348)"
Branch: stable/2024.2

commit b3bd1fd42f7f9da2d03dcb2dcac4dea6f7bc1523
Author: Yatin Karel <email address hidden>
Date: Mon Aug 18 17:21:49 2025 +0530

    Re Reduce number of routers created in test_gateway_chassis_rebalance

    Was originally reduced in [1] but got restored in [2] during rebase.
    This patch re applies it.

    [1] https://review.opendev.org/c/openstack/neutron/+/874765
    [2] https://review.opendev.org/c/openstack/neutron/+/874760

    Related-Bug: #2007353
    Signed-off-by: Yatin Karel <email address hidden>
    Change-Id: I5765b32d6d33eda88ab0206e75e57b67a4afc4b5
    (cherry picked from commit f7e056198485b5d42bcc8a1807dab3ec710cb7e1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/958015
Committed: https://opendev.org/openstack/neutron/commit/4086ec1c97d136a79f2954b4d296c99c10dd6696
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit 4086ec1c97d136a79f2954b4d296c99c10dd6696
Author: Yatin Karel <email address hidden>
Date: Mon Aug 18 17:21:49 2025 +0530

    Re Reduce number of routers created in test_gateway_chassis_rebalance

    Was originally reduced in [1] but got restored in [2] during rebase.
    This patch re applies it.

    [1] https://review.opendev.org/c/openstack/neutron/+/874765
    [2] https://review.opendev.org/c/openstack/neutron/+/874760

    Related-Bug: #2007353
    Signed-off-by: Yatin Karel <email address hidden>
    Change-Id: I5765b32d6d33eda88ab0206e75e57b67a4afc4b5
    (cherry picked from commit f7e056198485b5d42bcc8a1807dab3ec710cb7e1)