Gate: Fuctional test_virtual_port_host_update failes recently

[Lajos Katona]

neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovsdb_monitor.TestNBDbMonitorOverTcp.test_virtual_port_host_update failes recently quite often, example failure (see [1]):

ft1.2: neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovsdb_monitor.TestNBDbMonitorOverTcp.test_virtual_port_host_updatetesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)
  File "/usr/lib/python3.10/unittest/mock.py", line 1369, in patched
    return func(*newargs, **newkeywargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/ovn/mech_driver/ovsdb/test_ovsdb_monitor.py", line 403, in test_virtual_port_host_update
    mock_update_vip_host.assert_called_once_with(vip['id'], None)
  File "/usr/lib/python3.10/unittest/mock.py", line 930, in assert_called_once_with
    raise AssertionError(msg)
AssertionError: Expected 'update_virtual_port_host' to be called once. Called 2 times.
Calls: [call('18c414c7-f897-4b39-bdb4-d35953df1d2f', None),
 call('18c414c7-f897-4b39-bdb4-d35953df1d2f', None)].

Opensearch link:
https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(build_status,build_name),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22Expected%20update_virtual_port_host%20to%20be%20called%22'),sort:!())

[1]: https://872de5c590dd926ff0db-30e72828a36544d0c7466f2989d78bfe.ssl.cf1.rackcdn.com/885341/2/check/neutron-functional-with-uwsgi/3079c44/testr_results.html

[Lajos Katona (lajos-katona)]

It seems that https://review.opendev.org/c/openstack/neutron/+/883681 is related to the issue.

[Lajos Katona (lajos-katona)]

I can reproduce it in my local env (the 2nd run hit it)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/886167

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/886167
Committed: https://opendev.org/openstack/neutron/commit/593278550a8185dfc52fc6c327ca1198df11e422
Submitter: "Zuul (22348)"
Branch: master

commit 593278550a8185dfc52fc6c327ca1198df11e422
Author: elajkat <email address hidden>
Date: Wed Jun 14 15:15:06 2023 +0200

    Functional: assert multiple calls for update_virtual_port_host

    test_virtual_port_host_update assumed update_virtual_port_host
    to be called once, but the test started to fail when [1] was
    merged and OVN version was bumped to v22.03.3 (not yet released).
    PortBindingUpdateVirtualPortsEvent with event ROW_DELETE
    received twice in the failing case.

    [1]: https://review.opendev.org/c/openstack/neutron/+/880890

    Closes-bug: #2023634
    Change-Id: I0ac92e3692fb1ba9667fee2f6fbf531a57b7b5b8

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 23.0.0.0b3 development milestone.

[yatin (yatinkarel)]

<< when [1] was merged and OVN version was bumped to v22.03.3 (not yet released).
Seeing the same faillure in stable/zed and stable/2023.1
- https://224da0befc3b7b6e8e72-1be58a446142080925c61267400301cb.ssl.cf5.rackcdn.com/894009/1/gate/neutron-functional-with-uwsgi/f58c60b/testr_results.html
- https://fbf109e10bffd8a657b2-ace28e47b1213bdab56305960a7f8056.ssl.cf5.rackcdn.com/893553/1/check/neutron-functional-with-uwsgi/4b5e791/testr_results.html

And both the branches running with OVN_BRANCH=v21.06.0 so the failure doesn't look related to v22.03.3 as mentioned in the commit message, will propose backport for the fix as it's happening after backports of https://review.opendev.org/q/I34caf7d0212ccb4bd7259c4414e7c3994bd8da4d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/894264

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/894265

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/894264
Committed: https://opendev.org/openstack/neutron/commit/4a7e7f2d258df8e1f93c3e397d165bdb437354ac
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 4a7e7f2d258df8e1f93c3e397d165bdb437354ac
Author: elajkat <email address hidden>
Date: Wed Jun 14 15:15:06 2023 +0200

    Functional: assert multiple calls for update_virtual_port_host

    test_virtual_port_host_update assumed update_virtual_port_host
    to be called once, but the test started to fail when [1] was
    merged and OVN version was bumped to v22.03.3 (not yet released).
    PortBindingUpdateVirtualPortsEvent with event ROW_DELETE
    received twice in the failing case.

    Backport Note: It's required since [2] merged, ovn version deployed is
    not related to the failure.

    [1]: https://review.opendev.org/c/openstack/neutron/+/880890
    [2]: https://review.opendev.org/q/I34caf7d0212ccb4bd7259c4414e7c3994bd8da4d

    Closes-bug: #2023634
    Change-Id: I0ac92e3692fb1ba9667fee2f6fbf531a57b7b5b8
    (cherry picked from commit 593278550a8185dfc52fc6c327ca1198df11e422)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/894265
Committed: https://opendev.org/openstack/neutron/commit/40c2afc2dc89fe17fc150d11b25338445f64af20
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 40c2afc2dc89fe17fc150d11b25338445f64af20
Author: elajkat <email address hidden>
Date: Wed Jun 14 15:15:06 2023 +0200

    Functional: assert multiple calls for update_virtual_port_host

    test_virtual_port_host_update assumed update_virtual_port_host
    to be called once, but the test started to fail when [1] was
    merged and OVN version was bumped to v22.03.3 (not yet released).
    PortBindingUpdateVirtualPortsEvent with event ROW_DELETE
    received twice in the failing case.

    Backport Note: It's required since [2] merged, ovn version deployed is
    not related to the failure.

    [1]: https://review.opendev.org/c/openstack/neutron/+/880890
    [2]: https://review.opendev.org/q/I34caf7d0212ccb4bd7259c4414e7c3994bd8da4d

    Closes-bug: #2023634
    Change-Id: I0ac92e3692fb1ba9667fee2f6fbf531a57b7b5b8
    (cherry picked from commit 593278550a8185dfc52fc6c327ca1198df11e422)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.2.0 release.