[ovn-octavia-provider] functional jobs are unstable

[Takashi Kajinami]

Functional tests of ovn-octavia-providers are very unstable and fail quite often.
This has been caught in https://review.opendev.org/c/openstack/ovn-octavia-provider/+/907582 .

According to the job history -release is more unstable than -master

https://zuul.opendev.org/t/openstack/builds?job_name=ovn-octavia-provider-functional-release&project=openstack%2Fovn-octavia-provider&change=907582&skip=0

https://zuul.opendev.org/t/openstack/builds?job_name=ovn-octavia-provider-functional-master&project=openstack%2Fovn-octavia-provider&change=907582&skip=0

The failures seen there are quite random and I could not find any test case (or error) consistently failing.

[Lajos Katona (lajos-katona)]

There's really some filures:

https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(build_name,build_status),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22line%2059,%20in%20test_create_lb_custom_network%22'),sort:!())

https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(build_name,build_status),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22line%20236,%20in%20test_fip_on_lb_vip%22'),sort:!())

[Fernando Royo (froyoredhat)]

From first analysis I have identified three main exceptions on failing tests, all are triggered on interactions between ovn-provider and neutron creating or updating a port.

[1] https://paste.openstack.org/show/bYHJ8kawic245Nuz7cVO/
[2] https://paste.openstack.org/show/bt78nDQ7OE3qs4Bf2uPO/
[3] https://paste.openstack.org/show/bjyeeN7GgCKg9OmPgA8d/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/915266
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/8d3e5c7ed3064abac33d152ed74e2f0427d4e676
Submitter: "Zuul (22348)"
Branch: master

commit 8d3e5c7ed3064abac33d152ed74e2f0427d4e676
Author: Fernando Royo <email address hidden>
Date: Mon Apr 8 13:06:16 2024 +0200

    Adding isolation to functional tests

    Due to some events received among different tests in the
    same worker, this patch add '--isolated' tag to stestr
    to run every test on a unique worker and in a different stestr
    process.

    Closes-Bug: #2058400

    Change-Id: I9ff255c110774590f53811162572eef596ff9a04

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/917719