test_assert_pings_during_br_phys_setup_not_lost_in_vlan_to_flat functional tests randomly fails

[yatin]

Seen 4 failures in last 8 days, 3 in master, 1 in stable/2023.2
Fails as:-
ft1.18: neutron.tests.functional.agent.test_l2_ovs_agent.TestOVSAgent.test_assert_pings_during_br_phys_setup_not_lost_in_vlan_to_flattesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/test_l2_ovs_agent.py", line 270, in test_assert_pings_during_br_phys_setup_not_lost_in_vlan_to_flat
    self._test_assert_pings_during_br_phys_setup_not_lost(provider_net)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/test_l2_ovs_agent.py", line 305, in _test_assert_pings_during_br_phys_setup_not_lost
    with net_helpers.async_ping(ns_phys, [ip_int]) as done:
  File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 150, in async_ping
    f.result()
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 131, in assert_async_ping
    ns_ip_wrapper.netns.execute(
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 776, in execute
    return utils.execute(cmd, check_exit_code=check_exit_code,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py", line 157, in execute
    raise exceptions.ProcessExecutionError(msg,
neutron_lib.exceptions.ProcessExecutionError: Exit code: 1; Cmd: ['ip', 'netns', 'exec', 'test-0245f548-3f29-48f0-8d28-0cfe7e68dcc9', 'ping', '-W', 1, '-c', '1', '10.148.117.171']; Stdin: ; Stdout: PING 10.148.117.171 (10.148.117.171) 56(84) bytes of data.

--- 10.148.117.171 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

; Stderr:

Example failures:-
https://03c08251732a708a286f-1284cae0458f2bcc716f47c02bf394ec.ssl.cf2.rackcdn.com/openstack/ffb419b2456b44899920be36e9599cc5/testr_results.html
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a5c/openstack/a5ca7963eaf34ac3846c631e3b52c272/testr_results.html
https://d20c26a0475556dff012-de0170061a4c33caf35d62b404ee94ad.ssl.cf1.rackcdn.com/openstack/d17a470830d5471f8d6f309742a645fe/testr_results.html
https://b19b5ab35f9ced465dd5-2eb50734132c0e56282483bcdf57bf8a.ssl.cf5.rackcdn.com/openstack/8a6caec32ede46adbdcc6aa3b9e8e1c1/testr_results.html

Opensearch(creds: openstack/openstack):-
https://opensearch.logs.openstack.org/_dashboards/app/data-explorer/discover?security_tenant=global#?_a=(discover:(columns:!(_source),isDirty:!f,sort:!()),metadata:(indexPattern:'94869730-aea8-11ec-9e6a-83741af3fdcd',view:discover))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_q=(filters:!(),query:(language:kuery,query:'message:%22in%20test_assert_pings_during_br_phys_setup_not_lost_in_vlan_to_flat%22'))

[Lajos Katona (lajos-katona)]

Some history for it from Opensearch:
https://opensearch.logs.openstack.org/_dashboards/app/data-explorer/discover/?security_tenant=global#?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(discover:(columns:!(build_branch,build_status,build_name),interval:auto,sort:!()),metadata:(indexPattern:'94869730-aea8-11ec-9e6a-83741af3fdcd',view:discover))&_q=(filters:!(),query:(language:kuery,query:'message:%22line%20270,%20in%20test_assert_pings_during_br_phys_setup_not_lost_in_vlan_to_flat%22'))

[yatin (yatinkarel)]

recent failure
    https://6c5513d00fc4b6428bbc-5f206d1ac6837606572ca22039b53566.ssl.cf5.rackcdn.com/openstack/b416a8d4e831465582e8ce8d66c1be98/testr_results.html
    https://ff823c36eb3fb946bf74-3b6b124462defb6cc5627ee2cad4e887.ssl.cf1.rackcdn.com/openstack/dea4e53eeedb419d984f7d1310c2bfc4/testr_results.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/952209

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/952209
Committed: https://opendev.org/openstack/neutron/commit/7b2d968c277c6c68c6a04daaa58a5a792cac69a3
Submitter: "Zuul (22348)"
Branch: master

commit 7b2d968c277c6c68c6a04daaa58a5a792cac69a3
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Jun 10 09:16:02 2025 +0000

    [FT] Improve ``test_assert_pings_during_br_phys_setup_not_lost*`` tests

    The tests ``test_assert_pings_during_br_phys_setup_not_lost*`` check if
    the ping is possible between two ports created on physical networks
    (VLAN or flat). Now the method that creates the physical bridges on the
    OVS agent ``setup_physical_bridges`` is executed once.

    The loop to check the ping between the two ports uses a
    ``ThreadPoolExecutor``, thus it is possible to execute and check several
    pings at the same time.

    Closes-Bug: #2107775
    Change-Id: I14ec444a4aff3d09b0b8fff8d44f6964e8f7eb5d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2025.1
Review: https://review.opendev.org/c/openstack/neutron/+/952551

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.2
Review: https://review.opendev.org/c/openstack/neutron/+/952552

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/neutron/+/952553

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/952552
Committed: https://opendev.org/openstack/neutron/commit/2ac9bc4fe76e4bac34fcb2db984fc5bc13b43647
Submitter: "Zuul (22348)"
Branch: stable/2024.2

commit 2ac9bc4fe76e4bac34fcb2db984fc5bc13b43647
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Jun 10 09:16:02 2025 +0000

    [FT] Improve ``test_assert_pings_during_br_phys_setup_not_lost*`` tests

    The tests ``test_assert_pings_during_br_phys_setup_not_lost*`` check if
    the ping is possible between two ports created on physical networks
    (VLAN or flat). Now the method that creates the physical bridges on the
    OVS agent ``setup_physical_bridges`` is executed once.

    The loop to check the ping between the two ports uses a
    ``ThreadPoolExecutor``, thus it is possible to execute and check several
    pings at the same time.

    Closes-Bug: #2107775
    Change-Id: I14ec444a4aff3d09b0b8fff8d44f6964e8f7eb5d
    (cherry picked from commit 7b2d968c277c6c68c6a04daaa58a5a792cac69a3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/952553
Committed: https://opendev.org/openstack/neutron/commit/7b0a44c02b9d60211158149a9b04928fb8d81925
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit 7b0a44c02b9d60211158149a9b04928fb8d81925
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Jun 10 09:16:02 2025 +0000

    [FT] Improve ``test_assert_pings_during_br_phys_setup_not_lost*`` tests

    The tests ``test_assert_pings_during_br_phys_setup_not_lost*`` check if
    the ping is possible between two ports created on physical networks
    (VLAN or flat). Now the method that creates the physical bridges on the
    OVS agent ``setup_physical_bridges`` is executed once.

    The loop to check the ping between the two ports uses a
    ``ThreadPoolExecutor``, thus it is possible to execute and check several
    pings at the same time.

    Closes-Bug: #2107775
    Change-Id: I14ec444a4aff3d09b0b8fff8d44f6964e8f7eb5d
    (cherry picked from commit 7b2d968c277c6c68c6a04daaa58a5a792cac69a3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/952551
Committed: https://opendev.org/openstack/neutron/commit/b86123a73f5d364ac654a61a0dda1ed0d6eae07f
Submitter: "Zuul (22348)"
Branch: stable/2025.1

commit b86123a73f5d364ac654a61a0dda1ed0d6eae07f
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Jun 10 09:16:02 2025 +0000

    [FT] Improve ``test_assert_pings_during_br_phys_setup_not_lost*`` tests

    The tests ``test_assert_pings_during_br_phys_setup_not_lost*`` check if
    the ping is possible between two ports created on physical networks
    (VLAN or flat). Now the method that creates the physical bridges on the
    OVS agent ``setup_physical_bridges`` is executed once.

    The loop to check the ping between the two ports uses a
    ``ThreadPoolExecutor``, thus it is possible to execute and check several
    pings at the same time.

    Closes-Bug: #2107775
    Change-Id: I14ec444a4aff3d09b0b8fff8d44f6964e8f7eb5d
    (cherry picked from commit 7b2d968c277c6c68c6a04daaa58a5a792cac69a3)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 27.0.0.0b1 Flamingo development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 24.2.1 Caracal release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 25.2.1 Dalmatian release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 26.0.2 Epoxy release.