[fwaas] Scenario job fails randomly

[yatin]

Seen couple of occurrences across different releases:-

Failures:-
- https://zuul.opendev.org/t/openstack/build/30c625cd86aa40e6b6252689a7e88910 neutron-tempest-plugin-fwaas-2023-1
Traceback (most recent call last):
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/api/test_fwaasv2_extensions.py", line 269, in test_create_show_delete_firewall_group
    body = self.firewall_groups_client.create_firewall_group(
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 25, in create_firewall_group
    return self.create_resource(uri, post_data)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 62, in create_resource
    resp, body = self.post(req_uri, req_post_data)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 300, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 922, in _error_checker
    raise exceptions.ServerFault(resp_body, resp=resp,
tempest.lib.exceptions.ServerFault: Got server fault
Details: Request Failed: internal server error while processing your request.

- https://zuul.opendev.org/t/openstack/build/0d0fbfc009cb4142920ddb96e9695ec0 neutron-tempest-plugin-fwaas-2023-2

Traceback (most recent call last):
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/scenario/test_fwaas_v2.py", line 220, in test_icmp_reachability_scenarios
    self.check_vm_connectivity(
  File "/opt/stack/tempest/tempest/scenario/manager.py", line 983, in check_vm_connectivity
    self.assertTrue(self.ping_ip_address(ip_address,
  File "/usr/lib/python3.10/unittest/case.py", line 687, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true : Timed out waiting for 172.24.5.235 to become reachable

- https://zuul.opendev.org/t/openstack/build/5ed8731220654e9fb67ba910a3f08c25 neutron-tempest-plugin-fwaas-zed
Traceback (most recent call last):
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/fwaas/api/test_fwaasv2_extensions.py", line 344, in test_update_firewall_group
    self.firewall_groups_client.delete_firewall_group(fwg_id)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 38, in delete_firewall_group
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 339, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 867, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'FirewallGroupInUse', 'message': 'Firewall group a20ead7e-40ca-488b-a751-a36e5fb4119a is still active.', 'detail': ''}

- https://zuul.opendev.org/t/openstack/build/fbe68294562a43b492dd2ba66dec9d43 neutron-tempest-plugin-fwaas-2023-2
Traceback (most recent call last):
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/scenario/test_fwaas_v2.py", line 220, in test_icmp_reachability_scenarios
    self.check_vm_connectivity(
  File "/opt/stack/tempest/tempest/scenario/manager.py", line 983, in check_vm_connectivity
    self.assertTrue(self.ping_ip_address(ip_address,
  File "/usr/lib/python3.10/unittest/case.py", line 687, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true : Timed out waiting for 172.24.5.126 to become reachable

[Lajos Katona (lajos-katona)]

I see another issue, with the msg:
The specified regex doesn't match with anythingERROR: InvocationError for command /opt/stack/tempest/.tox/tempest/bin/tempest run --regex '^neutron_tempest_plugin\.fwaas' --concurrency=4 (exited with code 1)

Examples:

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bfd/897792/3/check/neutron-fwaas-v2-dsvm-tempest-multinode/bfd1bcd/job-output.txt

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_533/896741/5/check/neutron-fwaas-v2-dsvm-tempest-multinode/533daa0/job-output.txt

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Patch: https://review.opendev.org/c/openstack/neutron-fwaas/+/898231