[fwaas] test_update_firewall_group fails randomly

[yatin]

Seen twice till now recently:-
- https://a78793e982809689fe25-25fa16d377ec97c08c4e6ce3af683bd9.ssl.cf5.rackcdn.com/881232/1/check/neutron-tempest-plugin-fwaas/b0730f9/testr_results.html
- https://53a7c53d508ecea7485c-f8ccc2b7c32dd8ba5caab7dc1c36a741.ssl.cf5.rackcdn.com/881232/1/gate/neutron-tempest-plugin-fwaas/5712826/testr_results.html

Fails as below:-
traceback-1: {{{
Traceback (most recent call last):
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/api/test_fwaasv2_extensions.py", line 130, in _try_delete_firewall_group
    self.firewall_groups_client.delete_firewall_group(fwg_id)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 38, in delete_firewall_group
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 339, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 867, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'FirewallGroupInUse', 'message': 'Firewall group 57266ed6-c39c-4be2-80d8-649469adf7eb is still active.', 'detail': ''}
}}}

traceback-2: {{{
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc
    return func(*args, **kwargs)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 100, in delete_firewall_policy
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 339, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 867, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'FirewallPolicyInUse', 'message': 'Firewall policy 0e23c50e-28a9-41e5-829c-9a67d058bafd is being used.', 'detail': ''}
}}}

traceback-3: {{{
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc
    return func(*args, **kwargs)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 100, in delete_firewall_policy
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 339, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 867, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'FirewallPolicyInUse', 'message': 'Firewall policy a47d0031-65cb-40bf-86e1-ba3842c295aa is being used.', 'detail': ''}
}}}

traceback-4: {{{
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc
    return func(*args, **kwargs)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 75, in delete_firewall_rule
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 339, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 867, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'FirewallRuleInUse', 'message': 'Firewall rule a1bcea8b-0c2d-4d7c-88de-cf8ea2e5ef13 is being used.', 'detail': ''}
}}}

traceback-5: {{{
Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/lib/common/utils/test_utils.py", line 87, in call_and_ignore_notfound_exc
    return func(*args, **kwargs)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 75, in delete_firewall_rule
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 339, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 867, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'FirewallRuleInUse', 'message': 'Firewall rule 8941cc56-fc60-45f8-8ee0-0008422faa1f is being used.', 'detail': ''}
}}}

Traceback (most recent call last):
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/api/test_fwaasv2_extensions.py", line 341, in test_update_firewall_group
    self.firewall_groups_client.delete_firewall_group(fwg_id)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.10/site-packages/neutron_tempest_plugin/fwaas/services/v2_client.py", line 38, in delete_firewall_group
    return self.delete_resource(uri)
  File "/opt/stack/tempest/tempest/lib/services/network/base.py", line 42, in delete_resource
    resp, body = self.delete(req_uri)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 339, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 742, in request
    self._error_checker(resp, resp_body)
  File "/opt/stack/tempest/tempest/lib/common/rest_client.py", line 867, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'FirewallGroupInUse', 'message': 'Firewall group 57266ed6-c39c-4be2-80d8-649469adf7eb is still active.', 'detail': ''}

[Mamatisa Nurmatov (isabek)]

The latest builds is "success" - https://zuul.opendev.org/t/openstack/builds?job_name=neutron-tempest-plugin-fwaas

[Lajos Katona (lajos-katona)]

Opensearch link:
https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(_source),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22Details:%20%7B!'type!':%20!'FirewallGroupInUse!',%20!'message!':%20!'Firewall%22'),sort:!())

Example log lines (from https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_be4/882976/1/check/neutron-tempest-plugin-fwaas/be46ecc/controller/logs/screen-q-svc.txt )

.....
May 18 04:01:12.133868 np0034062031 neutron-server[57824]: DEBUG neutron_fwaas.services.firewall.fwaas_plugin_v2 [None req-c51659a0-ca9e-41fa-a422-4d3cc367d06b tempest-FWaaSv2ExtensionTestJSON-640939317 tempest-FWaaSv2ExtensionTestJSON-640939317-project-member] neutron_fwaas.services.firewall.fwaas_plugin_v2.FirewallPluginV2 method get_firewall_rule called with arguments (<neutron_lib.context.Context object at 0x7fe339b02e30>, '544802fd-bcf2-49f2-be97-74e5df68301a') {'fields': None} {{(pid=57824) wrapper /usr/local/lib/python3.10/dist-packages/oslo_log/helpers.py:65}}
May 18 04:01:12.141114 np0034062031 neutron-server[57824]: DEBUG neutron_fwaas.services.firewall.fwaas_plugin_v2 [None req-c51659a0-ca9e-41fa-a422-4d3cc367d06b tempest-FWaaSv2ExtensionTestJSON-640939317 tempest-FWaaSv2ExtensionTestJSON-640939317-project-member] neutron_fwaas.services.firewall.fwaas_plugin_v2.FirewallPluginV2 method delete_firewall_rule called with arguments (<neutron_lib.context.Context object at 0x7fe339b02e30>, '544802fd-bcf2-49f2-be97-74e5df68301a') {} {{(pid=57824) wrapper /usr/local/lib/python3.10/dist-packages/oslo_log/helpers.py:65}}
May 18 04:01:12.149207 np0034062031 neutron-server[57824]: INFO neutron.api.v2.resource [None req-c51659a0-ca9e-41fa-a422-4d3cc367d06b tempest-FWaaSv2ExtensionTestJSON-640939317 tempest-FWaaSv2ExtensionTestJSON-640939317-project-member] delete failed (client error): There was a conflict when trying to complete your request.
May 18 04:01:12.150059 np0034062031 neutron-server[57824]: INFO neutron.wsgi [None req-c51659a0-ca9e-41fa-a422-4d3cc367d06b tempest-FWaaSv2ExtensionTestJSON-640939317 tempest-FWaaSv2ExtensionTestJSON-640939317-project-member] 10.208.224.99,10.208.224.99 "DELETE /networking/v2.0/fwaas/firewall_rules/544802fd-bcf2-49f2-be97-74e5df68301a HTTP/1.1" status: 409 len: 346 time: 0.0190337
....

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/883826

[ZhouHeng (zhouhenglc)]

In test_update_firewall_group testing, the associated ports of the firewall will be updated, and after updating the associated ports, the firewall group will be deleted. At this time, the status of the firewall group may be ACTIVE or PENDING_UPDATE. The current code of neutron-fwaas is in an active state and cannot be deleted, PENDING_UPDATE state allows deletion. That's why the probability of deletion failure occurs.
We should update the testing and delete all associated ports before deleting the firewall group. We should also update the neutron-fwaas code. If there are associated ports and the status is PENDING, deletion is not allowed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-fwaas/+/884333

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/883826
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/7b1f9b11e66b81eabdc8361c89f8ccfdb3448f9a
Submitter: "Zuul (22348)"
Branch: master

commit 7b1f9b11e66b81eabdc8361c89f8ccfdb3448f9a
Author: zhouhenglc <email address hidden>
Date: Thu May 25 08:42:36 2023 +0800

    [fwaas]Disassociate all ports before deleting firewall group

    If the firewall group has associated ports, the firewall group may be in
    an active state. Firewall group is active, deletion is not allowed.
    This patch cannot solve the bug[1] at the moment. Because there are
    other bugs[2] present.
    We plan to avoid bugs[2] through patches[3] now. To solve the bug[1].
    There are other patches[1] here to try and solve the bug[2].

    Partial-Bug: #2018967

    [1] https://bugs.launchpad.net/neutron/+bug/2018967
    [2] https://bugs.launchpad.net/neutron/+bug/2021457
    [3] https://review.opendev.org/c/openstack/neutron-fwaas/+/884333
    [4] https://review.opendev.org/c/openstack/neutron-fwaas/+/884335

    Change-Id: Id328446be8591fac3affe2cf8adeaeee09697085

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-fwaas/+/884333
Committed: https://opendev.org/openstack/neutron-fwaas/commit/5b56eaf3b082fddaf219a8a6d7af53ba4840480c
Submitter: "Zuul (22348)"
Branch: master

commit 5b56eaf3b082fddaf219a8a6d7af53ba4840480c
Author: zhouhenglc <email address hidden>
Date: Thu May 25 16:06:07 2023 +0800

    Firewall group associated with ports is not allowed to be deleted

    Currently, we determine that the firewall group is in use based on
    its ACTIVE status. But the firewall group may have just updated
    the port and is currently PENDING_UPDATE status, deletion should
    not be allowed at this time.
    This patch changes the judgment method for deleting firewall
    groups, no longer based on their status. But like other neutron
    resources, based on whether or not they are associated.

    Closes-Bug: #2018967
    Depends-On: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/883826

    Change-Id: Ib7ab0daf9f6de45125ffc9408f865fc0964ff339

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-fwaas 19.0.0.0rc1 release candidate.