deletion of Network Policies enforced on same pod make controller restart

[Maysa de Macedo Souza]

When multiple Network policy deletion affecting the
same pod happens at the same time, a Not Found exception
can be raised making the controller restart.

2019-12-17 01:47:47.588 1 DEBUG kuryr_kubernetes.handlers.asynchronous [-] Asynchronous handler stopped processing group 05e19cdf-206e-11ea-9993-fa163e044615 _done /usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/asynchron
ous.py:102
2019-12-17 01:47:47.749 1 DEBUG kuryr_kubernetes.controller.drivers.network_policy [-] Deleting KuryrNetPolicy CRD np-allow-pod-a-to-pod-b-using-pod-selector _del_kuryrnetpolicy_crd /usr/local/lib/python3.6/site-packages/kuryr_kubernetes/
controller/drivers/network_policy.py:740
2019-12-17 01:47:47.749 1 DEBUG kuryr_kubernetes.k8s_client [-] Delete /apis/openstack.org/v1/namespaces/network-policy-9919/kuryrnetpolicies/np-allow-pod-a-to-pod-b-using-pod-selector delete /usr/local/lib/python3.6/site-packages/kuryr_k
ubernetes/k8s_client.py:185
2019-12-17 01:47:47.878 1 DEBUG kuryr_kubernetes.handlers.asynchronous [-] Asynchronous handler started processing f5f9040f-206e-11ea-9993-fa163e044615 _run /usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/asynchronous.py:
64
2019-12-17 01:47:47.906 1 DEBUG neutronclient.v2_0.client [-] Error message: {"NeutronError": {"message": "Security group 669aff8c-852b-4260-b04c-95ccbfb1a998 does not exist", "type": "SecurityGroupNotFound", "detail": ""}} _handle_fault_
response /usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py:259
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry [-] Report handler unhealthy NetworkPolicyHandler: neutronclient.common.exceptions.NotFound: Security group 669aff8c-852b-4260-b04c-95ccbfb1a998 does not exist
Neutron server returns request_ids: ['req-b90ea3f9-3ee3-46a8-b526-3fccce6494f8']
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry Traceback (most recent call last):
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry self._handler(event)
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 77, in __call__
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry self.on_deleted(obj)
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/policy.py", line 119, in on_deleted
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry self._drv_vif_pool.update_vif_sgs(pod, pod_sgs)
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 1131, in update_vif_sgs
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry self._vif_drvs[pod_vif_type].update_vif_sgs(pod, sgs)
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 173, in update_vif_sgs
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry self._drv_vif.update_vif_sgs(pod, sgs)
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/neutron_vif.py", line 112, in update_vif_sgs
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry security_groups)
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 808, in update_port
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry revision_number=revision_number)
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 2399, in _update_resource
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry return self.put(path, **kwargs)
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 363, in put
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry headers=headers, params=params)
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry headers=headers, params=params)
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry self._handle_fault_response(status_code, replybody, resp)
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry exception_handler_v20(status_code, error_body)
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry File "/usr/local/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry request_ids=request_ids)
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry neutronclient.common.exceptions.NotFound: Security group 669aff8c-852b-4260-b04c-95ccbfb1a998 does not exist
2019-12-17 01:47:47.908 1 ERROR kuryr_kubernetes.handlers.retry Neutron server returns request_ids: ['req-b90ea3f9-3ee3-46a8-b526-3fccce6494f8']

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/699415

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/699415
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=ed6e9c1384b315c6e44b136d65a000911626dd6d
Submitter: Zuul
Branch: master

commit ed6e9c1384b315c6e44b136d65a000911626dd6d
Author: Maysa Macedo <email address hidden>
Date: Tue Dec 17 13:38:13 2019 +0000

    Protect from sg Not Found on multiple np enforcement

    When multiple Network policies deletion affecting the
    same pod happens at the same time, a Not Found exception
    can be raised causing a controller restart. This commit
    fixes the issue by ensuring the policy deletion is retried
    and the sgs list is updated.

    Change-Id: I55a0f81ab6e30ca672e3f3d4865aed3cd96a94d2
    Closes-bug: 1856709

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/701391

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/701391
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=d2463afa17b9a7d3399cc005ea77d1296e62e812
Submitter: Zuul
Branch: stable/train

commit d2463afa17b9a7d3399cc005ea77d1296e62e812
Author: Maysa Macedo <email address hidden>
Date: Tue Dec 17 13:38:13 2019 +0000

    Protect from sg Not Found on multiple np enforcement

    When multiple Network policies deletion affecting the
    same pod happens at the same time, a Not Found exception
    can be raised causing a controller restart. This commit
    fixes the issue by ensuring the policy deletion is retried
    and the sgs list is updated.

    Change-Id: I55a0f81ab6e30ca672e3f3d4865aed3cd96a94d2
    Closes-bug: 1856709
    (cherry picked from commit ed6e9c1384b315c6e44b136d65a000911626dd6d)