Runtime error due to changing the dictionary during iteration

[Luis Tomas Bolivar]

Error raised at remove_sg_from_pools due to py3 and changing the dictionary while iteration over it:

2019-12-24 10:53:40.073 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {'type': 'DELETED', 'object': {'kind': 'NetworkPolicy', 'apiVersion': 'networking.k8s.io/v1', 'metadata': {'name': 'allow-to-server-a-pod-selector', 'namespace': 'network-policy-9843', 'selfLink': '/apis/networking.k8s.io/v1/namespaces/network-policy-9843/networkpolicies/allow-to-server-a-pod-selector', 'uid': 'f1469d78-f392-4b56-a0e3-7f4bd3ab9603', 'resourceVersion': '50385', 'generation': 1, 'creationTimestamp': '2019-12-24T10:50:13Z', 'annotations': {'kuryrnetpolicy_selfLink': '/apis/openstack.org/v1/namespaces/network-policy-9843/kuryrnetpolicies/np-allow-to-server-a-pod-selector'}}, 'spec': {'podSelector': {'matchLabels': {'pod-name': 'client-a'}}, 'egress': [{'ports': [{'protocol': 'UDP', 'port': 53}, {'protocol': 'TCP', 'port': 53}]}, {'to': [{'podSelector': {'matchLabels': {'pod-name': 'server'}}}]}], 'policyTypes': ['Egress']}}}: RuntimeError: dictionary changed size during iteration
2019-12-24 10:53:40.073 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2019-12-24 10:53:40.073 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__
2019-12-24 10:53:40.073 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2019-12-24 10:53:40.073 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/retry.py", line 78, in __call__
2019-12-24 10:53:40.073 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2019-12-24 10:53:40.073 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 77, in __call__
2019-12-24 10:53:40.073 1 ERROR kuryr_kubernetes.handlers.logging self.on_deleted(obj)
2019-12-24 10:53:40.073 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/handlers/policy.py", line 123, in on_deleted
2019-12-24 10:53:40.073 1 ERROR kuryr_kubernetes.handlers.logging self._drv_vif_pool.remove_sg_from_pools(crd_sg, net_id)
2019-12-24 10:53:40.073 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 1079, in remove_sg_from_pools
2019-12-24 10:53:40.073 1 ERROR kuryr_kubernetes.handlers.logging vif_drv.remove_sg_from_pools(sg_id, net_id)
2019-12-24 10:53:40.073 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python3.6/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 309, in remove_sg_from_pools
2019-12-24 10:53:40.073 1 ERROR kuryr_kubernetes.handlers.logging for sg_key, ports in pool_ports.items():
2019-12-24 10:53:40.073 1 ERROR kuryr_kubernetes.handlers.logging RuntimeError: dictionary changed size during iteration
2019-12-24 10:53:40.073 1 ERROR kuryr_kubernetes.handlers.logging
2019-12-24 10:53:49.163 1 INFO werkzeug [-] 10.196.0.15 - - [24/Dec/2019 10:53:49] "GET /alive HTTP/1.1" 500 -

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/701705

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/701705
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=a538eae343001c67dc3f7d816f173734b6163f1f
Submitter: Zuul
Branch: master

commit a538eae343001c67dc3f7d816f173734b6163f1f
Author: Luis Tomas Bolivar <email address hidden>
Date: Thu Jan 9 11:52:47 2020 +0100

    Fix problem of changing size during dict iteration

    Change-Id: I8bfcd5d6b33ab1d759a9bf8066398021eddcdfa2
    Closes-Bug: 1859005

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/702014

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/702014
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=eeed574bb80cd4010fcf5945c642e3140b74d3b4
Submitter: Zuul
Branch: stable/train

commit eeed574bb80cd4010fcf5945c642e3140b74d3b4
Author: Luis Tomas Bolivar <email address hidden>
Date: Thu Jan 9 11:52:47 2020 +0100

    Fix problem of changing size during dict iteration

    Change-Id: I8bfcd5d6b33ab1d759a9bf8066398021eddcdfa2
    Closes-Bug: 1859005
    (cherry picked from commit a538eae343001c67dc3f7d816f173734b6163f1f)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/702841

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/702841
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=c7b8816c350fc3394044e76524153cad07215336
Submitter: Zuul
Branch: master

commit c7b8816c350fc3394044e76524153cad07215336
Author: Luis Tomas Bolivar <email address hidden>
Date: Thu Jan 16 11:57:47 2020 +0100

    Fix more problem of changing size during dict iteration

    Change-Id: Ie1d7fb2d7d62c1a43a32d513fdcbbb265bebff98
    Closes-Bug: 1859005

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/703602

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/703602
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=7f17c94476715e9d9b50d9d483deb20ab4412a2d
Submitter: Zuul
Branch: stable/train

commit 7f17c94476715e9d9b50d9d483deb20ab4412a2d
Author: Luis Tomas Bolivar <email address hidden>
Date: Thu Jan 16 11:57:47 2020 +0100

    Fix more problem of changing size during dict iteration

    Change-Id: Ie1d7fb2d7d62c1a43a32d513fdcbbb265bebff98
    Closes-Bug: 1859005
    (cherry picked from commit c7b8816c350fc3394044e76524153cad07215336)