namespace deletion failures upon kuryr-controller restarts

[Luis Tomas Bolivar]

Seems kuryr-controller restarts can lead to process namespace deletion event twice, leading to the next error as the kuryrnet CRD object is already deleting:
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {u'object': {u'status': {u'phase': u'Terminating', u'conditions': [{u'status': u'False', u'lastTransitionTime': u'2019-10-08T15:58:21Z', u'reason': u'ResourcesDiscovered', u'type': u'NamespaceDeletionDiscoveryFailure', u'message': u'All resources successfully discovered'}, {u'status': u'False', u'lastTransitionTime': u'2019-10-08T15:45:51Z', u'reason': u'ParsedGroupVersions', u'type': u'NamespaceDeletionGroupVersionParsingFailure', u'message': u'All legacy kube types successfully parsed'}, {u'status': u'False', u'lastTransitionTime': u'2019-10-08T15:45:51Z', u'reason': u'ContentDeleted', u'type': u'NamespaceDeletionContentF
ailure', u'message': u'All content successfully deleted, may be waiting on finalization'}, {u'status': u'False', u'lastTransitionTime': u'2019-10-08T15:45:51Z', u'reason': u'Content
Removed', u'type': u'NamespaceContentRemaining', u'message': u'All content successfully removed'}, {u'status': u'False', u'lastTransitionTime': u'2019-10-08T15:45:51Z', u'reason': u
'ContentHasNoFinalizers', u'type': u'NamespaceFinalizersRemaining', u'message': u'All content-preserving finalizers finished'}]}, u'kind': u'Namespace', u'spec': {}, u'apiVersion':
u'v1', u'metadata': {u'name': u'e2e-events-7637', u'deletionTimestamp': u'2019-10-08T15:45:26Z', u'labels': {u'e2e-framework': u'events', u'e2e-run': u'705b6fe8-e9e2-11e9-ba63-80184
4f3e3fc'}, u'resourceVersion': u'57205', u'creationTimestamp': u'2019-10-08T15:44:36Z', u'annotations': {u'openshift.io/sa.scc.supplemental-groups': u'1000550000/10000', u'openshift
.io/sa.scc.mcs': u's0:c23,c22', u'openstack.org/kuryr-namespace-label': u'{"e2e-framework": "events", "e2e-run": "705b6fe8-e9e2-11e9-ba63-801844f3e3fc"}', u'openstack.org/kuryr-net-
crd': u'ns-e2e-events-7637', u'openshift.io/node-selector': u'', u'openshift.io/sa.scc.uid-range': u'1000550000/10000'}, u'selfLink': u'/api/v1/namespaces/e2e-events-7637', u'uid':
u'88493014-e9e2-11e9-850f-fa163ea158fc'}}, u'type': u'DELETED'}: TypeError: 'NoneType' object has no attribute '__getitem__'
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/retry.py", line 56, in __call__
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 77, in __call__
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging self.on_deleted(obj)
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/namespace.py", line 143, in on_delete
d
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging self._drv_vif_pool.delete_network_pools(net_crd['spec']['netId'])
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging TypeError: 'NoneType' object has no attribute '__getitem__'
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/687520

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/687520
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=efae1f5211e3907ff58853155b785f0e8fb34e9b
Submitter: Zuul
Branch: master

commit efae1f5211e3907ff58853155b785f0e8fb34e9b
Author: Luis Tomas Bolivar <email address hidden>
Date: Wed Oct 9 11:41:37 2019 +0200

    Avoid namespace deletion error if processing a duplicated event

    In case of a kuryr-controller restart, if the namespace deletion
    event was already executed and it is retriggered, it can make the
    kuryr-controller fail as the kuryrnet CRD would have been already
    deleted and we cannot obtain the subnet id from it. In that case
    the event just need to be skip.

    Change-Id: I14fcf820d94319541dd7bf13893e82cf0373969c
    Closes-Bug: 1847453

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/687872

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/687872
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=7444eab6494af92dc75ecb5c47e8f1f0299fbf61
Submitter: Zuul
Branch: stable/train

commit 7444eab6494af92dc75ecb5c47e8f1f0299fbf61
Author: Luis Tomas Bolivar <email address hidden>
Date: Wed Oct 9 11:41:37 2019 +0200

    Avoid namespace deletion error if processing a duplicated event

    In case of a kuryr-controller restart, if the namespace deletion
    event was already executed and it is retriggered, it can make the
    kuryr-controller fail as the kuryrnet CRD would have been already
    deleted and we cannot obtain the subnet id from it. In that case
    the event just need to be skip.

    Change-Id: I14fcf820d94319541dd7bf13893e82cf0373969c
    Closes-Bug: 1847453
    (cherry picked from commit efae1f5211e3907ff58853155b785f0e8fb34e9b)