namespace deletion failures upon kuryr-controller restarts

Bug #1847453 reported by Luis Tomas Bolivar on 2019-10-09
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kuryr-kubernetes
Undecided
Luis Tomas Bolivar

Bug Description

Seems kuryr-controller restarts can lead to process namespace deletion event twice, leading to the next error as the kuryrnet CRD object is already deleting:
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {u'object': {u'status': {u'phase': u'Terminating', u'conditions': [{u'status': u'False', u'lastTransitionTime': u'2019-10-08T15:58:21Z', u'reason': u'ResourcesDiscovered', u'type': u'NamespaceDeletionDiscoveryFailure', u'message': u'All resources successfully discovered'}, {u'status': u'False', u'lastTransitionTime': u'2019-10-08T15:45:51Z', u'reason': u'ParsedGroupVersions', u'type': u'NamespaceDeletionGroupVersionParsingFailure', u'message': u'All legacy kube types successfully parsed'}, {u'status': u'False', u'lastTransitionTime': u'2019-10-08T15:45:51Z', u'reason': u'ContentDeleted', u'type': u'NamespaceDeletionContentF
ailure', u'message': u'All content successfully deleted, may be waiting on finalization'}, {u'status': u'False', u'lastTransitionTime': u'2019-10-08T15:45:51Z', u'reason': u'Content
Removed', u'type': u'NamespaceContentRemaining', u'message': u'All content successfully removed'}, {u'status': u'False', u'lastTransitionTime': u'2019-10-08T15:45:51Z', u'reason': u
'ContentHasNoFinalizers', u'type': u'NamespaceFinalizersRemaining', u'message': u'All content-preserving finalizers finished'}]}, u'kind': u'Namespace', u'spec': {}, u'apiVersion':
u'v1', u'metadata': {u'name': u'e2e-events-7637', u'deletionTimestamp': u'2019-10-08T15:45:26Z', u'labels': {u'e2e-framework': u'events', u'e2e-run': u'705b6fe8-e9e2-11e9-ba63-80184
4f3e3fc'}, u'resourceVersion': u'57205', u'creationTimestamp': u'2019-10-08T15:44:36Z', u'annotations': {u'openshift.io/sa.scc.supplemental-groups': u'1000550000/10000', u'openshift
.io/sa.scc.mcs': u's0:c23,c22', u'openstack.org/kuryr-namespace-label': u'{"e2e-framework": "events", "e2e-run": "705b6fe8-e9e2-11e9-ba63-801844f3e3fc"}', u'openstack.org/kuryr-net-
crd': u'ns-e2e-events-7637', u'openshift.io/node-selector': u'', u'openshift.io/sa.scc.uid-range': u'1000550000/10000'}, u'selfLink': u'/api/v1/namespaces/e2e-events-7637', u'uid':
u'88493014-e9e2-11e9-850f-fa163ea158fc'}}, u'type': u'DELETED'}: TypeError: 'NoneType' object has no attribute '__getitem__'
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/retry.py", line 56, in __call__
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 77, in __call__
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging self.on_deleted(obj)
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/namespace.py", line 143, in on_delete
d
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging self._drv_vif_pool.delete_network_pools(net_crd['spec']['netId'])
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging TypeError: 'NoneType' object has no attribute '__getitem__'
2019-10-08 15:58:22.764 1 ERROR kuryr_kubernetes.handlers.logging

Changed in kuryr-kubernetes:
assignee: nobody → Luis Tomas Bolivar (ltomasbo)

Fix proposed to branch: master
Review: https://review.opendev.org/687520

Changed in kuryr-kubernetes:
status: New → In Progress

Reviewed: https://review.opendev.org/687520
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=efae1f5211e3907ff58853155b785f0e8fb34e9b
Submitter: Zuul
Branch: master

commit efae1f5211e3907ff58853155b785f0e8fb34e9b
Author: Luis Tomas Bolivar <email address hidden>
Date: Wed Oct 9 11:41:37 2019 +0200

    Avoid namespace deletion error if processing a duplicated event

    In case of a kuryr-controller restart, if the namespace deletion
    event was already executed and it is retriggered, it can make the
    kuryr-controller fail as the kuryrnet CRD would have been already
    deleted and we cannot obtain the subnet id from it. In that case
    the event just need to be skip.

    Change-Id: I14fcf820d94319541dd7bf13893e82cf0373969c
    Closes-Bug: 1847453

Changed in kuryr-kubernetes:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/687872
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=7444eab6494af92dc75ecb5c47e8f1f0299fbf61
Submitter: Zuul
Branch: stable/train

commit 7444eab6494af92dc75ecb5c47e8f1f0299fbf61
Author: Luis Tomas Bolivar <email address hidden>
Date: Wed Oct 9 11:41:37 2019 +0200

    Avoid namespace deletion error if processing a duplicated event

    In case of a kuryr-controller restart, if the namespace deletion
    event was already executed and it is retriggered, it can make the
    kuryr-controller fail as the kuryrnet CRD would have been already
    deleted and we cannot obtain the subnet id from it. In that case
    the event just need to be skip.

    Change-Id: I14fcf820d94319541dd7bf13893e82cf0373969c
    Closes-Bug: 1847453
    (cherry picked from commit efae1f5211e3907ff58853155b785f0e8fb34e9b)

tags: added: in-stable-train
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers