kuryr-kubernetes

Neutron internal error when activating vif or adding subports to trunk crashes kuryr-controller

Bug #1848012 reported by Luis Tomas Bolivar on 2019-10-14

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	kuryr-kubernetes	Fix Released	Undecided	Luis Tomas Bolivar

Bug Description

Neutron errors when trying to check if a port is active or when trying to add new subports to a trunk may crash kuryr-controller, while they should simply be retried:

Activation error:
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last): │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/logging.py", line 37, in __call__ │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event) │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/retry.py", line 56, in __call__ │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging self._handler(event) │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/handlers/k8s_base.py", line 72, in __call__ │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging self.on_present(obj) │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/handlers/vif.py", line 148, in on_present │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging self._drv_vif_pool.activate_vif(pod, vif) │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 1052, in activate_vif │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging self._vif_drvs[vif_drv_alias].activate_vif(pod, vif) │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/vif_pool.py", line 169, in activate_vif │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging self._drv_vif.activate_vif(pod, vif) │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/neutron_vif.py", line 90, in activate_vif │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging port = neutron.show_port(vif.id).get('port') │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 799, in show_port │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging return self.get(self.port_path % (port), params=_params) │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging headers=headers, params=params) │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging headers=headers, params=params) │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 282, in do_request │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging headers=headers) │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 340, in do_request │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging return self.request(url, method, **kwargs) │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 328, in request │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging resp = super(SessionClient, self).request(*args, **kwargs) │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 248, in request │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging return self.session.request(url, method, **kwargs) │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 888, in request │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging resp = send(**kwargs) │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 995, in _send_request │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging raise exceptions.ConnectFailure(msg) │
2019-10-14 10:44:01.816 1 ERROR kuryr_kubernetes.handlers.logging ConnectFailure: Unable to establish connection to http://10.0.0.112:9696/v2.0/ports/811223fc-1a49-4d14-9e6f-3bf53e580990: ('Connection aborted.',│
BadStatusLine("''",))

Sub2019-10-14 11:31:42.882 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif [-] Error happened during subport addition to trunk: InternalServerError: Request Failed: internal server error while processin│g your request.
│2019-10-14 11:31:42.882 1 ERROR
kuryr_kubernetes.controller.drivers.nested_vlan_vif Traceback (most recent call last): │2019-10-14 11:31:42.882 1 ERROR kuryr_kubernetes.controller.drivers.nested_vlan_vif File "/usr/lib/python2.7/site-packages/kuryr_kubernetes/controller/drivers/nested_vlan_vif.py", line 96, in request_vifs │2019-10-14 11:31:42.882 1 ERROR
kuryr_kubernetes.controller.drivers.nested_vlan_vif {'sub_ports': subports_info}) │2019-10-14 11:31:42.882 1 ERROR
kuryr_kubernetes.controller.drivers.nested_vlan_vif File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 2132, in trunk_add_subports │2019-10-14 11:31:42.882 1 ERROR

kuryr_kubernetes.controller.drivers.nested_vlan_vif return self.put(self.subports_add_path % (trunk), body=body) 2019-10-14 11:31:42.882 1 ERROR
kuryr_kubernetes.controller.drivers.nested_vlan_vif File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 363, in put 2019-10-14 11:31:42.882 1 ERROR
kuryr_kubernetes.controller.drivers.nested_vlan_vif headers=headers, params=params) 2019-10-14 11:31:42.882 1 ERROR
kuryr_kubernetes.controller.drivers.nested_vlan_vif File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request 2019-10-14 11:31:42.882 1 ERROR
kuryr_kubernetes.controller.drivers.nested_vlan_vif headers=headers, params=params) 2019-10-14 11:31:42.882 1 ERROR
kuryr_kubernetes.controller.drivers.nested_vlan_vif File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request 2019-10-14 11:31:42.882 1 ERROR
kuryr_kubernetes.controller.drivers.nested_vlan_vif self._handle_fault_response(status_code, replybody, resp) 2019-10-14 11:31:42.882 1 ERROR
kuryr_kubernetes.controller.drivers.nested_vlan_vif File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response 2019-10-14 11:31:42.882 1 ERROR
kuryr_kubernetes.controller.drivers.nested_vlan_vif exception_handler_v20(status_code, error_body) 2019-10-14 11:31:42.882 1 ERROR
kuryr_kubernetes.controller.drivers.nested_vlan_vif File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20 2019-10-14 11:31:42.882 1 ERROR
kuryr_kubernetes.controller.drivers.nested_vlan_vif request_ids=request_ids) 2019-10-14 11:31:42.882 1 ERROR
kuryr_kubernetes.controller.drivers.nested_vlan_vif InternalServerError: Request Failed: internal server error while processing your request. 2019-10-14 11:31:42.882 1 ERROR

kuryr_kubernetes.controller.drivers.nested_vlan_vif Neutron server returns request_ids: ['req-4e9c19dc-4533-4ddf-b5c5-91d8f2a1e28c'] port addition error:

Tags:

Luis Tomas Bolivar (ltomasbo) on 2019-10-14

Changed in kuryr-kubernetes:
assignee:	nobody → Luis Tomas Bolivar (ltomasbo)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-14: Fix proposed to kuryr-kubernetes (master)

Fix proposed to branch: master
Review: https://review.opendev.org/688410

Changed in kuryr-kubernetes:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-16: Fix merged to kuryr-kubernetes (master)

Reviewed: https://review.opendev.org/688410
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=9cd4ad3b14a4e6e59b9178743d04f6d22ebb5eb8
Submitter: Zuul
Branch: master

commit 9cd4ad3b14a4e6e59b9178743d04f6d22ebb5eb8
Author: Luis Tomas Bolivar <email address hidden>
Date: Mon Oct 14 16:22:17 2019 +0200

Avoid controller crash upon unexpected neutron error handling ports

    This PS ensures that kuryr-controller does not crash if neutron
    cannot be reached to check the port status, or if neutron replies
    with unexpected error when trying to add ports to the trunks

Change-Id: I037a882e1231118b3e85abace8cc13c4e79ee6da
Closes-Bug: 1848012

Changed in kuryr-kubernetes:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-17: Fix proposed to kuryr-kubernetes (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/689084

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-24: Fix merged to kuryr-kubernetes (stable/train)

Reviewed: https://review.opendev.org/689084
Committed: https://git.openstack.org/cgit/openstack/kuryr-kubernetes/commit/?id=ef19d84bebb27de51df48463bcc13ecf5d0afb24
Submitter: Zuul
Branch: stable/train

commit ef19d84bebb27de51df48463bcc13ecf5d0afb24
Author: Luis Tomas Bolivar <email address hidden>
Date: Mon Oct 14 16:22:17 2019 +0200

Avoid controller crash upon unexpected neutron error handling ports

    This PS ensures that kuryr-controller does not crash if neutron
    cannot be reached to check the port status, or if neutron replies
    with unexpected error when trying to add ports to the trunks

    Change-Id: I037a882e1231118b3e85abace8cc13c4e79ee6da
    Closes-Bug: 1848012
    (cherry picked from commit 9cd4ad3b14a4e6e59b9178743d04f6d22ebb5eb8)

tags:

added: in-stable-train

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.