Service edit (scale out number of members) - cause exception

[Yossi Boaron]

****** The root cause of this bug is Octavia**********

A. In case of LB backend=Octavia, all the List+filter fields functions at Kuryr-controller LB driver (e.g: [1]) will fail on exception at Octavia code since current Octavia code doesn't support GET list + filters option.

B. There's an open patch at Octavia/master to address this case [2].

C. This bug @Octavia causes Kuryr's failures/exception (mainly in service edit use cases).
for example, the following use case:

kubectl run --image=celebdor/kuryr-demo kuryr-demo1
kubectl scale deploy/kuryr-demo1 --replicas=2
kubectl expose deploy/kuryr-demo1 --port 80 --target-port 8080

wait till service/Openstack LoadBlancer is Active and run:

kubectl scale deploy/kuryr-demo1 --replicas=4

Will cause the following [3] exception @Kuryr.

[1]https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/controller/drivers/lbaasv2.py#L250

[2] https://review.openstack.org/#/c/559842/

[3] Exception @Kuryr:

Apr 30 11:53:54 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging [-] Failed to handle event {u'object': {u'kind': u'Endpoints', u'subsets': [{u'addresses': [{u'ip': u
Apr 30 11:53:55 fgfgfgfg python[30687]: /namespaces/default/endpoints/kuryr-demo1', u'uid': u'01ca7489-491f-11e8-9789-fa163e0a8c14'}}, u'type': u'ADDED'}: InternalServerError: 500-{u'debuginfo': u'Traceback (mos
Apr 30 11:53:55 fgfgfgfg python[30687]: Neutron server returns request_ids: ['req-6e76e33e-ce38-48f2-8393-9734233ae3a0']
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging Traceback (most recent call last):
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/logging.py", line 37, in __call__
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/retry.py", line 55, in __call__
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging self._handler(event)
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/k8s_base.py", line 75, in __call__
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging self.on_present(obj)
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/handlers/lbaas.py", line 242, in on_p
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging if self._sync_lbaas_members(endpoints, lbaas_state, lbaas_spec):
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/handlers/lbaas.py", line 308, in _syn
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging if self._sync_lbaas_pools(endpoints, lbaas_state, lbaas_spec):
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/handlers/lbaas.py", line 418, in _syn
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging if self._sync_lbaas_listeners(endpoints, lbaas_state, lbaas_spec):
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/handlers/lbaas.py", line 475, in _syn
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging if self._sync_lbaas_loadbalancer(endpoints, lbaas_state, lbaas_spec):
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/handlers/lbaas.py", line 565, in _syn
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging service_type=lbaas_spec.type)
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/drivers/lbaasv2.py", line 50, in ensu
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging self._find_loadbalancer)
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/drivers/lbaasv2.py", line 392, in _en
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging result = find(obj)
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/drivers/lbaasv2.py", line 258, in _fi
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging vip_subnet_id=loadbalancer.subnet_id)
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1124, in list_loadbalanc
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging retrieve_all, **_params)
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging for r in self._pagination(collection, path, **params):
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging res = self.get(path, params=params)
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging headers=headers, params=params)
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging headers=headers, params=params)
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging self._handle_fault_response(status_code, replybody, resp)
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_re
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging exception_handler_v20(status_code, error_body)
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler
Apr 30 11:53:55 fgfgfgfg python[30687]: 2018-04-30 11:53:54.995 30687 ERROR kuryr_kubernetes.handlers.logging request_ids=request_ids)
lines 599-648/1

[Yossi Boaron (yossi-boaron-1234)]

A. It seems that running the use case described above doesn't lead to call of function [1].
which means no exception raised.

B. Anyhow, any use case that will lead to '_find_loadbalancer' function call --> will trigger an exception

[1]https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/controller/drivers/lbaasv2.py#L250

[Luis Tomas Bolivar (ltomasbo)]

Seems this patch fixed this problem, right? https://review.openstack.org/#/c/559842/

[Yossi Boaron (yossi-boaron-1234)]

@Luis - Yes - we can close this bug