K8S service resource creation - fails

[Yossi Boaron]

Steps to reproduce :
--------------------

1. Use the following file as a basic reference for Devstack local.conf :
 https://github.com/openstack/kuryr-kubernetes/blob/master/devstack/local.conf.sample

2. Disable Octavia in local.conf
Change the following from 'True' to 'False' -
https://github.com/openstack/kuryr-kubernetes/blob/master/devstack/local.conf.sample#L41

3. Deploy Devstack

4. Create k8s-service as follows:

  kubectl run --image=celebdor/kuryr-demo kuryr-demo666
  kubectl scale deploy/kuryr-demo666 --replicas=2
  kubectl expose deploy/kuryr-demo666 --port 80 --target-port 8080

Observations:
--------------

1. Load Balancer wasn't created in OpenStack

[stack@rrrrr devstack]$ neutron lbaas-loadbalancer-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
+--------------------------------------+--------------------+----------------------------------+-------------+---------------------+----------+
| id | name | tenant_id | vip_address | provisioning_status | provider |
+--------------------------------------+--------------------+----------------------------------+-------------+---------------------+----------+
| 5e26133d-e3e4-43bf-b09d-df9196a03e84 | default/kubernetes | b3d386c7f756489a8881464c5d7b8e01 | 10.0.0.129 | ACTIVE | haproxy |
+--------------------------------------+--------------------+----------------------------------+-------------+---------------------+----------+
[stack@rrrrr devstack]$

2. From Kuryr-controller logs :

"
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.092 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.092 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.093 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.093 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.093 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.094 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.094 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.095 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.095 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.095 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.096 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.096 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.096 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.097 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.097 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.097 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.098 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.098 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.098 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.099 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.099 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.099 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.100 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.100 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.100 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.101 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.101 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.101 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.102 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.102 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.102 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.103 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.103 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/han
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.103 14844 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.151,lb_ip=None,ports=[LBaa
Feb 20 08:48:47 rrrrr python[14844]: 2018-02-20 08:48:47.104 14844 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthy LoadBalancerHandler __call__
Kuryr didn't to create Loadbalancer in Open "

[Yossi Boaron (yossi-boaron-1234)]

More logs - (of customized version - added aleep and debug prints) - the service name demo12345
---------------------------

Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.373 20955 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/dispatch.py:6
Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.374 20955 DEBUG dogpile.lock [-] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/lock.py:90
Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.374 20955 DEBUG dogpile.lock [-] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/lock.py:123
Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.374 20955 DEBUG dogpile.lock [-] value creation lock <dogpile.cache.backends.null.NullLock object at 0x3c0d310> acquired _enter_create /usr/lib/python2.7/
Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.374 20955 DEBUG dogpile.lock [-] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/lock.py:144
Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.615 20955 DEBUG kuryr_kubernetes.handlers.asynchronous [-] Asynchronous handler started processing /api/v1/namespaces/default/pods/kuryr-demo12345-6d8c5d7
Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.752 20955 DEBUG dogpile.lock [-] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/lock.py:150
Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.811 20955 DEBUG kuryr_kubernetes.handlers.asynchronous [-] Asynchronous handler started processing /api/v1/namespaces/default/services/kuryr-demo12345 _ru
Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.812 20955 DEBUG kuryr_kubernetes.handlers.asynchronous [-] Asynchronous handler started processing /api/v1/namespaces/default/endpoints/kuryr-demo12345 _r
Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.311 20955 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/dispatch.py:6
Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.311 20955 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] LBaaS spec is missing for /api/v1/namespaces/default/services/kuryr-demo12345 _has_ip_chang
Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.312 20955 DEBUG dogpile.lock [-] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/lock.py:90
Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.312 20955 DEBUG dogpile.lock [-] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/lock.py:123
Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.312 20955 DEBUG dogpile.lock [-] value creation lock <dogpile.cache.backends.null.NullLock object at 0x3c0de50> acquired _enter_create /usr/lib/python2.7/
Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.312 20955 DEBUG dogpile.lock [-] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/lock.py:144
Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.315 20955 DEBUG kuryr_kubernetes.handlers.di...

[Michal Dulko (michal-dulko-f)]

This got fixed with https://review.openstack.org/#/c/545270, am I right?