More logs - (of customized version - added aleep and debug prints) - the service name demo12345 --------------------------- Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.373 20955 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/dispatch.py:6 Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.374 20955 DEBUG dogpile.lock [-] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/lock.py:90 Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.374 20955 DEBUG dogpile.lock [-] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/lock.py:123 Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.374 20955 DEBUG dogpile.lock [-] value creation lock acquired _enter_create /usr/lib/python2.7/ Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.374 20955 DEBUG dogpile.lock [-] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/lock.py:144 Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.615 20955 DEBUG kuryr_kubernetes.handlers.asynchronous [-] Asynchronous handler started processing /api/v1/namespaces/default/pods/kuryr-demo12345-6d8c5d7 Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.752 20955 DEBUG dogpile.lock [-] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/lock.py:150 Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.811 20955 DEBUG kuryr_kubernetes.handlers.asynchronous [-] Asynchronous handler started processing /api/v1/namespaces/default/services/kuryr-demo12345 _ru Feb 20 11:40:28 rrrrr python[20955]: 2018-02-20 11:40:28.812 20955 DEBUG kuryr_kubernetes.handlers.asynchronous [-] Asynchronous handler started processing /api/v1/namespaces/default/endpoints/kuryr-demo12345 _r Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.311 20955 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/dispatch.py:6 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.311 20955 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] LBaaS spec is missing for /api/v1/namespaces/default/services/kuryr-demo12345 _has_ip_chang Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.312 20955 DEBUG dogpile.lock [-] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/lock.py:90 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.312 20955 DEBUG dogpile.lock [-] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/lock.py:123 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.312 20955 DEBUG dogpile.lock [-] value creation lock acquired _enter_create /usr/lib/python2.7/ Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.312 20955 DEBUG dogpile.lock [-] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/lock.py:144 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.315 20955 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/dispatch.py:6 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.315 20955 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] YOBO - spec=None on_present /opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/handler Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.315 20955 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] YOBO - endpoints={u'kind': u'Endpoints', u'subsets': None, u'apiVersion': u'v1', u'metadata Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.458 20955 DEBUG dogpile.lock [-] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/lock.py:150 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.459 20955 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Setting LBaaSServiceSpec annotation: LBaaSServiceSpec(ip=10.0.0.130,lb_ip=None,ports=[LBaaS Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.459 20955 DEBUG kuryr_kubernetes.k8s_client [-] Annotate /api/v1/namespaces/default/endpoints/kuryr-demo12345: ['openstack.org/kuryr-lbaas-spec'] annotate Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.463 20955 DEBUG kuryr_kubernetes.k8s_client [-] Annotate /api/v1/namespaces/default/services/kuryr-demo12345: ['openstack.org/kuryr-lbaas-spec'] annotate Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.615 20955 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/dispatch.py:6 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.616 20955 DEBUG dogpile.lock [-] NeedRegenerationException _enter /usr/lib/python2.7/site-packages/dogpile/lock.py:90 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.616 20955 DEBUG dogpile.lock [-] no value, waiting for create lock _enter_create /usr/lib/python2.7/site-packages/dogpile/lock.py:123 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.616 20955 DEBUG dogpile.lock [-] value creation lock acquired _enter_create /usr/lib/python2.7/ Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.616 20955 DEBUG dogpile.lock [-] Calling creation function _enter_create /usr/lib/python2.7/site-packages/dogpile/lock.py:144 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.618 20955 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/dispatch.py:6 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.618 20955 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] YOBO - spec=None on_present /opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/handler Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.618 20955 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] YOBO - endpoints={u'kind': u'Endpoints', u'subsets': None, u'apiVersion': u'v1', u'metadata Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.715 20955 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/dispatch.py:6 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.716 20955 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] YOBO - spec=None on_present /opt/stack/kuryr-kubernetes/kuryr_kubernetes/controller/handler Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.716 20955 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] YOBO - endpoints={u'kind': u'Endpoints', u'subsets': None, u'apiVersion': u'v1', u'metadata Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.762 20955 DEBUG dogpile.lock [-] Released creation lock _enter_create /usr/lib/python2.7/site-packages/dogpile/lock.py:150 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.819 20955 DEBUG kuryr_kubernetes.controller.handlers.vif [-] Setting VIF annotation: VIFBridge(active=False,address=fa:16:3e:7b:9e:18,bridge_name='qbr309d Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.820 20955 DEBUG kuryr_kubernetes.k8s_client [-] Annotate /api/v1/namespaces/default/pods/kuryr-demo12345-6d8c5d7488-pbhb9: ['openstack.org/kuryr-vif'] ann Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.965 20955 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/dispatch.py:6 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.966 20955 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.130,lb_ip=None,ports=[LBaa Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.966 20955 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] YOBO - spec=LBaaSServiceSpec(ip=10.0.0.130,lb_ip=None,ports=[LBaaSPortSpec],project_id='cc1 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.967 20955 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] YOBO - endpoints={u'kind': u'Endpoints', u'subsets': None, u'apiVersion': u'v1', u'metadata lines Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.820 20955 DEBUG kuryr_kubernetes.k8s_client [-] Annotate /api/v1/namespaces/default/pods/kuryr-demo12345-6d8c5d7488-pbhb9: ['openstack.org/kuryr-vif'] ann Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.965 20955 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/dispatch.py:6 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.966 20955 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.130,lb_ip=None,ports=[LBaa Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.966 20955 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] YOBO - spec=LBaaSServiceSpec(ip=10.0.0.130,lb_ip=None,ports=[LBaaSPortSpec],project_id='cc1 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.967 20955 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] YOBO - endpoints={u'kind': u'Endpoints', u'subsets': None, u'apiVersion': u'v1', u'metadata Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.967 20955 DEBUG kuryr_kubernetes.handlers.retry [-] Report handler unhealthyX LoadBalancerHandler __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/ha Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.967 20955 DEBUG kuryr_kubernetes.handlers.retry [-] YOBOYOBO!! __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/retry.py:72 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.967 20955 DEBUG kuryr_kubernetes.handlers.retry [-] yex = 'NoneType' object is not iterable!! __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/handle Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.967 20955 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/dispatch.py:6 Feb 20 11:40:29 rrrrr python[20955]: 2018-02-20 11:40:29.968 20955 DEBUG kuryr_kubernetes.controller.handlers.lbaas [-] Got LBaaSServiceSpec from annotation: LBaaSServiceSpec(ip=10.0.0.130,lb_ip=None,ports=[LBaa Feb 20 11:40:30 rrrrr python[20955]: 2018-02-20 11:40:30.330 20955 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/dispatch.py:6 Feb 20 11:40:30 rrrrr python[20955]: 2018-02-20 11:40:30.331 20955 DEBUG kuryr_kubernetes.controller.handlers.vif [-] Got VIF from annotation: VIFBridge(active=False,address=fa:16:3e:7b:9e:18,bridge_name='qbr309 Feb 20 11:40:30 rrrrr python[20955]: 2018-02-20 11:40:30.377 20955 DEBUG kuryr_kubernetes.handlers.retry [-] Handler VIFHandler failed (attempt 1; ResourceNotReady: Resource not ready: VIFBridge(active=False,add Feb 20 11:40:30 rrrrr python[20955]: 2018-02-20 11:40:30.678 20955 DEBUG kuryr_kubernetes.controller.handlers.vif [-] Setting VIF annotation: VIFBridge(active=False,address=fa:16:3e:c9:73:1b,bridge_name='qbr70bb Feb 20 11:40:30 rrrrr python[20955]: 2018-02-20 11:40:30.679 20955 DEBUG kuryr_kubernetes.k8s_client [-] Annotate /api/v1/namespaces/default/pods/kuryr-demo12345-6d8c5d7488-gm824: ['openstack.org/kuryr-vif'] ann Feb 20 11:40:31 rrrrr python[20955]: 2018-02-20 11:40:31.184 20955 DEBUG kuryr_kubernetes.handlers.dispatch [-] 1 handler(s) available __call__ /opt/stack/kuryr-kubernetes/kuryr_kubernetes/handlers/dispatch.py:6 Feb 20 11:40:31 rrrrr python[20955]: 2018-02-20 11:40:31.185 20955 DEBUG kuryr_kubernetes.controller.handlers.vif [-] Got VIF from annotation: VIFBridge(active=False,address=fa:16:3e:c9:73:1b,bridge_name='qbr70b l