Kazoo Driver Does not retry connections with Zookeeper

[Rohit Jaiswal]

https://github.com/openstack/tooz/blob/master/tooz/drivers/zookeeper.py#L339 can support proxying options to KazooClient in addition to just host. eg retrying connections in case of connection failure is one such option that can be supported in Tooz KazooDriver.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/223259

[Rohit Jaiswal (rohit-jaiswal-3)]

Ceilometer uses Tooz for agent coordination and configurable connection retries will be useful to build resilience against random connection failures.

For example i see this in notification agent logs:

(kazoo.client): 2015-09-11 18:49:35,331 DEBUG connection _submit Sending request(xid=2): Create(path=u'/tooz/ceilometer.notification/b279f2ed-fe04-4113-b374-4627745c711c', data='\xc4\x00', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=1)
(kazoo.client): 2015-09-11 18:49:38,485 Level 5 connection _submit Sending request(xid=-2): Ping()
(kazoo.client): 2015-09-11 18:49:41,450 WARNING connection _connect_attempt Connection dropped: outstanding heartbeat ping not received
(kazoo.client): 2015-09-11 18:49:41,450 WARNING connection _connect_attempt Transition to CONNECTING
(kazoo.client): 2015-09-11 18:49:41,450 INFO client _session_callback Zookeeper connection lost
(ceilometer.openstack.common.threadgroup): 2015-09-11 18:49:41,463 ERROR threadgroup wait
Traceback (most recent call last):
  File "/opt/stack/venv/ceilometer-20150911T173109Z/lib/python2.7/site-packages/ceilometer/openstack/common/threadgroup.py", line 145, in wait
    x.wait()
  File "/opt/stack/venv/ceilometer-20150911T173109Z/lib/python2.7/site-packages/ceilometer/openstack/common/threadgroup.py", line 47, in wait
    return self.thread.wait()
  File "/opt/stack/venv/ceilometer-20150911T173109Z/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
    return self._exit_event.wait()
  File "/opt/stack/venv/ceilometer-20150911T173109Z/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "/opt/stack/venv/ceilometer-20150911T173109Z/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    return self.greenlet.switch()
  File "/opt/stack/venv/ceilometer-20150911T173109Z/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/opt/stack/venv/ceilometer-20150911T173109Z/lib/python2.7/site-packages/ceilometer/openstack/common/service.py", line 491, in run_service
    service.start()
  File "/opt/stack/venv/ceilometer-20150911T173109Z/lib/python2.7/site-packages/ceilometer/notification.py", line 143, in start
    self.partition_coordinator.join_group(self.group_id)
  File "/opt/stack/venv/ceilometer-20150911T173109Z/lib/python2.7/site-packages/ceilometer/coordination.py", line 125, in join_group
    join_req.get()
  File "/opt/stack/venv/ceilometer-20150911T173109Z/lib/python2.7/site-packages/tooz/drivers/zookeeper.py", line 427, in get
    return self._handler(self._kazoo_async_result, timeout, **self._kwargs)
  File "/opt/stack/venv/ceilometer-20150911T173109Z/lib/python2.7/site-packages/tooz/drivers/zookeeper.py", line 137, in _join_group_handler
    raise coordination.ToozError(utils.exception_message(e))
ToozError
(kazoo.client): 2015-09-11 18:49:41,550 WARNING connection zk_loop Failed connecting to Zookeeper within the connection retry policy.
(kazoo.client): 2015-09-11 18:49:41,551 INFO client _session_callback Zookeeper session lost, state: CLOSED
(kazoo.client): 2015-09-11 18:49:41,551 Level ...

[Rohit Jaiswal (rohit-jaiswal-3)]

The above stack trace is observed during notification agent start up in HA mode

[Rohit Jaiswal (rohit-jaiswal-3)]

Also, the implementation of join_group [1] in ceilometer/coordination.py should handle ToozError, currently it just handles MemberAlreadyExist and GroupNotCreated exceptions.

[1] https://github.com/openstack/ceilometer/blob/master/ceilometer/coordination.py#L115-L133

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/223259
Committed: https://git.openstack.org/cgit/openstack/tooz/commit/?id=4ae41738220e48148cc5ef2a3ecf350e067ff753
Submitter: Jenkins
Branch: master

commit 4ae41738220e48148cc5ef2a3ecf350e067ff753
Author: Joshua Harlow <email address hidden>
Date: Mon Sep 14 11:47:45 2015 -0700

    Allow more kazoo specific client options to be proxied through

    Closes-Bug: #1495663

    Change-Id: I3710845f9e66ab574eb358000b859dc96cbf08a0

[gordon chung (chungg)]

is this still a problem with ceilometer or is it address by bug 1496982

[Rohit Jaiswal (rohit-jaiswal-3)]

https://review.openstack.org/#/c/224919/ (fix for bug 1496982) will just address issues when joining group so that the agent gets a chance to initialize listeners and start up correctly.

bug 1495663 is about adding a param to config to cap conn retries and should be useful for any other operations with Tooz/Zookpr. I think this is more generic fix and depends on the python-tooz fix above.

[gordon chung (chungg)]

ack, so a change is still required

[gordon chung (chungg)]

corresponding tooz lib has been released.