Change tooz coordination service backend from memcached to etcd

[Tytus Kurek]

Currently designate charm uses memcached as a backend for the tooz coordination service.

The following issues were found when running OpenStack cloud in a production:

# cat /var/log/designate/designate-zone-manager.log.1 | grep ERROR
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall [-] Fixed interval looping call 'designate.zone_manager.service.Service._coordinator_run_watchers' failed
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall Traceback (most recent call last):
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 136, in _run_loop
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw)
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/designate/coordination.py", line 120, in _coordinator_run_watchers
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall self._coordinator.run_watchers()
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/tooz/drivers/memcached.py", line 545, in run_watchers
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall result = super(MemcachedDriver, self).run_watchers(timeout=timeout)
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/tooz/coordination.py", line 502, in run_watchers
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall MemberLeftGroup(group_id, member_id)))
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/tooz/coordination.py", line 102, in run
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall return list(map(lambda cb: cb(*args, **kwargs), self))
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/tooz/coordination.py", line 102, in <lambda>
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall return list(map(lambda cb: cb(*args, **kwargs), self))
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/designate/coordination.py", line 151, in _on_group_change
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall members, self._my_partitions = self._update_partitions()
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/designate/coordination.py", line 176, in _update_partitions
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall members, self._my_id, self._partitions)
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/designate/coordination.py", line 161, in _partition
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall my_index = members.index(me)
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall ValueError: 'juju-bd82de-32-lxd-18:5b4b87d6-b4fd-4230-90df-94da98534499' is not in list
2018-03-12 18:51:50.861 4398 ERROR oslo.service.loopingcall

The logs are not verbose enough to clearly define the root cause, but the most probable one seems to be a timeout when querying the memcached backend. This assumption is based on the following findings:

* the "run_watchers" function from "/usr/lib/python2.7/dist-packages/tooz/drivers/memcached.py" file does not exit on timeout:

    def run_watchers(self, timeout=None):
        result = super(MemcachedDriver, self).run_watchers(timeout=timeout)
        self.run_elect_coordinator()
        return result

* it is mentioned in the official tooz documentation (https://docs.openstack.org/tooz/latest/user/drivers.html) that "the memcached driver is a basic implementation and provides little resiliency, though it’s much simpler to setup; a lot of the features provided in tooz are based on timeout (heartbeats, locks, etc) so are less resilient than other backends".

Although the above issue is environment-specific, I would like to raise a concern that in current charm implementation the tooz coordination service uses a backend which is not recommended by the upstream community. Therefore, I would like to suggest to change it to zookeper, which according to the upstream recommendations is the reference implementation. Moreover, as with zookeeper "primitives are based on sessions (and typically require careful selection of session heartbeat periodicity and server side configuration of session expiry)", the issue described above should not take place when using zookeeper instead of memcached.

I am happy to work on the patch, but before I start I would like to get a green light, so that I could be ensured that the patch is going to be merged at some point.

[Tytus Kurek (tkurek)]

I discussed that offline with Kiko and Ante and the suggestion is to use redis instead of zookeeper. Anyway, I would still appreciate a feedback before I start working on that.

[James Page (james-page)]

Would etcd be an option? The etcd charm is well maintained and will form part of the upcoming secrets management work with vault, so I'd rather move to etcd rather than introduce redis or zookeeper - neither charm is fantastically well supported right now.

[Dmitrii Shcherbakov (dmitriis)]

I don't think redis is much better for coordination purposes than memcached being a mostly in-memory store.

https://docs.openstack.org/tooz/latest/user/drivers.html#characteristics
https://docs.openstack.org/tooz/latest/user/compatibility.html

Subjectively, a backing store for a coordination system should be consistent.

This thread has some considerations discussed: http://lists.openstack.org/pipermail/openstack-dev/2017-March/113885.html

Based on what I see, designate requires leadership and grouping support:

https://github.com/openstack/designate/blob/master/designate/coordination.py

From the compatibility page I can see that only zookeeper checks all boxes while being a consistency-oriented system.

There are two drivers for etcd (2 and 3). etcd 3 supports distributed lock manager (DLM) functionality and group membership but not leadership: https://github.com/openstack/tooz/blob/master/tooz/drivers/etcd3.py

Designate uses those two methods in coordination.py which are not implemented for etcd3 in tooz:
"@staticmethod
    def watch_elected_as_leader(group_id, callback):
        raise tooz.NotImplemented

    @staticmethod
    def unwatch_elected_as_leader(group_id, callback):
        raise tooz.NotImplemented"

[Tytus Kurek (tkurek)]

In my opinion the decision should be driven by the following question:

Which upstream tool fits the best for this purpose?

and not by:

Which charm which could replace memcached is better maintained?

Perhaps, it requires a broader discussion.

[Dmitrii Shcherbakov (dmitriis)]

The best fit would be etcd3 if it had support for leadership methods because Designate uses them in the coordination code.

leader_methods = [
    'watch_elected_as_leader',
    'unwatch_elected_as_leader',
    'stand_down_group_leader',
    'get_leader',
]

As of Queens only Memcached, Redis, Zake and Zookeeper have that supported:
https://github.com/openstack/tooz/blob/stable/queens/tools/compat-matrix.py#L104-L119

[Dmitrii Shcherbakov (dmitriis)]

Even better, we could simply use Designate worker and Producer, which do not require the leadership methods mentioned in #5, instead of zone manager and pool manager which are deprecated for removal in Rocky .

https://bugs.launchpad.net/charm-designate/+bug/1773190

From the source code perspective, the pool manager which is deprecated for removal uses LeaderElection:

designate/pool_manager/service.py|27| from designate import coordination
designate/pool_manager/service.py|82| class Service(service.RPCService, coordination.CoordinationMixin,
designate/pool_manager/service.py|191| self._pool_election = coordination.LeaderElection(

While Producer code does not - it relies on the partitioner code from coordination.py

designate/producer/service.py|21| from designate import coordination
designate/producer/service.py|35| class Service(service.RPCService, coordination.CoordinationMixin,
designate/producer/service.py|69| self._partitioner = coordination.Partitioner(
designate/producer/service.py|70| self._coordinator, self.service_name, self._coordination_id,

I can find only Group references there.

https://github.com/openstack/designate/blob/stable/queens/designate/coordination.py#L147-L246
    def _get_members(self, group_id):
        get_members_req = self._coordinator.get_members(group_id)
...
        if self._coordinator:
            self._coordinator.watch_join_group(
                self._group_id, self._on_group_change)
            self._coordinator.watch_leave_group(
                self._group_id, self._on_group_change)

Worker code does not even import coordination module and hence does not rely on tooz.

This would allow us to simply switch to etcd.