conductor: race between processes to join servicegroups when zk driver is used

[Pawel Palucki]

The bugs manifests only when two processes of the same service tries to register the same node.

When multiple processes of one service (nova-conductor for now) tries to join servicegroup and zookeeper driver is used
there is a race between processes: all processes tries to register itself in zookeeper as the same member in the same namespace.

Zookeeper path looks like this:

/servicegroups/conductor/MEMBER_ID

Each process tries to create this node, which already exists.

This ends up with each process trying endlessly register itself which ends with traceback:

Traceback (most recent call last):
  File "/opt/stack/nova/nova/servicegroup/drivers/zk.py", line 106, in join
    member = membership.Membership(self._session, path, member_id)
  File "/usr/local/lib/python2.7/dist-packages/evzookeeper/membership.py", line 130, in __init__
    self.refresh(quiet=False)
  File "/usr/local/lib/python2.7/dist-packages/evzookeeper/membership.py", line 155, in refresh
    self._join()
  File "/usr/local/lib/python2.7/dist-packages/evzookeeper/membership.py", line 203, in _join
    raise RuntimeError("Duplicated membership name %s" % path)
RuntimeError: Duplicated membership name /servicegroups/conductor/MEMBER_ID

For now only nova-conductor is affected because it's the one only service that forks.

There is not other consequences except polluted logs and confusion of operator.

ubuntu 14.04 + zookeeper 3.4.5

The bug is related to other bug [1] that any of the processes isn't going to register itself, because processes locks on
communication with zookeeper.

[1] https://bugs.launchpad.net/nova/+bug/1389782

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/133479

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/133500

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/133500
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e61bf70146c47a99394a143c598ebd73409eca47
Submitter: Jenkins
Branch: master

commit e61bf70146c47a99394a143c598ebd73409eca47
Author: Pawel Palucki <email address hidden>
Date: Fri Nov 7 14:41:49 2014 +0100

    Fix conductor processes race trying to join servicegroup (zk driver)

    When conductor is run in multi process manner and zk (zookeeper) driver
    is used as servicegroup driver, there is a problem because each process
    tries to manage own Membership object to the same zookeeper path.

    This ends with raising exceptions:

    RuntimeError: Duplicated membership name /servicegroups/conductor/MEMBER_ID

    Zookeeper driver uses Membership (evzookeeper) class with path related
    to service type and AFAIK it isn't correct that many process will be
    responsible for the same ephemeral node. From my research it is not
    supported but evzookeeper (Membership class) - so we can ignore the
    exception or give each process his own node.

    If we ignore exception (silent it) and when first registered process dies
    and the ephemeral node disappears, another process will create it. It will
    work but hides the information about overall structure of services and
    also causes that each process endlessly will be trying to create a node
    (sending invalid create node requests to zookeeper). IMO is not a clean solution.

    So there is another solution that each process has its own node. This
    fix does that.

    The best unique identifier for process is pid, so the chosen solution,
    reorganizes the structure of zookeeper tree by adding one more level
    with process ids.

    The zookeeper tree before looks like this:

    /servicesgroups/SERVICE/MEMBER

    and after path will look like this:

    /servicegroups/SERVICE/MEMBER/PID
    eg.
    /servicegroups/conductor/foo/12345

    This solution also assumes, that servicegroup driver will not check existence of
    member node, but existence of subnodes (pids) - which corresponds to existence
    of processes of given service.

    In general we will have more granular information about whole system -
    for exmaple we can check number of processes of given service on each node.

    To answer the question: is service on given node works, we have to check
    number of ephemeral "pids" nodes in get_all() method.

    Closes-bug: #1390511

    Related-bug: #1389782
    Related-bug: #1382153

    Change-Id: I478845b6921dcfb9e9af5a45283a8569051b4f4f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/133479
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=afe86b6f29033a472cab1b52dd0724bb3c6dfb82
Submitter: Jenkins
Branch: master

commit afe86b6f29033a472cab1b52dd0724bb3c6dfb82
Author: Michal Dulko <email address hidden>
Date: Wed Feb 4 12:44:12 2015 +0100

    Fix conductor servicegroup joining when zk driver is used

    When conductor is run as multiprocess (default for multi core system) and
    zk (zookeeper) is used as servicegroup_driver then conductor is unable to join
    servicegroup because of shared zookeeper handle (and probably socket)
    between parent and children processes.

    It's found the problem lies in zookeeper c library implementation.
    Proof can be seen in related bug #1389782.

    This fix follows the idea used by memcache and db driver that
    servicegroup_api._driver object is used in lazy manner.
    This means that like connection to memcache and session to database,
    zookeeper handle (zk session in driver) isn't created until required by
    worker (child process).

    Additional note: before fix, during Service object creation the
    prefix in zookeeper was created. That was the probably reason the session was
    established so early. In my opinion the eagerness of this is not necessary
    and namespace can be created by child process as well.

    Closes-Bug: #1389782

    Related-bug: #1390511
    Related-bug: #1382153

    Change-Id: I9b386ef1f9268d19d04879ec89e5684170f3862a