conductor: race between processes to join servicegroups when zk driver is used
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Undecided
|
Pawel Palucki |
Bug Description
The bugs manifests only when two processes of the same service tries to register the same node.
When multiple processes of one service (nova-conductor for now) tries to join servicegroup and zookeeper driver is used
there is a race between processes: all processes tries to register itself in zookeeper as the same member in the same namespace.
Zookeeper path looks like this:
/servicegroups/
Each process tries to create this node, which already exists.
This ends up with each process trying endlessly register itself which ends with traceback:
Traceback (most recent call last):
File "/opt/stack/
member = membership.
File "/usr/local/
self.
File "/usr/local/
self._join()
File "/usr/local/
raise RuntimeError(
RuntimeError: Duplicated membership name /servicegroups/
For now only nova-conductor is affected because it's the one only service that forks.
There is not other consequences except polluted logs and confusion of operator.
ubuntu 14.04 + zookeeper 3.4.5
The bug is related to other bug [1] that any of the processes isn't going to register itself, because processes locks on
communication with zookeeper.
Changed in nova: | |
assignee: | nobody → Pawel Palucki (pawel-palucki-q) |
Changed in nova: | |
milestone: | none → kilo-2 |
status: | Fix Committed → Fix Released |
Changed in nova: | |
milestone: | kilo-2 → 2015.1.0 |
Related fix proposed to branch: master /review. openstack. org/133479
Review: https:/