Zookeeper driver doesn't work with multi-process neutron-server

[Li Ma]

Zookeeper driver is under review:
https://review.openstack.org/#/c/264113/4

This bug report is to track the progress on this critical problem for zookeeper.

Problem Discovered:
Zookeeper operations are all blocked forever in Df core plugin, but works very well in DF_local_controller, using the same Zookeeper driver codebase.

The difference between DF_local_controller and neutron-server is that DF_local_controller is single process but neutron-server starts with multiple processes.

In neutron-server:

It first initialize core plugin and service plugins.
After that, it passes plugin objects to sub-process (workers).
It means that all the plugin objects are copied to multiple processes.

When initializing DF core plugin, it creates a DB client via DB-API layer. Zookeeper client-side has some local states which seems incompatible with multi-workers.

If we can initialize plugin objects after forking, it should work. However, it seems not realistic. You have to modify the neutron core mechanism. As a result, I will try to find how to make it in DF side.

[Eran Gampel (eran-gampel)]

You can do some thing similar to what I did in the
Pub sub patch use a multi process queue
And just have one process send the message s
For all the process

or
You can initialize the client in the post fork
Initialize method in the df plugin
And run multiple clients

I am in the air port now I will add the links to the
Patch and the call back later

[Li Ma (nick-ma-z)]

Thanks for your suggestion. Actually I just solved it using lazy initialization. The zookeeper is working. :-)
I'm testing all the functionality (without PUB/SUB).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/264113
Committed: https://git.openstack.org/cgit/openstack/dragonflow/commit/?id=4033d3833b6f71f549d752b28163dbe15cbda265
Submitter: Jenkins
Branch: master

commit 4033d3833b6f71f549d752b28163dbe15cbda265
Author: Li Ma <email address hidden>
Date: Sat Dec 26 13:31:30 2015 +0800

    Add Zookeeper driver for DB

    The devstack only supports ubuntu,
    because I don't have other envs.

    I'll update devstack scripts when I set
    up other envs.

    To solve the stateful client bug, lazy
    initialization is introduced.

    Change-Id: I3e74601f2b867b695578f18be2faf7efc33bc7aa
    Implements: blueprint zookeeper-driver
    Closes-Bug: #1534504