Neutron multiple api workers can't send cast message to agent when use zeromq

[Dongcan Ye]

When I set api_workers > 0 in Neutron configuration, delelting or adding router interface, Neutron L3 agent can't receive message from Neutron Server.
In this situation, L3 agent report state can cast to Neutron Server, meanwhile it can receive cast message from Neutron Server.(use call method)

Obviously, Neutron Server can use cast method for sending message to L3 agent, But why cast routers_updated fails? This also occurs in other Neutron agent.

Then I make a test, write some codes in Neutron server starts or l3_router_plugins, sends cast periodic message to L3 agent directly. From L3 agent rpc-zmq-receiver log file shows it receives message from Neutron Server.

By the way, everything works well when api_workers = 0.

Test environment:
neutron(master) + oslo.messaging(master) + zeromq

[Elena Ezhova (eezhova)]

Could you please specify how did you configure Neutron to use zmq? I am especially interested in the driver you used (MatchMakerRedis, MatchmakerRing).

[Dongcan Ye (hellochosen)]

Ok, eezhoova. I only use MatchmakerRing, because MatchMakerRedis can't work here.

In three node(controller, network, compute), matchmaker in neutron conf like this:
rpc_zmq_matchmaker = oslo.messaging._drivers.matchmaker_ring.MatchMakerRing

Here I use haproxy node for listen neturon service .

Compute and Network node:
In neutron_matchmaker_ring.json, I assign all neutron topic and host(host point to node ha, you can point to controller node when disable haproxy), here is a slice of it:
{
 "network": ["ha"],
    "subnet": ["ha"],
    "port": ["ha"],
    "security_group": ["ha"],
    "l2population": ["ha"],
    "create": ["ha"],
    "delete": ["ha"],
    "update": ["ha"],
    "q-agent-notifier": ["ha"],
    "q-plugin": ["ha"]
}

Controller node:
In neutron_matchmaker_ring.json, I also assign all neutron topic and host(host point to network node and compute node), here is a slice of it:
{
    "network": ["network","compute"],
    "subnet": ["network","compute"],
    "port": ["network","compute"],
    "security_group": ["network","compute"],
    "l2population": ["network","compute"],
    "create": ["network","compute"],
    "delete": ["network","compute"],
    "update": ["network","compute"],
    "q-agent-notifier": ["network","compute"],
    "q-plugin": ["network","compute"]
}

[Elena Ezhova (eezhova)]

Dongcan, thank you for providing the details.

[Elena Ezhova (eezhova)]

As I found out the problem is in zmq context which is a singleton and thus is created only once. [1] This leads to problems when there is more than one process working with it. [2]
The solution is to make zmq context thread-local by using threading.local class. [3]

I have a working fix that I will upload shortly.

[1] https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_zmq.py#L813
[2] http://lists.zeromq.org/pipermail/zeromq-dev/2011-December/014900.html
[3] https://docs.python.org/2/library/threading.html#threading.local

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/126914

[Elena Ezhova (eezhova)]

If we look at the problem from the Neutron side, we can see that while Neutron server starts, it firstly loads core plugin and service plugins, which start message handling server, and only then forks to create api-workers. As a result, all child processes get the same copy of the context.

In this case, instead of making singletony ZeroMQ Context thread-local, that is supposed to be used for threads and not processes, it is better to create new Context for each socket. This will prevent such situations from happening and will guarantee that each process works with its own Context.

I have updated the proposed fix accordingly.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/126914
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=0d49793e340728416c0c7b1bf964b54efd7e5acb
Submitter: Jenkins
Branch: master

commit 0d49793e340728416c0c7b1bf964b54efd7e5acb
Author: Elena Ezhova <email address hidden>
Date: Wed Oct 8 18:18:20 2014 +0400

    Create ZeroMQ Context per socket

    ZeroMQ Context is a singleton and thus is created only once. This leads
    to problems when there is more than one process working with it.
    For example, while Neutron server starts, it firstly loads core
    plugin and service plugins, which start message handling server,
    and only then forks to create api-workers. As a result, all child
    processes get the same copy of the context.

    Creating new Context for each socket will prevent such situations
    from happening and will guarantee that each process works with its
    own Context.

    Change-Id: I56912e39b119c20f6f23311fc2c7c4b9e9e480d0
    Closes-Bug: #1364814

[Dongcan Ye (hellochosen)]

Good job, thanks eezhoova.

[James Page (james-page)]

Released in Ubuntu in 1.5.1.