ZmqProxy's PUB sockets cause eventlet polling loop - 100% cpu utilization

[Erica Windisch]

In some(?) versions of ZeroMQ and/or Eventlet, the ZmqProxy (used by nova-rpc-zmq-receiver) can consume 100% cpu.

This seems to be caused by a bug or quirk in the ZMQ_FD polling, either in Eventlet or ZeroMQ itself.

Originally reported by Joe Gordon. Confirmed by Eric Windisch

[Erica Windisch (ewindisch)]

https://review.openstack.org/14337

[Stuart Stent (stuart-stent)]

Is there an update to this bug?

Has this been patched upstream , if so what version of zeromq fixes it

[Erica Windisch (ewindisch)]

Not yet fixed upstream.

The following git-fork/branch includes a fix:
 https://bitbucket.org/zedshaw/eventlet

I have debs with this alternative eventlet.green.zmq code patched in. They'll be uploaded to
https://launchpad.net/~ewindisch/+archive/eventlet-zmq-et

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/19312

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/19312
Committed: http://github.com/openstack/oslo-incubator/commit/ab043101589a9167357ca317ca610a3f4144747c
Submitter: Jenkins
Branch: master

commit ab043101589a9167357ca317ca610a3f4144747c
Author: Eric Windisch <email address hidden>
Date: Wed Jan 9 12:30:47 2013 -0500

    zmq-receiver msg forwarding in greenthreads

    Because PUSH sockets can block and there
    may not be PULL consumers, the oslo-zmq-receiver
    process may stall in certain senarios.

    Improves error handling in ZmqProxy.

    This addresses bug 1097856.

    Additionally, this bug seems to improve
    (but not necessarily fix) the behavior
    reported in bug 1065532.

    Change-Id: I6df6035a6676c5bcdddaec7a332ac77e621ba9f3

[Erica Windisch (ewindisch)]

The previous patch was at best a partial workaround and should not have resolved this ticket.

Eventlet needs to be bumped to version 0.12.0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/21012

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/21012
Committed: http://github.com/openstack/oslo-incubator/commit/4a9a7dc39c296455349a59346d22cc06fc5364c8
Submitter: Jenkins
Branch: master

commit 4a9a7dc39c296455349a59346d22cc06fc5364c8
Author: Eric Windisch <email address hidden>
Date: Fri Feb 1 11:56:42 2013 -0500

    Bump eventlet to 0.12.0

    This version has important fixes required
    for ZeroMQ (spurious wakeups).

    Fixes bug 1065532.

    Change-Id: I16b9220db1f5bf16a24644887963ce18e4e5ecbd