Create separate queue for state reports

[Eugene Nikanorov]

In big clusters having hundreds of nodes, neutron rpc workers could be consumed by rpc requests so much that they can't process state reports from agents on time.

That lead to a condition when agents begin to "flap", appear dead and alive. This in turn causes rescheduling which loads neutron-server even more, creating self-sustaining loop.

[Eugene Nikanorov (enikanorov)]

https://review.openstack.org/#/c/226362/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/226362
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7267d75fdd3f90af759d71e9490cd41d41ba6d98
Submitter: Jenkins
Branch: master

commit 7267d75fdd3f90af759d71e9490cd41d41ba6d98
Author: Eugene Nikanorov <email address hidden>
Date: Tue Sep 22 18:51:56 2015 +0400

    Use separate queue for agent state reports.

    This optimization is needed for big clusters with hundreds
    of agents where the spike of activity may trigger a burst
    of RPC requests that would prevent neutron-server from processing
    agent heart beats in time, triggering resource rescheduling.

    This will be further optimized by running dedicated RPC workers
    for state reports processing.

    Related-Bug: #1496410
    Change-Id: Id86a1f962aaa4f64011d57ae55d240f890cca4f7

[YAMAMOTO Takashi (yamamoto)]

https://review.openstack.org/#/c/230333/

[Armando Migliaccio (armando-migliaccio)]

@Eugene: do you intend to leverage [1] to have some workers only process state updates? How far off are you from doing that?

If I read the bug title, I'd say the job here is done and what you're about to do can be tracked in another bug report?

What do you think?

[Armando Migliaccio (armando-migliaccio)]

Forgot this:

[1] https://blueprints.launchpad.net/neutron/+spec/multiple-rpc-workers

[Armando Migliaccio (armando-migliaccio)]

I meant 'I read the bug title, I'd say the job here is done', besides the dedicated worker part ;)

Ok, I need more coffee.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/230333
Committed: https://git.openstack.org/cgit/openstack/networking-midonet/commit/?id=3b0bae71386d15ac31db0e4a9a0bb90080050f52
Submitter: Jenkins
Branch: master

commit 3b0bae71386d15ac31db0e4a9a0bb90080050f52
Author: YAMAMOTO Takashi <email address hidden>
Date: Fri Oct 2 18:07:36 2015 +0900

    Listen alternative topic for state reports

    Recently Neutron agents have been changed to use topics.REPORTS
    for state reports. [1] Update our plugin to accept the topic as well.

    In case of MidoNet, this topic is used to accept messages from
    Neutron dhcp and metadata agents. While MidoNet v2015.09 and
    later will have an alternative metadata proxy [2] which doesn't use
    these agents, it's nice to maintain the old way at least for
    a cycle. And v1 plugin needs this anyway.

    [1] Id86a1f962aaa4f64011d57ae55d240f890cca4f7
    [2] https://github.com/midonet/midonet/blob/c44eadbf1bd676e232099487b81cfcf6189d3bba/docs/openstack-metadata-usage.md

    Related-Bug: #1496410
    Change-Id: I65aeb3497cc11e0603a8c7af9bae129088924935

[Eugene Nikanorov (enikanorov)]

Armando, filed a separate bug report for adding dedicated workers: https://bugs.launchpad.net/neutron/+bug/1505217
Corresponding patch: https://review.openstack.org/#/c/233605/

[Eugene Nikanorov (enikanorov)]

Also, changed the bug title (removed 'with dedicated workers')

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/240106

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/liberty
Review: https://review.openstack.org/240167

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/240106
Committed: https://git.openstack.org/cgit/openstack/networking-midonet/commit/?id=d242171489c2d008c144c480941d0d284ef659bb
Submitter: Jenkins
Branch: master

commit d242171489c2d008c144c480941d0d284ef659bb
Author: YAMAMOTO Takashi <email address hidden>
Date: Thu Oct 29 12:12:24 2015 +0900

    Remove no longer necessary conditional

    It was necessary to make this compatible with Liberty.
    It's no longer necessary as now master is open for Mitaka.

    Related-Bug #1496410
    Change-Id: I951a35abeae08febad97ac6bf4dadca68a1a9758

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/kilo
Review: https://review.openstack.org/241370

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ann Kamyshnikova (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/240167

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/241370
Reason: No way it will merge due to breakage in upgrade scenarios.