[Perfomance] Incorrect usage of notification listeners

Bug #1570468 reported by Kirill Bespalov
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Status tracked in Mitaka
Liberty
New
Undecided
Kirill Bespalov
Mitaka
Fix Released
High
Kirill Bespalov

Bug Description

Notification agent in the Ceilometer for each target creates its own NotificationListener, but it allow by design to listen on many targets simultaneously.

It leads to perfomance overhead for the agent and overloading an amqp broker.

You can explore current implementation here:
https://github.com/openstack/ceilometer/blob/master/ceilometer/notification.py#L246

For example by default Ceilometer uses 60 targets:
6 sources (cpu_source, meter_source, etc) x 10 processing queues for coordination stuff (IPC queues mechanisms):

   ceilometer-pipe-cpu_source:cpu_delta_sink-0.sample
   ceilometer-pipe-cpu_source:cpu_delta_sink-1.sample
   ceilometer-pipe-cpu_source:cpu_delta_sink-2.sample
   ...
   ceilometer-pipe-cpu_source:cpu_delta_sink-9.sample
   ...

So, after initialization will be run 2 workers proccess of notification agents, then each worker will be create 60 NotificationListener's and for each listener will be created a dedicated TCP connection to the amqp broker and it's own thread pool executor (default size 64).

As a result, every controller node with the notification agen has around 120 TCP connections to the amqp broker and 7680 started greenthreads.

Changed in ceilometer:
assignee: nobody → Kirill Bespalov (k-besplv)
description: updated
description: updated
description: updated
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/306035

Changed in ceilometer:
status: New → In Progress
gordon chung (chungg)
Changed in ceilometer:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/308366

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)

Reviewed: https://review.openstack.org/306035
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=726b2d4d67ada3df07f36ecfd81b0cf72881e159
Submitter: Jenkins
Branch: master

commit 726b2d4d67ada3df07f36ecfd81b0cf72881e159
Author: Kirill Bespalov <email address hidden>
Date: Tue Apr 19 13:29:02 2016 +0300

    Fix notification listeners usage

    Closes-Bug: #1570468

    Create the single notification pipeline listener and to aggregate
    targets and endpoints in it.

    In this implementation the incoming notification msg will be dispatched
    in a relevant endpoint by its own filter rule based on publisher id
    of a notifier, which sent this message.

    Each pipeline notifier has a conforming publisher id based on
    a pipeline name in which it to produce messages.

    Change-Id: If8c1c6838fd291ef8dbdd4d27c78fe5a36522178

Changed in ceilometer:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/312025

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (stable/mitaka)

Reviewed: https://review.openstack.org/312025
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=6292e0e42b8d4602e08a876a53141c3ddbdcbb58
Submitter: Jenkins
Branch: stable/mitaka

commit 6292e0e42b8d4602e08a876a53141c3ddbdcbb58
Author: Kirill Bespalov <email address hidden>
Date: Wed Apr 20 17:23:02 2016 +0300

    Fix notification listeners usage

    Closes-Bug: #1570468

    Create the single notification pipeline listener and to aggregate
    targets and endpoints in it.

    In this implementation the incoming notification msg will be dispatched
    in a relevant endpoint by its own filter rule based on publisher id
    of a notifier, which sent this message.

    Each pipeline notifier has a conforming publisher id based on
    a pipeline name in which it to produce messages.

    Change-Id: If8c1c6838fd291ef8dbdd4d27c78fe5a36522178
    (cherry picked from commit 726b2d4d67ada3df07f36ecfd81b0cf72881e159)

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on ceilometer (stable/mitaka)

Change abandoned by gordon chung (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/308366
Reason: https://review.openstack.org/#/c/312025/

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/ceilometer 6.1.0

This issue was fixed in the openstack/ceilometer 6.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/332417

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ceilometer (master)
Download full text (8.8 KiB)

Reviewed: https://review.openstack.org/332417
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=cd1004c8c6855892d5259bc18e2f819317b0c3df
Submitter: Jenkins
Branch: master

commit f7e27a2cf6a8ad084f36575a05a856cdbd9e528c
Author: Julien Danjou <email address hidden>
Date: Wed Jun 1 10:23:51 2016 +0200

    dispacher/gnocchi: measures sent fix logging

    Currently, the logging is only done when measures are posted on retry. If it
    works on the first call, no logging is done. This can be confusing for users
    that try to debug the Gnocchi dispatcher.

    Change-Id: Ife7879fe0c153cec1a79c7d57529fda82b7b7c16

commit 5976031a7e6bd0bea6713f96ec7f892b945f9f86
Author: Ken'ichi Ohmichi <email address hidden>
Date: Mon Jun 6 14:35:28 2016 +0900

    Copy images_client from tempest + Correct concurrency of gabbi 1.22.0

    Now the gate test of ceilometer is broken, because the ceilometer test
    is using nonstable tempest interfaces and these interfaces are changed.
    QA team defines interfaces which are not under tempest.lib as unstable
    clearly according to

    https://github.com/openstack/tempest/blob/master/doc/source/library.rst#stability

    So current test way is against the QA policy, so here is a workaround.
    Copy&paste is not good, almost wrong. However, if still continuing the
    tests, it would be a nice option to copy&paste and pass the gate test.
    After QA team defines stable interfaces, that is a nice time to switch
    using the stable interfaces with removing this copy&paste code.

    Closes-Bug: #1589426

    Depends-On: I3abd9049560ee507b3610ab482c697a239f13a3b
    (cherry picked from commit 6c0b97b1e6fc03bdf84bf9b888761bd22ea0ad58)

    +

    When running gabbi under testr, concurrency grouping is controlled
    by a regex in .testr.conf. A module name change in gabbi 1.22.0
    inadvertently broke the grouping. This change should work for
    old and new versions.

    (cherry picked from commit 8fcbc8de4ea878bdec1ef8754eaf965e176854b3)

    Change-Id: I30266f1b690ea105511d9b1162755bcc85e2e69a

commit 9c08a89fe2dec2965bc63764b95d68fc1b1f12c3
Author: ZhiQiang Fan <email address hidden>
Date: Thu Jun 2 20:19:28 2016 +0800

    catch DriverLoadFailure for get_transport optional

    new oslo.messaging 5.2.0 release will raise
    oslo_messaging.transport.DriverLoadFailure if no existent transport
    driver, hence breaks our code and unit test as well.

    This patch fixes it by adding DriverLoadFailure to catch list.

    Also:

    tempest: import ImagesClient as ImagesClientV2

    This commit [1] renamed ImagesClientV2 to ImagesClient so Ceilometer
    Tempest tests do not work anymore.

    [1] https://github.com/openstack/tempest/commit/2fea405bd6ea8674c8daaf8ebb788ae33d52c976

    Change-Id: I02b5d1b3f4a9169cbf428a5eff9c6705494741af
    Closes-Bug: #1588151
    (cherry picked from commit d7f8ce94e433da8af3660ecf52715a04c221fde5)
    (cherry picked from commit f6b2406fffdd4deae508b9f251691fe9b70cb1ee)

commit 7951eb5edff433d12be7fe16ffbaa6ea140b37bf
Author: ZhiQiang Fan <email address hidden>
Date: ...

Read more...

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote : Fix included in openstack/ceilometer 7.0.0.0b2

This issue was fixed in the openstack/ceilometer 7.0.0.0b2 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.