nova notifier should send first event immediately

[Kevin Benton]

The nova batch notifier currently waits to batch up events on every single notification. This means even in a quiet system, the batching mechanism will add a delay to every single notification to Nova. This extra 2 seconds (or whatever it is configured to) builds up over time and adds quite a bit of delay to tempest runs (2 seconds * number of booted VMs).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/300245

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/300245
Reason: This review is > 4 weeks without comment and currently blocked by a core reviewer with a -2. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and contacting the reviewer with the -2 on this review to ensure you address their concerns.

[Kevin Benton (kevinbenton)]

Quantified some time spent waiting in the nova notifier here:

Results across the threads waiting:

kbenton-mbp:analyze kbenton$ grep "Total wasted" screen-q-svc.txt.gz | awk '{ print $3 }' | sort | uniq
2967
2968
2969
kbenton-mbp:analyze kbenton$ grep "Total wasted" screen-q-svc.txt.gz | grep 2967 | awk -F "Total wasted: " '{ print $2 }' | tail -n 1
232.0 _notify /opt/stack/new/neutron/neutron/notifiers/batch_notifier.py:76
kbenton-mbp:analyze kbenton$ grep "Total wasted" screen-q-svc.txt.gz | grep 2968 | awk -F "Total wasted: " '{ print $2 }' | tail -n 1
236.0 _notify /opt/stack/new/neutron/neutron/notifiers/batch_notifier.py:76
kbenton-mbp:analyze kbenton$ grep "Total wasted" screen-q-svc.txt.gz | grep 2969 | awk -F "Total wasted: " '{ print $2 }' | tail -n 1
190.0 _notify /opt/stack/new/neutron/neutron/notifiers/batch_notifier.py:76

So we cumulatively spent 3-4 minutes on each thread waiting to send what ended up being single events.

I would like to just turn this into a regular rate limiter to eliminate this pessimistic waiting period.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/300245
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=255e8a839db0be10c98b5d9f480ce476e2f2e171
Submitter: Jenkins
Branch: master

commit 255e8a839db0be10c98b5d9f480ce476e2f2e171
Author: Kevin Benton <email address hidden>
Date: Thu Mar 31 05:03:12 2016 -0700

    Turn nova notifier into a proper rate limiter

    This adjusts the batching logic in the Nova notifier to immediately
    send and then sleep to allow batching of subsequent calls in the batch
    interval.

    So rather than always wait for 2 seconds to elapse while batching,
    batching will only occur in the 2 second period after a call is made.
    This turns the batch notifier into a standard queuing rate limiter.

    The upside to this is a single port create results in an immediate
    notification to Nova without a delay.

    The downside is now that a sudden burst of 6 port creations to a
    previously idle server will result in 2 notification calls to Nova
    (1 for the first call and another for the other 5).

    Closes-Bug: #1564648
    Change-Id: I82f403441564955345f47877151e0c457712dd2f

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/447684

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/447684
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b7e12b276a58830130781eb9a81154834698b62b
Submitter: Jenkins
Branch: stable/ocata

commit b7e12b276a58830130781eb9a81154834698b62b
Author: Kevin Benton <email address hidden>
Date: Thu Mar 31 05:03:12 2016 -0700

    Turn nova notifier into a proper rate limiter

    This adjusts the batching logic in the Nova notifier to immediately
    send and then sleep to allow batching of subsequent calls in the batch
    interval.

    So rather than always wait for 2 seconds to elapse while batching,
    batching will only occur in the 2 second period after a call is made.
    This turns the batch notifier into a standard queuing rate limiter.

    The upside to this is a single port create results in an immediate
    notification to Nova without a delay.

    The downside is now that a sudden burst of 6 port creations to a
    previously idle server will result in 2 notification calls to Nova
    (1 for the first call and another for the other 5).

    Closes-Bug: #1564648
    Change-Id: I82f403441564955345f47877151e0c457712dd2f
    (cherry picked from commit 255e8a839db0be10c98b5d9f480ce476e2f2e171)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 10.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.0.0b1 development milestone.